INFERENCE / SEE THE SCENE · STEP 1One picture becomes three sheets of numbers
ViT · mix patches. Squares stand for 256 image patches. Across 12 blocks, tokens exchange information; the gold summary token (CLS) becomes H.
MLP · mix features. One H vector widens to 2,048 hidden features, then narrows to Z. Dots stand for groups of features.
Connections are schematic, not measured weights.
L2 reads the L1 state, not the image. Follow the purple Z chart down to the blue encoder. The bars are recorded values; the dots and connections show the architecture.
TRAINING EXAMPLE / LEARN AN ACTION CODE · STEP 1Ten commands become two blocks
The scene encoder tells us where. This separate input records what movement was requested. Future commands are not supplied during planning.
Watch block 1 pass through the embedder, then block 2 use the same network. Each block produces its own vector. The passes are shown in turn for clarity; the model can batch them.
L1 described each block separately. The L2 action encoder reads both embeddings in time order to describe the whole ten-command chunk.
Tile values are rounded · x in white, y in gold. Select any move to inspect it.
INFERENCE / L2: CHOOSE A TARGET · STEP 1Where do the first 800 plans come from?
Previous example [4] · reference only
Queue values and statistics are from the frozen checkpoint. Trial values are a fixed CPU initialization example, not the original move-10 random draw. No predictions or rankings yet.
MODEL ATLAS / SEARCH & NAVIGATEComponent map
Follow either input stream: pixels describe the scene; recorded commands describe movement. The planner searches trial codes rather than copying the training example.
Whole component map · stages in path order
Solid = animation · dashed = I/O diagramEach row reads input → component → output. Select any box to locate it. “I/O only” opens its contract diagram; “Animated” opens the illustration.
NARRATED ANIMATION · KOKORO / GEORGEFrom pixels to a turn
Read each panel top to bottom: input → operation → output. Shapes omit single batch axes.
START HERE / ONE TRIP THROUGH FOUR ROOMSCan the dot reach the ring?
One try is called an episode. It starts when the simulator places the agent and sets a goal. A move is just one action inside that try.
SLOW MOTION / FIRST DECISION
Zoom in: why does move 21 differ in two runs?
EPISODE TIMELINE · WHY ZOOM IN HERE?
① Moves 1–10
Normal planner
② Moves 11–20
Same first target
③ Frame 20
Same scene · next target differs
④ Move 21 ↙
First different x/y request
Who produces move 21?
“21” is the move’s position in the recording—not a model input. Follow the arrows to see how the lower planner produces it.
A · SAVED TARGET
B · REPLANNED
What was scored?
At step 10, plan 792 scored 315.64: squared difference between its predicted final features and the goal features. The researchers picked it for this comparison, not because it won. That score belongs to the full saved plan, not to its first target plus later replanning. Actual observation images, requested commands and positions · g13-p43 · diagrams schematic · ◎ goal