MODEL ANATOMY / FOURROOM
Inside H-JEPA: follow one move
Ten recorded commands enter the model. Four numbers come out. What do they mean, where do they go, and why aren’t they a prediction?
Start at Recorded movements in the map above. There’s no chosen route through a doorway here, just a log of what the agent actually did. Follow those commands and you’ll find two different uses for the same experience: learning how movement changes a scene, and learning a compact vocabulary of movements to try later.
Five moves, then ten
Each command has two numbers: movement along x and y. Take commands 0 through 4. Packing them makes a block of ten numbers; it doesn’t change any of their values. The L1 action encoder turns that block into 192 features. Commands 5 through 9 go through the very same encoder. We now have two vectors in time order, not one vector made by adding them.
The L2 action encoder reads the pair together and compresses it into a four-number macro-action. This is a description of the moves. It is not a predicted position. To see the distinction in the map, select L2 state predictor and hover or focus each underlined [192] input. One comes from the state path. The other comes from the L2 action embedder, which prepares the macro-action for the predictor. Same shape, different information.
FIELD GUIDE / INTERACTIVE PLATES
Take a tensor apart.
Choose something to follow. Each plate opens here, at the point where a shape changes. The brackets describe dimensions, not measured values.
What makes a prediction right?
Consider one clip from a training batch. It carries seven camera images, five raw steps apart, and 35 movement commands. At L1, suppose the training window starts at frame 0. The predictor sees the encoded scene at 0 and the embedding of moves 0–4. Its job is to predict the state at frame 5. The answer already exists in the log: take the actual image at 5 and run it through the image encoder. Compare the two vectors, not two pictures.
L2 asks the question over ten moves instead of five. Each clip supplies three prediction–target pairs at each level. The prediction error is averaged across a batch of 128 clips; each level also has a term that discourages its state representations from collapsing. The two level losses are added before one backward pass and one optimizer update. Inspect a prediction and its target →
The queue isn’t a leaderboard
Watch the wire marked WRITE ROW. During training the encoder saves a detached copy of the first macro-action from each sampled clip. It starts on the first batch, without waiting for a good prediction. The queue holds the most recent 4,000 codes, not the best 4,000. A saved row has four numbers but no episode ID and no original commands. That’s why the explorer labels its historical source unknown rather than showing you an unrelated clip.
Now switch to inference. The planner reads down each column to estimate the mean and spread of one code coordinate. Those statistics help initialize candidate sequences. The illustrated bank contains 800 candidates, each with 11 four-number macro-actions. At that point none has been scored. Generating a plan worth trying and recognizing it as good are different problems.
H-JEPA paper · See §2.1 and Figure 2 for the two time scales. Return to the field guide ↑ · Standalone walkthrough ↗