V3 stage three (“C”) — the controller through the occlusion band

Technical log. Hardware: Apple M1, 8 GB, CPU. Interpreter python, every command from the repo root. The v3 VAE (runs/vae_v3/vae.pt) and all five v3 dynamics checkpoints were frozen, and no dataset was modified. Read docs/v3/00_v3_design.md §3-C first, then README_M3.md and README_FIX3.md for what stage two established and which checkpoint is the fair one.

The one-sentence question for stage three of v3: the paddle has to commit while the ball is behind the band — does it?

The answer, up front, and it is not the one the design predicted

No trained controller commits during the occlusion, and it does not need to, because the band as configured does not actually pose the problem. The memoryless upper bound — a privileged tracker that is switched off the instant the ball goes behind the band — scores 0.99 interceptions per floor visit against the full oracle’s 1.00, and 0.97 on the long-move bin where the design document predicted it would collapse. On the default band there is essentially nothing for object permanence to buy.

The geometric argument in the design had two holes, and they are both worth writing down because they are the kind of arithmetic that is easy to get wrong:

• The paddle is 0.26 wide. A “required move” of 0.40 is covered once the paddle’s centre is within 0.13 of the landing point, so only 0.27 of it has to be travelled.
• The ball is readable long before it is fully out. The design counted the ~10 frames between the band’s bottom edge (y = 0.28) and contact height. But the ball has radius 0.08, so a sliver of it is visible from the moment its centre reaches y = 0.36 — roughly 15 frames of run-up at a typical |vy|, and the stage-one result is that V encodes exactly that sliver correctly.

15 frames × 0.030 per frame + 0.13 of paddle = ~0.58 of reach with no memory whatsoever. The reach curve (reach_curve.png) confirms it empirically: wait_and_see sits at 1.00 all the way out to a required move of 0.6 and only falls to 0.80 beyond it, on 10 visits.

Where the bound does bite is the taller bands, which were collected for M’s memory-horizon test and turn out to be the only occlusion regime in this project where a controller has to remember. There wait_and_see drops to 0.86 (band 0.22–0.64, ~16 hidden frames) and 0.65 (band 0.16–0.70, ~25 hidden frames) against an oracle that stays at 0.99. And on those bands the fair controller does not close the gap: ctrl_v3 scores 0.78 and 0.53, i.e. below the memoryless bound at both heights.

Scoring the C-stage predictions from docs/v3/00_v3_design.md §4

1. Files added or changed

Artifacts: runs/ctrl_v3/, runs/ctrl_v3_tau1/, runs/ctrl_v3_ff/, runs/ctrl_v3_poshead/, runs/ctrl_v3_emerge/, runs/ctrl_v3_z_only/ (each with controller.pt, history.json, fitness.png, dream_vs_real.png), and runs/ctrl_eval_v3/.

2. Commands, in order, with wall clock

The six trainings ran three at a time with OMP_NUM_THREADS=2 (runs/_train_ctrl_v3.sh). Per-generation time is therefore 2–3× the solo figure, and the second batch was slower than the first (2 400 s against 1 000 s for the same 200 generations) because the machine had been saturated for half an hour by then — the same thermal/contention artefact README_FIX3 §2 saw.

COMMON="--vae runs/vae_v3/vae.pt --data data/v3/train data/v3/train_mix \
        --occluder --inputs zh --reward dense \
        --dream-steps 150 --popsize 32 --rollouts 16 --generations 200 --sigma0 0.5 \
        --real-eval-every 5 --real-eval-episodes 24 --real-eval-seed-base 7000 \
        --real-eval-metric interceptions"

# batch 1, concurrently
python -m wm.train_controller --out runs/ctrl_v3      --rnn runs/rnn_v3/rnn.pt    --temperature 0.5 $COMMON  # 1048 s
python -m wm.train_controller --out runs/ctrl_v3_tau1 --rnn runs/rnn_v3/rnn.pt    --temperature 1.0 $COMMON  # 1061 s
python -m wm.train_controller --out runs/ctrl_v3_ff   --rnn runs/rnn_v3_ff/rnn.pt --temperature 0.5 $COMMON  #  878 s

# batch 2, concurrently
python -m wm.train_controller --out runs/ctrl_v3_poshead --rnn runs/rnn_v3_poshead/rnn.pt --temperature 0.5 $COMMON            # 2408 s
python -m wm.train_controller --out runs/ctrl_v3_emerge  --rnn runs/rnn_v3_emerge/rnn.pt  --temperature 0.5 $COMMON            # 2387 s
python -m wm.train_controller --out runs/ctrl_v3_z_only  --rnn runs/rnn_v3/rnn.pt         --temperature 0.5 $COMMON --inputs z # 2357 s

# the evaluation: 16 policy rows x 150 episodes, two taller bands, GIFs   1334 s
python -m wm.eval_controller_v3 --out runs/ctrl_eval_v3

python -m pytest tests/ -q        # 116 passed

Wall clock from the first training launch to the last figure: 1 h 43 min.

Sample budget. Each training run: 15 360 000 dream steps (32 × 16 × 150 × 200) against 196 800 real steps (41 real checks × 24 episodes × 200), a ratio of 78 : 1 — the whole point of the world model, and the same ratio as v1 and v2. The evaluation itself spent 480 000 real steps in distribution (16 rows × 150 × 200) plus 120 000 on the taller bands.

3. Training

Two things to read off this table before any of the occlusion analysis.

• Dream fitness is anti-correlated with real skill across runs. emerge has the highest dream return (156.1) and the second-worst real score; ctrl_v3 has nearly the lowest dream return and the best real score. Dream return is a quantity in M’s own head and each M has its own head, so the column is not comparable across rows — that is the standard caveat — but it is a useful reminder that “the dream got better” is never evidence.
• ctrl_v3’s own within-run transfer correlation is −0.01. Its dream fitness rose smoothly for 200 generations while its real score peaked around generation 80 and then drifted down (see dream_vs_real_all.png, top left). This is the v1 failure mode, and it is why params_best_real — selected on the periodic 24-episode check — beats params_last_dream on almost every row below.

4. The headline: interceptions per floor visit by required move

The required move is |paddle_x at the frame the ball became fully hidden on its way down − the ball’s landing x|. 150 episodes × 200 steps, seeds 5000–5149, identical starts for every row. Full table, CIs and per-bin visit counts in runs/ctrl_eval_v3/summary.md.

Four readings, in order of how much they change the story.

1. wait_and_see is flat. 1.00 / 1.00 / 0.97. The split was designed to separate permanence from reaction, and on this band it cannot, because reaction is enough. Every comparison in the rest of the table is therefore a comparison of reaction quality, not of memory.
2. ctrl_v3_z_only does not collapse — it wins the long bin. 0.91, ahead of ctrl_v3’s 0.59. Its policy is close to “stand still (78 % STAY), then move hard when the ball appears”, which is exactly the optimal memoryless strategy given a 0.26-wide paddle, and standing still also keeps it near the middle of the box where the next ball is more likely to be. The design called this the sharp negative control; it is, and it fired the other way.
3. The privileged ceiling is not a ceiling. ctrl_v3_poshead (0.69) is below ctrl_v3 (0.85) everywhere except the short bin. Its M carries four times the hidden-ball ball_x information (R² 0.71 vs 0.17, README_FIX3) and the controller built on it is worse. Same for emerge (0.76): the two models with the most positional memory make the second- and third-worst zh controllers. Whatever a controller reads out of h on this task, it is not hidden-ball position.
4. The feed-forward floor is a real floor — 0.53 overall, 0.09 long — and the gap from it to ctrl_v3 (0.85) is the largest effect in the table. But that gap is not permanence: z_only, which also has no memory of the hidden ball, scores 0.80. What rnn_v3’s h is worth to this controller is velocity, the v1 quantity, and the ff model’s h is a one-frame function of z that cannot carry it.

stay scoring 1.00 in the short bin is a selection effect, not skill: a stationary paddle’s “short required move” visits are by definition the visits where the ball happened to come down on it.

5. The reach curve: where a memoryless policy actually breaks

† one visit. Counts per slice are in summary.md; anything under ~15 visits is noise, and the >0.6 column has 0–11 visits per row.

This is the table that kills the design’s premise. The memoryless bound is at the ceiling out to 0.6 of required move. ctrl_v3 is below it from 0.2 onward — it is not limited by what it knows, it is limited by how well it reacts.

6. Paddle motion during occlusion

Over the descending hidden runs (mean 9.6 frames on the default band), with the visits whose required move is under 0.15 excluded — on those, standing still is the correct behaviour and a low score would mean nothing.

The two “toward” columns disagree, and the disagreement is the finding. The raw fraction (the statistic the design asked for) is dominated by how twitchy a policy is: ctrl_v3_ff scores the highest of any trained row (0.514, well above the 1/3 chance line) purely because it is in motion on 99.5 % of hidden frames — when it moves, it is toward the landing point exactly 51.7 % of the time, which is chance. Conditioning on the frames where the policy actually moved inverts the ranking: ctrl_v3 0.62, tau1 0.83, ff 0.52, poshead 0.46. So the recurrent controllers do carry a weak above-chance signal about where the hidden ball is going, and the memory floor carries none — which is the result the design wanted.

And it does not matter. The displacement column is the behavioural bottom line, and every trained row is between 0.00 and 0.10 of the required move against the oracle’s 0.26. ctrl_v3 moves at all on 13 % of hidden frames and covers 3 % of the distance. No trained controller commits during the occlusion in any quantity that affects an outcome. What they do instead is stand still and react fast, which — §5 — is enough.

Note that oracle is not a ceiling for these columns either: it chases the ball’s current x rather than its landing x and is usually already inside its dead zone when the ball vanishes. The meaningful references are chance and wait_and_see’s structural 0.000.

7. Taller bands: the only regime where memory would pay

60 episodes each, seeds 6000+ and 6100+, bands the VAE, the RNN and the controller have all never seen.

Read the second row: as the occlusion lengthens, the memoryless bound finally separates from the oracle — 0.99 → 0.86 → 0.65 — which is exactly the effect the default band was supposed to produce and did not. That is where a controller with object permanence would show up, and none of ours does: ctrl_v3 is below wait_and_see at both heights, and on the tallest band the privileged poshead controller (0.61) is the best of the three trained rows for the first time — a hint that the position head buys something once the occlusion is long enough to need it, on 60 episodes and therefore not a claim.

How much of the degradation is V’s fault. The frozen encoder on frames from the datasets collected at those bands:

That is a 133× and 262× increase in reconstruction error. V was trained on one band position and treats a different one as an out-of-distribution object; ctrl_v3_ff’s flat 0.53 / 0.53 / 0.51 says the floor policy is insensitive to all of it, but for every row that reads z seriously the taller bands confound “longer occlusion” with “broken encoder”, and no split of this experiment can separate them. The taller-band numbers are therefore a lower bound on what these controllers could do with an encoder that had seen the band. Re-training V on mixed band heights is the obvious next step and was not done here.

8. The GIFs

• runs/ctrl_eval_v3/real_play_ctrl_v3.gif — seed 5100, chosen out of the 150 evaluation episodes as the longest required move (0.53) on which the paddle moved during the occlusion (4 of 8 hidden frames toward the landing point; 2 interceptions in the episode). This is a cherry-pick and it is the point of the figure: it is a demo that the behaviour exists at all, not a sample of the average episode — §6 is the average, and the average is 13 % of frames moved.
• runs/ctrl_eval_v3/dream_play_ctrl_v3.gif — the controller inside its own dream, decoded by V (best of 16 dreams by predicted contact, as in v1/v2). The dreamed ball is present on only 36 % of frames, against 84 % in the real episode next to it (measured by counting ball-coloured pixels in both GIFs). The band and the paddle are rendered crisply and the paddle moves sensibly; the ball disappears behind the band and frequently never comes back, or reappears somewhere unrelated a few frames later.

That last number is, I think, the mechanism behind everything in §6. CMA-ES cannot learn to commit early in a dream where the ball does not reliably come back out. README_M3 measured the same thing from the dynamics side — the dreamed hidden ball keeps its vertical direction on 60 % of stretches at τ = 0 and 50 % at τ = 1, a coin flip — and this is what that failure looks like from inside the controller’s objective.

9. τ = 0.5 against τ = 1.0

τ = 0.5 is better by 0.06 interceptions per visit (the CIs overlap: [0.80, 0.91] vs [0.74, 0.84]), which is consistent with README_M3’s recommendation to train at low temperature on a dream whose conservation is poor. The more interesting column is the last one: the τ = 1.0 controller learned to stand almost perfectly still during occlusions (2.3 % of hidden frames moved). In a noisier dream the ball’s hidden trajectory is even less predictable, so the optimiser finds the “wait” policy faster. It is the right response to a bad dream and it is the opposite of what the experiment was looking for.

params_last_dream vs params_best_real is worth one line: the gap is small on the two rnn_v3 runs (0.85 → 0.81, 0.79 → 0.81) and large on emerge (0.76 → 0.57) and on the long bin nearly everywhere. Selecting on 24 real episodes every 5 generations is buying 0.04–0.19 interceptions per visit, and that is the cost of the dream being exploitable.

10. Surprises and honest failures

10.1 The experiment’s premise was not checked before the experiment was built. The one measurement that decides whether stage three of v3 has a question at all — “how well can a privileged policy with no memory do?” — takes four minutes and needed no trained controller. It was not in the design, it is the first row of the results now, and it says 0.99/1.00. Every hour spent training six controllers was spent measuring reaction speed. The v1/v2 lesson was “the oracle must score ≈1.0 to validate the detector”; the v3 lesson is the memoryless bound must score < 1.0 to validate the task.

10.2 The feed-forward carried state was a zero vector, and it would have been invisible. wm.rnn’s feed-forward branch returned the incoming dummy h from forward, which is correct for everything README_M3 did (every probe reads parts["h"]) and wrong for the one thing stage three does (h[0][0]). Shapes match, training runs, the number comes out — and ctrl_v3_ff would have been a z-only controller under a different name, i.e. the memory floor would have been measuring the absence of z rather than the absence of memory. Caught by writing the test before the run. tests/test_controller_v3.py::test_feedforward_h_is_the_mlp_layer_not_a_zero_dummy pins it.

10.3 episode_masses would have read ball_visible as a mass. Same class of bug in the other direction: the v2 harness inferred “this rollout has masses” from states.shape[-1] >= 7, and v3’s seventh column is ball_visible. The derived per-episode speed 0.022 / m would have been 0.022 / 0 on any episode whose first frame had the ball behind the band, and that speed sets the floor-visit threshold — i.e. the denominator of the headline metric. Now the caller passes mass_from_color explicitly and a test asserts it.

10.4 The raw “toward fraction” ranks the memory floor first. §6. The statistic the design specified (chance 1/3, STAY counts as not-toward) is confounded by movement rate to the point of inverting the ranking. The fix is one extra column, not a different experiment, but the lesson is that a per-frame action statistic needs its denominator thought about: ff is in motion 99.5 % of the time and scores 0.51; ctrl_v3 is in motion 13 % of the time and scores 0.08, and ctrl_v3 is the one with the signal.

10.5 The required-move bins are policy-dependent. The required move is measured from this policy’s own paddle position, so the three bins do not hold the same visits for different rows: ctrl_v3 generates 46 long visits and z_only generates 70. This is conditioning on a post-treatment variable and it cannot be avoided without measuring the required move from a fixed reference paddle (which would then not be “the move this policy had to make”). All counts are reported per row; comparisons within a bin are between different visit sets and should be read with that in mind.

10.6 The taller-band test is confounded by V. §7. 133–262× reconstruction error means the generalisation result is not clean, and I did not find a way to separate the two causes with the checkpoints that exist.

10.7 ctrl_v3’s dream-to-real transfer correlation is −0.01, the worst in the project (v2’s best runs were 0.4–0.7). Its real score peaked near generation 80 and declined for the remaining 120 generations while dream fitness rose monotonically. 200 generations is too many for this dream; 100 would have given the same controller for half the compute.

11. What I would do next

1. Fix the task before fixing the agent. Either widen the band to the taller geometry (0.16–0.70, where wait_and_see scores 0.65) or narrow the paddle to ~0.10 and re-run exactly this suite. Both make the memoryless bound bite; the band is the better choice because the datasets already exist.
2. Re-train V on mixed band heights so the taller-band controller result is about the controller.
3. Give the dream a reason to keep the ball. The 36 %-of-frames number in §8 is the binding constraint on everything stage three could have shown. The emerge objective (README_FIX3) improved what h knows and made the dream worse; what is needed is the opposite trade.
4. Report the memoryless bound first, in every future stage. It costs minutes and it is the only thing that establishes there is a question.

The two questions the brief asked, answered plainly

(i) Does the fair controller act on memory during occlusion, and how far is it from the privileged ceiling and the memoryless floor?

Not in any quantity that matters. ctrl_v3 moves on 13 % of hidden frames and covers 3 % of the required move while blind, against the oracle’s 26 %. There is a weak signal — when it moves, 62 % of those moves are toward the landing point, against the feed-forward floor’s 51.7 % (chance) — so h is carrying something about the hidden ball and the memoryless model’s h is not. It does not reach behaviour. On skill, the ordering is inverted relative to the labels: ctrl_v3 0.85 > privileged poshead 0.69 > ff floor 0.53, with z_only at 0.80. The fair controller is above the “ceiling” and the gap it opens over the floor is a velocity gap, not a permanence gap.

(ii) Does the ceiling controller exploit permanence?

No. ctrl_v3_poshead — whose M carries hidden ball_x at R² 0.71 against rnn_v3’s 0.17 — scores worse on every skill metric (0.69 vs 0.85 overall, 0.23 vs 0.59 on long moves) and its paddle drifts away from the landing point during occlusions (displacement −0.036). Four times the positional memory in h bought nothing, and the only band where it looks best is the tallest one, on 60 episodes.

So: object permanence is not the binding constraint for this task. The memoryless upper bound scores 0.99 of the oracle on the band this stage was run on, which means there was at most 0.01 of headroom for permanence to buy before a single controller was trained. The binding constraints are reaction quality and — for the world-model controllers specifically — a dream that loses the ball on two thirds of its frames.