What the experiment tells us
Keep Gemini as the first reviewer candidate, but fix image association before another full evaluation. The primary held-out run met both framing and review gates in 17 of 24 core runs. It is not ready to become an automatic production acceptance gate.
- Obvious visible defects were detected. All eight held-out visible-defect reviews requested repair with the known crop. All eight offscreen-defect views were accepted. This supports reviewing the delivered crop while enforcing the narrated subject separately.
- The moving-crop failure was an image-label association problem. All six patio runs produced boxes one image late, starting with the source photo. Their crops retained the target adequately in only 51–57% of frames. A separate diagnostic put the label after each image and explicitly distinguished the source. Both new clean-patio runs retained the full target in every frame. This is a useful fix, but it was found after inspecting the holdout.
- A source-faithfulness check can still invent a disagreement. Both known-crop reviews of the clean entrance called its awning “floating,” at 95–100% confidence. That same gap exists in the source. One model-crop repeat also rejected it. The model's confidence is not a calibrated probability of an actual generation defect.
- Missing evidence produced inconsistent decisions. For the doorway absent from the only source photo, the four fidelity reviews produced two unresolved results, one confident rejection, and one acceptance. Plausibility must not substitute for evidence.
- Sampling limits are real. Every regular review missed the quarter-second kitchen defect because none of its images contained it. A denser diagnostic detected it. A thin defect entering during the fade was also missed when present in the known crop.
Next: adopt explicit image association, rerun against a fresh held-out moving scene, and then test cached real footage. Keep geometric crop validation outside the model. Preserve unresolved outcomes when evidence is missing, and evaluate confidence on human-labelled real examples before allowing automatic regeneration or acceptance.
A development fixture bug initially made an intended occluder pop into existence. Gemini correctly noticed it. The three original runs are retained and marked as flawed fixtures; corrected controls were rerun successfully. The tail prototype also exceeded its proposed peak-speed limit, so a separate slower version was rendered and checked. Neither correction overwrites the frozen primary results.
Watch the evidence
Choose a case and repeat. The players start muted and loop. Compare the clean source, landscape candidate, known crop, and Gemini's proposed framing. These are the actual tested clips, shown as smaller playback copies.
Ground truth and framing baselines
Green box: narrated target. Red box: planted defect at 3 seconds. These overlays were never supplied to Gemini. The brief-defect challenge is clean at this timestamp.
Every case, including failures
“Sample truth” describes what was visible in the exact images supplied to the reviewer. The brief-defect challenge also fails full-video fidelity despite having clean regular samples. Unresolved and skipped decisions remain visible.
| Case / repeat | Variant | Focus | Oracle review B | Combined C | Target retained |
|---|
The image-label correction, on the same moving scene
The primary crop trails the table because the boxes are associated with the preceding image. The two diagnostic runs use the same scene, sources, script, model, and schema, with explicit labels immediately after each image. Both retain 100% of the target in every decoded frame. These are post-holdout diagnostics.
A confident false alarm, visible in both images
Gemini claimed that the candidate's awning was detached while the source's was attached. The authored geometry is unchanged. Both images contain the gap. These close views let you inspect the disagreement.
Did a stable crop freeze the ending?
The final second starts from cloned source frames. The fixed crop is the intentional freeze control. The primary crop moves 12 source pixels through a 0.75-second fade; its smooth ramp exceeds the proposed peak-speed limit. A separate six-pixel correction below respects the continuous-curve limit before pixel quantization. The current pan is the production-filter comparison.
Follow-up checks
Method, spending, and limits
What ran
28 planned synthetic cases plus three preserved flawed development variants: eight scene/script setups × clean, visible-defect, and offscreen-defect variants, plus four challenges. Fifteen development runs include the three corrected controls, followed by 32 held-out runs across two repeats. A focuses from dialogue and raw frames; B reviews the author crop; C reviews the model crop if it is feasible.
Each clip is five seconds at 24 fps, 1920 × 1080 before framing. Regular samples use frames 0, 36, 72, 108, and 119. Candidate samples have a 1024-pixel long edge. Source stills retain native resolution (1920 × 1080, except the narrower insufficient-evidence photo). This differs from the plan’s proposed uniform image-size cap. Portrait crops are 608 × 1080 before scaling to 1080 × 1920.
The narration is newly written in the shape of 16 saved historical planner entries. The homes are simple projected shapes with depth and camera parallax. Only Gemini model calls are paid. Script planning, scene generation, narration, avatar, production storage, Trigger dispatch, and repair generation are mocked.
How results were scored
Author-only masks calculate subject and defect visibility across all 120 frames using the delivered crop coordinates. Framing passes when at least 95% of the naturally visible target remains in at least 95% of frames. Finding localization matches at least 25% of the smaller truth/prediction box area; unmatched findings are listed for inspection.
Review decisions require schema and semantic validity. A supported material finding at confidence ≥0.75 requests mock repair. Lower confidence, insufficient evidence, conflicting output, and impossible framing remain unresolved. Supplemental results never replace the first-response score.
The initial batch used 144 requests; four further requests tested the labeling correction under the approved $20 ceiling. Request identities preserve actual image hashes, raw clip hash, dialogue, timing, crop recipe, prompt, schema, and policy. The persisted ledger reserves spending before requests and records returned costs. SDK retries are disabled. Price-limited routing uses Google AI Studio, temperature 1, low reasoning, and a 4,096-token output limit.
This is a small, correlated synthetic test. It does not establish photorealistic accuracy, prove spatial reasoning, calibrate confidence, validate a video model's repair ability, or validate production Resume. The held-out source palettes differ, but the object vocabulary is intentionally simple. Repeated scenes are not independent homes.
All report media is synthetic and embedded in this public document. Playback copies use a 960-pixel long edge; full-resolution tested outputs and request evidence remain in the local experiment bundle. No customer data, credentials, or signed media links are published.
Original experiment plan · Tours planning home base · Model reference