Prove the next implementation boundary
The first experiment supports the architecture. It does not establish production reliability: 17 of 24 held-out core runs met both framing and review gates. A labeling change fixed the moving crop in two later diagnostics, while false alarms and inconsistent treatment of missing evidence remained. Executed experiment report
Decision this experiment can support
Build a crop pilot behind a feature flag and a reviewer that supplies findings for human review. Determine whether real scenes benefit enough to justify those integrations.
Decision it cannot support by itself
Let model confidence automatically approve customer videos or purchase regeneration. Even a perfect small test needs broader real-world evaluation before that decision.
Gate 1. Test new motion before reusing the fix
Create four new synthetic clips with known target geometry: leftward motion, rightward motion, depth/parallax, and partial occlusion. Use a different room layout and target placement from the earlier patio. Do not reuse its video or change its colors and call it a new holdout.
Run focus twice per clip with the corrected packaging: the source is explicitly identified as a source, each image's label follows that image, and candidate frame IDs are unambiguous. Keep the model, schema, prompt, and sampling fixed before the first request. This gate takes eight calls.
Exit criterion: every frame ID maps to the correct candidate image, the intended target is selected in all eight runs, and each feasible crop retains at least 95% of the visible target in at least 95% of decoded frames. No invented IDs, unbounded crop paths, or silent center fallback. If it fails, diagnose it and create a new holdout before claiming the fix is verified.
Use known masks here. On real footage below, human annotations are an imperfect reference and must be reported as such.
Gate 2. Assemble a real cohort before choosing prompts
Target 24 scene packets from six distinct properties. Use eight scenes from two properties for development; reserve 16 scenes from four different properties for the held-out evaluation. Keep every scene, alternate render, and near-duplicate source from a property in the same split.
| Required per scene | Why it is needed |
|---|---|
| Raw landscape candidate | The actual provider output, with verified scene identity and dimensions. A finished portrait render cannot test a new landscape crop. |
| Exact source photos | The original source and any selected supporting photos, with content hashes and provenance. Do not add unrelated listing photos to make a verdict look better. |
| Saved dialogue and timing | The spoken scene chunk, its intended duration, and any existing retiming, extension, or fade recipe. Preserve the saved text for the real-footage test. |
| Evidence of reuse | Import through the Inspector/workbench manifest, then snapshot the inputs. Do not infer a scene relationship from filenames or substitute an edited final video. |
In the held-out split, aim for at least six human-confirmed clean scenes and six with material visible defects. Across the full cohort, also include four with defects outside a feasible script-preserving crop, four moving subjects, and four with insufficient source evidence or infeasible framing. Categories can overlap. Deliberately include unchanged architectural details that might look implausible when enlarged.
Choose from eligible caches before looking at model judgments. Log the selection and exclusions. A deliberately enriched test set measures behavior in these categories; it does not estimate how often production videos have defects.
Availability gate: two imported historical packets are currently available, with saved planning data and media inventories. Their eligible raw clips and distinct property identities still need auditing. Six eligible properties are a target, not a confirmed inventory. If only those two packets qualify, run an exploratory pilot and report the coverage gap instead of presenting it as an independent holdout.
Make the human answer key first
Before any real-scene model results
Josh reviews each source set, dialogue chunk, and full candidate clip. For every scene, record the intended subject, whether the source establishes it, whether a portrait crop is feasible, and any material departures from the sources.
Mark each defect's time interval and region. Distinguish a supported difference from an unphotographed area, ordinary occlusion, or an odd-looking feature already present in the source. "Cannot tell from these photos" is a valid label.
For feasible scenes, author a reference crop path and target boxes at half-second intervals, endpoints, and motion/defect boundaries. Inspect the entire clip between annotations, including the tail and fade.
After crops are rendered
Compare the current production pan, fixed center crop, and proposed crop with their identities hidden and ordering randomized. Ask whether each preserves the narrated feature, whether movement feels stable, and which result is preferable.
Have a second human independently label the 16 held-out scenes when available. Resolve disagreements before opening model results, or retain an uncertain label. If only one reviewer is available, label the results as a single-reviewer pilot rather than inventing agreement statistics.
Budget roughly 45–60 minutes for the first review of 24 scenes, then a shorter blind crop comparison. These are planning estimates, not measured review times.
Human boxes establish retention at annotated times; a full playback check catches failures between them. Do not label interpolated boxes or Gemini's own boxes as ground truth for every decoded frame. Evaluate geometry, crop bounds, and tail behavior on all decoded frames separately.
Four measurements on the same scene
Reuse the existing AI SDK/OpenRouter adapter and local workbench. Start with the same Gemini model ID and settings used in experiment 01, verifying route availability and pricing before execution. Use explicit after-image labels in every arm. Keep source-image resolution consistent across arms; resize candidate evidence to a 1024-pixel long edge and record actual dimensions.
| Arm | Input and output | Question answered |
|---|---|---|
| A · Focus | Source photos + exact dialogue + landscape frames at 0.5-second intervals and endpoints → target boxes → local crop path. | Does the corrected evidence format locate and preserve the narrated feature on real footage? |
| B · Review alone | Source photos + dialogue + dense samples from the human reference crop → findings and sufficiency judgment. | Can the reviewer recognize real differences when framing is already reasonable? |
| C · Combined flow | The same review prompt and dense schedule, using the crop produced by A. | Does focus followed by visible-output review produce a useful result? |
| D · Sampling comparison | The exact crop from C, reviewed with 1.5-second intervals and endpoints. | Which differences disappear when we supply fewer images? Crop and prompt stay constant. |
Use elapsed time and actual decoded timestamps, not hard-coded five-second clips or assumed frame rates. Add samples at known crop/retime boundaries, tail start, and during the fade in both review schedules. Deduplicate timestamps. Even a half-second schedule can miss a shorter event; measure that against full-video human labels.
Each reviewer call starts fresh and receives no previous model explanation or human answer key. The human reference crop is not automatically perfect. If no feasible crop exists, skip dependent model-crop reviews and score an explicit unresolved outcome; a limited diagnostic crop cannot count as a successful framing solution.
Preserve motion through extended tails
Use the corrected smooth tail recipe from experiment 01. Limit continuous pan speed to 2% of crop width per second and choose travel from the available subject-preserving range and tail duration. Blend position and velocity at the join; verify acceleration and decoded pixel steps. If there is no room to keep moving through the fade while retaining the subject, return an unresolved tail constraint. Freeze the numeric acceleration cap during development before scoring the holdout.
Keep decisions in a separate controller
Missing evidence, unresolved focus, infeasible framing, invalid references, or conflicting judgments must prevent automatic acceptance. Carry unresolved reasons between stages; a later "accept" must not silently erase an earlier evidence gap. Record confidence and the existing 0.75 policy as a shadow decision: what the system would have done, with no regeneration request sent.
Preserve the full finding, evidence references, confidence, and disagreement. Compare threshold outcomes against the human labels, but do not tune a threshold on the holdout and report the tuned result as held-out accuracy. A model vote is not independent human confirmation.
What would justify building the pilot?
| Measure | Proposed decision gate |
|---|---|
| Framing quality | At least 90% of feasible held-out scenes preserve the correct subject, with ≥95% target-box retention at ≥95% of human-annotated times and a passing full-clip review. Failures must be explicit; an accepted crop cannot omit an essential narrated feature. |
| Benefit over today's crop | Humans prefer the proposed crop on a majority of edge/moving scenes, with no loss of essential subjects on centered controls. Show wins, ties, and regressions by unique scene and property. |
| Reviewer usefulness | Target ≥90% precision and ≥90% recall for material visible findings on evaluable scenes, with at least six positive and six clean examples in the held-out cohort. Score scene-level defect decisions separately for B, C, and D against what is visible in that exact output. An unresolved positive remains a missed detection for recall. Report raw counts, unresolved rate, and unsupported findings. Insufficient category coverage makes this inconclusive. |
| Missing evidence | No insufficient-evidence or infeasible-framing challenge becomes an automatic acceptance in the controller. Report unnecessary abstentions on supported clean scenes so "always unresolved" cannot pass as a useful reviewer. |
| Sampling and repeats | Report dense versus sparse misses on identical crops, plus agreement across the two held-out repeats. Count repeated predictions separately from independent scenes; do not double the apparent sample size. |
| Ending and motion | No out-of-bounds crop or subject loss on extended tails. Check the bounded continuous motion recipe and decoded-frame movement through the fade; fading brightness alone is not motion. Record quantization and held frames, and use a human playback check for visible stepping. |
If cropping passes, review does not
Proceed with a human-reviewed crop pilot. Keep the fidelity model advisory and use the failures to design the next evaluation. The two capabilities do not need to ship together.
If both pass
Build the feature-flagged crop flow and review findings interface. Run in shadow mode on a broader representative sample before considering automatic acceptance or paid repair. This small enriched cohort cannot establish a safe autonomous threshold.
Execution order and a bounded budget
- Build the evidence contract and audit the cache. Snapshot exact assets and exclusions. Fix image association, dynamic timing, and uncertainty propagation in the isolated experiment code.
- Pass the fresh motion gate. Eight focus calls across four new clips and two repeats. Cache selection and human labeling can proceed independently.
- Run eight development scenes. A–D once per scene. Inspect mistakes, then freeze all prompts, schemas, sampling, settings, and controller rules.
- Run the 16 held-out scenes twice. Apply the frozen flow. Keep first responses, skipped branches, and failures. Score each repeat separately against the same unique-scene denominator; report a scene as consistently passing only when both repeats pass. No prompt revisions inside the scored holdout.
- Evaluate, inspect disagreements, and report. Supplemental requests remain separate. Each report row links the actual source set, dialogue, raw clip, baselines, reference/model crops, human labels, and model evidence.
| Request allocation | Maximum planned calls |
|---|---|
| Fresh motion verification | 4 clips × 2 repeats × focus = 8 |
| Real development | 8 scenes × A–D = 32 |
| Real holdout | 16 scenes × A–D × 2 repeats = 128 |
| Capability checks and follow-ups | Up to 16, including explicit retries and supplemental evidence |
| Total ceiling | 184 requests; skipped dependent arms reduce the total |
Plan to stay inside the remaining balance of the previously approved $20 cumulative ceiling. Experiment 01 used $0.5643465, leaving $19.4356535. Retain that spending history rather than starting a new $20 ledger. Reserve liability before dispatch, allow at most three concurrent requests, disable SDK retries, and stop on exhausted budget or unaccounted cost. The request count is a maximum, not a cost estimate.
No new model calls occur while preparing this plan. When executed, planner, video generation, narration, avatar, production database writes, uploads, and repair generation remain mocked or bypassed. Run current crop/normalization functions through isolated adapters. Trigger lifecycle, cross-Resume spending, and paid regeneration need their own integration checks afterward.
Deliverable: a private HTML report with playable real clips and blinded crop comparisons, plus immutable inputs, human labels, every model response, scored results, and an exact spending ledger. A public summary may contain aggregate findings, but real property media and identifying data remain in the private workbench.
Evidence and open prerequisites
This plan follows the completed synthetic experiment and the Tours v2 planning home base. Local inspection confirmed the corrected labeling diagnostic and two imported packet manifests. It did not establish eligibility for the proposed six-property cohort.
The outstanding prerequisites are enough eligible cached scenes, a human labeling session, and a frozen implementation of the corrected evidence contract. The numerical gates above are proposed pilot criteria, not validated production standards.