Review the scene.
Repair defects.
Frame what matters.

Status: proposed implementation plan. Updated September 6, 2026. Code inspected at 03c75a95; deployed behavior and model quality have not been verified by this research.

Tours v2 should inspect each generated scene against its source photos, repair clear visual defects with a bounded regeneration, and frame the accepted video around the subject that matters. This is the home base for that work. The synthetic Gemini experiment is complete. The next step is to verify the image-label correction on a fresh moving scene, then evaluate cached real footage.

Completed synthetic Gemini experiment

Read the executed results with embedded videos. The user approved a revised $20 budget. All runs completed with 148 model requests costing $0.5643465. Results retain 47 scene/repeat runs, including three corrected development controls. Eighteen of 24 held-out core crops met the framing gate; 17 met both framing and review gates. Explicit labels after each image fixed the moving-scene crop in two separate diagnostics. Confidence, insufficient source evidence, sparse sampling, and real-footage quality remain unresolved before production integration.

Next experiment: cached real footage

Read the next experiment plan, with an editable local copy. Verify the corrected image labels on four fresh moving scenes, then target 24 cached scenes split by property into development and held-out sets. Human labels come first. Compare focus, reference-crop review, combined crop/review, and dense versus sparse sampling. Eligible footage and human review are prerequisites. This is a proposed experiment; no new model calls have run.

Original synthetic experiment proposal

The original proposed slice was the Gemini review experiment, with an editable local HTML plan. It uses realistic mock planner output, 28 controlled scene cases, and separate measurements for focus, fidelity review with known crops, and the combined flow. Everything outside the reviewer path is mocked. The original proposal capped execution at 144 requests or $5. The user subsequently approved $20; the completed results above include four additional labeling-diagnostic requests. This synthetic capability check precedes the cached real-scene experiment described below.

What we found in the current code

• OpenRouter returns landscape video in the Tours workflow, as confirmed by Josh. The current worker uses createDirectSceneClipProvider, which sends the authoritative photo with aspect_ratio: "16:9". Portrait framing happens locally after generation. Probe actual returned dimensions for crop geometry; the older adapter and comments are not evidence of portrait output.
• The default Kling O1 duration policy chooses 5 or 10 seconds. Output normalization defaults to 1080 × 1920 at 24 fps. Forty-five frames at that output rate is 1.875 seconds; raw provider frame rates can differ. Sample by elapsed time.
• Wide footage receives a smooth pan across the central 40% of the available crop travel. Direction comes from the scene ID. Footage without horizontal crop room uses a centered cover crop and, when needed to extend motion, centered zoom. Traditional photo scenes also have a separate crop/zoom path.
• A scene already has its own Trigger task, with queue concurrency 10 and one task attempt. Parallel scene work exists. A model reviewer can fit inside this flow without introducing an open-ended agent framework.
• The provider video is downloaded into scratch storage, normalized, then deleted. The normalized clip is stored as reusable. There is no visual acceptance gate on this path; reuse returns before new rendering.
• The current direct provider uses the positive prompt and authoritative photo. It does not forward secondary references or a negative prompt. Code comments record a prior movement regression with provider-specific negative prompts; that is a reason to test positive corrective instructions first, not proof that hallucinations are prevented.

Script-guided framing before fidelity review

September 6 discussion update: give the framing step the exact dialogue chunk spoken during this scene, with its timing, alongside the source photos and raw landscape footage. The focal subject should follow what the script is discussing. For example, a line about the fireplace should guide framing toward the photographed fireplace. Narration supplies intent; photos establish what actually exists.

Proposed comparison workflow: generate landscape → infer focus from dialogue and photos → plan/render portrait framing → inspect the visible result → accept, reframe, inspect more closely, or regenerate → validate final composition. Regeneration returns to focus planning and review. This is a candidate replacement for the original inspect-before-crop workflow below, pending the cached-scene comparison.

Under this proposed policy, a defect confined to footage that never appears in the delivered crop would not cause regeneration. Preserve the raw footage as context, but distinguish raw-frame observations from visible-output failures. Inspect the actual cropped frames against the source photos, with the exact crop transform and dialogue available. The framing pass is not itself a fidelity verdict.

Evaluate visibility across the full crop path, transitions, retiming, cloned tails, and the fade. A defect outside the crop at one timestamp may enter later. Do not move the crop away from the feature being discussed merely to obtain a pass. A new crop or changed dialogue invalidates the visible-output review and requires another check; a replay cannot reuse a verdict tied to a different window. Include cropped-artifact/recipe hashes and coordinate/time mapping in review identity.

For the experiment, compare both operation orders on the same labeled clips. Measure visible defects missed, unnecessary regenerations avoided, script-focus retention, review cost/latency, and tail motion. Use cropped sampled frames for quick experiments, then verify the actual rendered portrait result; sampling is not proof that every output frame is clean.

This adds an open visibility-policy decision alongside D1–D4: should production accept an otherwise usable scene when all identified defects remain outside every delivered crop? The recommended experiment assumes yes, provided framing still shows the intended feature. Record final approval and visibility coverage before enforcement.

Proposed scene workflow

Original inspect-before-crop baseline. Compare it with the script-guided crop-first option above before choosing the production order.

1. Resolve whether an existing candidate and current review can be reused.
2. Generate or resume the scene's provider job. Persist its accepted job ID and candidate identity.
3. Store the raw candidate, probe dimensions/duration, and extract timestamped frames.
4. Review the authoritative source photo, any relevant supporting photos, and the ordered frames. Return fidelity findings and a focus proposal in one structured response.
5. Apply a deterministic policy: accept, inspect an uncertain interval more closely, or regenerate once with specific corrective feedback.
6. For an accepted candidate, validate a crop plan and render the portrait scene. Check framing at the sampled timestamps and check duration with the existing machinery.
7. Mark only accepted output reusable, then continue the existing narration, joins, presenter, and final mux flow.

Review scenes concurrently, initially with a proposed limit of three reviewer calls across workers in the deployment. Give review its own shared queue/rate limit; a per-worker semaphore alone would multiply under load. Each call sees the complete sampled scene to preserve temporal context. Use an additional reviewer or denser sampling only for uncertain findings. Multiple agents looking at isolated frames would lose motion context and multiply requests.

Visual review contract

Use explicit sampled images through OpenRouter first. The existing chat types already support image_url; native video would require additional types and provider-specific verification. OpenRouter documents both image and video input, but supported models, formats, payload limits, and structured output must be checked for the chosen route. Native video remains a comparison candidate, not a prerequisite.

Start with a frame near the beginning, every 1.5 seconds, and the last decodable frame. This gives roughly 5 frames for a 5-second clip and 8 for a 10-second clip. Attach actual timestamps, sample IDs, and source-photo IDs. Sample at 0.5-second intervals around suspicious motion when needed, with a fixed total frame cap. Sparse sampling can miss brief defects; measure those misses against full-speed human playback.

Provide the intended scene focus or narration alongside the visual evidence so the reviewer can distinguish the feature being discussed from merely salient objects. Text is intent, not visual proof. The reviewer returns:

• verdict: pass, fail, or uncertain; schema and policy versions; candidate content hash; reviewer model and route.
• findings[]: defect type, severity, sample IDs/timestamps, affected region, source-photo reference, a concrete observed contradiction, and a 0–1 model confidence score.
• focus: subject description, normalized bounding boxes at sampled times, preferred framing, and confidence. Coordinates refer to the full raw video after orientation handling.
• repair: a short positive constraint grounded in the source photo, when a repair is justified. This is data for a controlled prompt builder, never executable tool instructions.

Look for changed walls, duplicated windows/doors, bent counters, object births/disappearances, and unstable geometry. A new viewpoint, expected occlusion, camera motion, or an area absent from all photos is not automatically a proven defect. Distinguish contradicted geometry from unsupported content; use uncertain when the evidence cannot decide. Supporting photos should only corroborate the same scene, not lend facts from another room.

Require strict JSON schema on a supported route and validate it locally. Invalid JSON, timeout, missing images, and unsupported routes are reviewer failures, not hallucination verdicts. Allow one bounded response repair or transport retry where appropriate, without restarting video generation.

The 75% rule and bounded repair

Treat 0.75 as an experimental decision threshold, not a measured probability. A model saying it is 75% confident does not establish 75% accuracy. Record scores in shadow mode and compare them with human labels before enabling paid repair.

Proposed initial policy, subject to decisions D2 and D4 below. This is not yet a complete production acceptance specification:

• Accept when there is no supported material defect and the review is valid.
• Consider regeneration when a material source contradiction has score at least 0.75 and cites visible evidence. Confirm a borderline result with denser frames or a second review before spending on regeneration.
• Keep uncertain, unavailable, and failed reviews distinct. During shadow mode they only record evidence. During enforced operation, unresolved review must have an explicit outcome. Whether further review can resume depends on its remaining durable budget; exhausted generation must not offer another budget-resetting Resume.
• Propose at most one quality regeneration per scene generation lineage, meaning two video candidates total across Resume runs; the lineage contract remains open in D2. Review the replacement too. Keep candidates and findings separately; never erase the original evidence.
• If the replacement still fails, stop that scene with a clear recoverable outcome. A traditional photo-motion fallback is a product decision to approve separately; do not silently change the selected presentation.

Example corrective instruction: "Keep the single window shown in source photo A unchanged throughout. Use a smaller forward camera move and keep the photographed wall visible." Preserve the original scene intent, source image, timing contract, and model unless an experiment explicitly changes them. Reduce movement when evidence points to the camera revealing unsupported geometry.

Separate a quality regeneration from retrying a network request. Persist each accepted provider job before polling and resume that job after polling/download failures. The existing direct adapter keeps the accepted ID in memory and retries transient HTTP responses, including submission. Resolve ambiguous acceptance and provider idempotency support before automatic paid repair; a new attempt key in our database alone cannot guarantee that an ambiguous POST did not create a job.

Focus-aware portrait cropping

Have the model identify the focal region; keep crop geometry and motion limits in deterministic code. Start with one stable crop per scene. If the subject moves enough to need tracking, use a small set of keyframes with smoothing and limits on speed and acceleration. Do not add a second arbitrary pan on top of provider movement.

For a wide source of width W and height H, the full-height portrait crop has width C = H × 9 / 16. A focal center fx in normalized source coordinates gives x = clamp(fx × W − C / 2, 0, W − C). In implementation, account for pixel aspect ratio, rotation, integer/even-pixel alignment, and vertical overflow on taller inputs. Prefer fitting the subject's box with margin over following a center point alone.

For a 1920 × 1080 landscape image, the portrait window is about 608 pixels wide. That is why a subject near the left third can disappear behind a centered crop. A crop cannot keep two widely separated subjects fully visible, and it cannot recover a subject missing from the raw video. Return an explicit framing limitation in those cases.

Compile the plan in the provider normalizer before information is discarded. Map source timestamps through the existing slowdown, trimming, and tail-extension rules. Keep the last useful subject region as the framing constraint during extended footage, while preserving bounded camera motion as described below. Preserve transition handles and the motion-through-fade ending. The transition policy currently shares the scene-ID pan direction, so reconcile that direction when replacing the pan. Check presenter overlap in the final composition using existing presenter placement; do not reposition the member's presenter automatically.

Recorded review constraint, September 6: a stable crop can reintroduce frozen endings. The current normalizer clones the last source frame with tpad=stop_mode=clone before applying pan or zoom. Replacing that motion with a fixed crop would expose the freeze.

Proposed tail rule, pending D1: for cloned tails, continue a smooth pan within the range that keeps the final subject box and its margin inside the portrait crop. Where pan has insufficient room, use a subject-centered zoom, capped at the existing 8% total zoom and reduced further by the subject-fit constraint. Bound speed and acceleration, blend continuously from the preceding crop path, and reserve enough motion to continue through the visual fade. Compute the feasible path before rendering; do not hit a boundary early and freeze. If neither motion is feasible, report an unresolved tail-motion constraint and require another duration strategy rather than silently accepting a frozen ending. Final numeric pan-speed and acceleration caps must be set and recorded in the crop experiment.

Acceptance test: decode consecutive frames across the transition into cloned footage and through the fade. Verify subject retention, crop bounds, continuity, and actual geometric movement. Compare pre-fade scene frames and the final mux; fading brightness alone must not count as motion. Include fixtures where pan room runs out, zoom would clip the subject, and the final output reaches black. Human playback must confirm that the ending keeps moving without a jump.

A crop-only correction should reprocess the retained raw candidate without buying another video. A no-room portrait crop should be a no-op or a framing limitation. Low-confidence focus retains the existing crop as an explicitly recorded fallback during the experiment. Traditional photo scenes can adopt the same focus contract in a later slice.

Where implementation belongs

Paths beginning with scenes/ are relative to lib/tours/rendering/. These are integration locations, not a request to rewrite the whole renderer.

Persist generation attempts separately from reviews and crop outputs. A review model change should re-review retained footage; a crop change should re-render retained footage without new generation; under crop-first acceptance it must also re-review the changed visible output. Old cached clips are unreviewed under the new policy, not automatically accepted. Decide a bounded retention period for rejected raw candidates. Reuse current asset/event metadata for the offline experiment; durable production attempts may need a focused schema extension after the repository contract is settled. Any added migration must run locally before implementation is called done, without an unapproved database reset.

Today's build sequence

Aim to finish the evidence, reviewer experiment, and crop comparison today, then decide how much integration the results justify. That is a useful first milestone; this research does not establish a one-day production delivery estimate.

Start instruction for the next implementation task: implement the cached-scene sampler and structured-review experiment, plus a crop comparison using the existing FFmpeg normalizer. Keep automatic regeneration disabled until review accuracy and durable job acceptance are demonstrated. Use app-owned cached inputs and keep customer media private.

Validation and operating limits

Test policy boundaries at 0.74/0.75/0.76, valid/invalid/uncertain responses, material versus cosmetic findings, and the two-candidate cap. Exercise crash after provider acceptance, ambiguous submission, resume after download failure, repeated Resume after lineage exhaustion, concurrent lineage spending, reviewer failure, cancellation, changed source-photo contents/supporting selection/narration, conflicting verdicts and findings, stale policy reuse, and a crop-only rerun. Verify no failed candidate reaches the accepted cache.

Render fixtures for left/right/center subjects, portrait/no-room inputs, oversized subject boxes, moving focus, slowdowns, transition handles, and extended tails. Check actual decoded frame bounds and duration, not only FFmpeg string snapshots. Judge real clips at phone size and full playback; automated checks do not establish visual quality.

Track false-positive and missed-defect counts against human labels, repair success rate, unresolved scenes, crop preference/retention, p50/p95 added latency, reviewer usage, and incremental provider cost. The initial 6–10 scenes are a smoke set; reserve additional unseen examples before broad enforcement. Model selection and numeric release thresholds remain decisions from that evidence.

With S generated scenes, review cost V per candidate and regeneration cost G per scene, a one-regeneration cap bounds initial review plus quality-repair spend at S × (2V + G), excluding originals, targeted extra reviews, storage, and transport retries. Set separate total limits for extra reviews, per-tour spend, and elapsed time. Measure actual usage and current route prices before setting dollar limits. Record internal provider spend separately from Profile Usage Credits; retries must not silently change the member's credit charge.

Rollback flags should separate review recording, crop application, and automatic regeneration. An emergency bypass must remain distinguishable from a passed review. Keep completed artifacts and evidence when disabling a feature.

Trigger.dev execution design

Research recommendation, September 6. Keep one scene controller per generation lineage and give it a bounded evaluate-and-repair loop. The model examines evidence and proposes a repair or crop; deterministic application policy decides whether to accept, gather more evidence, regenerate, or stop. This execution recommendation leaves D1–D4 open.

The task tree should preserve the current tour parent and scene parallelism:

1. The tour parent starts scene controllers and the existing avatar work through Trigger's supported batch-and-wait APIs.
2. Each scene controller loads its durable lineage and coordinates candidate submission, collection/sampling, review, and portrait rendering as child tasks.
3. A structured review result selects the next allowed action. A rejected candidate is a completed review with a quality finding, not immediately a failed tour.
4. Only accepted portrait output returns to the existing joins/final mux. The parent handles terminal product outcomes once repair/review budgets are exhausted.

Separate tasks where retry safety or resource requirements change. Provider submission gets its own guarded operation; polling cannot resubmit it. Collecting raw footage and sampling use FFmpeg; review uses a small worker and a shared reviewer queue; portrait rendering uses an appropriate media worker. Pass stored artifact IDs and hashes between tasks instead of video buffers or temporary paths. An identical crop retry should not buy another scene or repeat a completed matching review. Under the crop-first option, changing the crop requires a new visible-output review.

Use triggerAndWait for dependent children and batch.triggerAndWait or batchTriggerAndWait for parallel children, checking each result. Do not wrap Trigger wait calls in Promise.all. Ordinary independent network reads can still use ordinary JavaScript concurrency. Trigger documents this task composition and version-locking behavior in triggering and versioning.

Keep generation lineage, attempts, immutable evidence, acceptance, and spending limits in the database. Use explicit global-scope Trigger keys built from lineage/operation/input identity to deduplicate dispatch across Resume. Trigger keys expire, and failed runs release their keys, so a key alone cannot guarantee one paid provider submission. An ambiguous provider response must remain unresolved until reconciled. The existing avatar submission split is a useful local example, but its render-run key does not provide the proposed cross-Resume contract. See Trigger idempotency.

Use wait.for for intentional provider polling delays. Current cloud documentation says waits longer than five seconds avoid compute usage, but timed waits release concurrency only after checkpointing at 60 seconds. A 20-second poll wait therefore still holds its slot. More generally, checkpointed controllers and external provider jobs outlive active queue slots. Keep provider admission/spending reservations separate from Trigger queue concurrency, and enforce a wall-clock deadline independently of per-attempt maxDuration. See wait behavior, queues, and compute duration.

Trigger also ships chat.agent and task-backed AI tools. For this first workflow, ordinary tasks plus a bounded reviewer loop are sufficient. If later experiments show value in letting the model request closer frame samples or another specialist review, expose only those bounded task-backed tools. A long-lived chat session becomes relevant if a person should steer a tour conversationally while it runs. Human decisions can also use authenticated application actions plus Trigger wait tokens without adding a chat UI. See agent overview and wait tokens.

Trigger-specific agent features are real and optional here: chat.agent provides durable multi-turn sessions; AgentChat delegates to a sub-agent session; and the installed ai.toolExecute(task) adapter makes a Trigger task executable as an AI SDK tool. Their value depends on the decisions we let a model make, rather than the number of steps. A fixed focus/review/repair workflow can use ordinary tasks. If the model dynamically chooses to inspect another interval, compare a source region, or try another valid framing, task-backed tools become useful even inside this small part of Tours. Budget and acceptance policy remain application-owned. Agent features, sub-agents.

The Trigger.dev research note contains the version check, task boundaries, failure/recovery rules, and recommended first experiment. These are research findings and proposals; no Trigger tasks or production settings were changed.

AI SDK choice for visual review

September 6 evaluation recommendation: use the installed Vercel AI SDK 6 for the new focus/review model calls through the existing OpenRouter adapter, subject to a live route compatibility check. Keep Tours' acceptance policy and video-provider lifecycle explicit. This is a proposed scoped adoption, not a migration of existing planners or video generation.

The repo already has ai 6.0.39, Zod 3.25.76, and createOpenRouterAiSdkProvider, built on @ai-sdk/openai with OpenRouter attribution headers. Other app paths already use AI SDK structured generation. Tours' handcrafted chat.json<T> helper parses a JSON object then casts it to T; downstream Tours code separately validates and normalizes scene plans and repairs narration word limits. Those domain rules still have a purpose.

An offline check using the actual existing adapter and mocked HTTP responses confirmed strict json_schema serialization, image input encoding, acceptance of the expected object, and rejection of a confidence string with AI_NoObjectGeneratedError. An arbitrary scene ID still passed a schema that only required a string, demonstrating the need for explicit evidence constraints. Each case made one mocked request with maxRetries: 0. This proves local adapter behavior only; it does not prove live model/route support or review quality.

Use a narrow Tours vision-review interface so callers receive a validated review or classified error without handling SDK details. Preserve app attribution, request/response IDs, usage, effective settings, and sanitized provenance. Start with explicit SDK retry settings and separately budget any schema-repair/model retry. Trigger attempts, SDK transport retries, and deliberate review or video repair must not multiply unnoticed.

Evaluate the existing adapter first. Add the OpenRouter-specific AI SDK provider only if required routing controls, provider metadata, or cost details cannot be expressed and preserved reliably through it. Check peer versions before choosing a release; installing the latest provider is not evidence of compatibility with the installed AI SDK. No dependency upgrade is required to run the proposed first comparison. Keep asynchronous video submission/polling in its existing provider seam while its durable acceptance contract is addressed.

If later the model selects additional inspection or reframing steps, AI SDK tools can connect to Trigger tasks through ai.toolExecute. A separate general agent framework is not justified by today's fixed review flow. The AI SDK evaluation records alternatives, compatibility, and the offline experiment. Structured output documentation.

Review decisions before production integration

Review recorded September 6, 2026. All four decisions below are open. The review supports proceeding with human labels, the cached-scene reviewer, and crop comparisons. These decisions do not expand or block that first slice. Proposed defaults here are discussion inputs, not approved production policy. Resolve them before integrating the behavior they govern.

D2 detail: distinguish a render run from a generation lineage. In the current code, Resume enables asset reuse while creating another render run. Proposed counters and spend reservations therefore belong to a stable lineage, with atomic updates across concurrent workers and attempts. Resuming polling, downloading, review, or cropping must not reset generation spend. An exhausted lineage needs an actionable state, such as choosing a supported fallback or explicitly starting a new budgeted generation; the product choice is still open. A changed narration/focus can invalidate review without automatically granting fresh video attempts.

D3 detail: proposed review inputs include the raw video content hash, authoritative-photo content hash, selected supporting-photo content hashes and their roles/order, supplied narration and intended focus, complete rendered prompt/messages, actual sampled-frame hashes and timestamps, reviewer model/route/settings, schema, and policy versions. Asset IDs or expiring URLs alone do not identify evidence contents. Proposed crop-output identity includes raw candidate hash, validated crop plan, target dimensions/frame rate, duration/retiming/tail settings, and renderer recipe version. Changing narration must not reuse an old focus decision by accident.

D4 detail: the production table must explicitly cover no material finding, a supported material finding below 0.75 (including 0.74), at/above 0.75, unsupported or uncertain findings, verdict/finding conflicts, invalid responses, reviewer outages, and exhausted review or generation budgets. Proposed default: a material finding below threshold is unresolved rather than automatically accepted; a pass verdict cannot override a material finding; a fail verdict without supporting evidence cannot alone authorize paid regeneration. Give unresolved cases a bounded clarification path, then an explicit unresolved outcome when its budget expires. The exact bands, number of clarification calls, and member-facing outcomes remain decisions to make, not implementation discretion hidden behind the word "borderline."

Close each decision with the chosen rule, supporting experiment or product rationale, acceptance tests, and approval date. D1 is informed by crop playback; D4 by reviewer evaluation. D2 and D3 primarily need lifecycle and data-contract design rather than a model benchmark.

Decisions to make from the first experiment

• Which vision model/route catches the labeled defects at acceptable cost and latency? Does a second model improve disputed cases enough to justify it?
• Does a stable focal crop solve most examples, or is a moving crop necessary immediately?
• Is 0.75 useful after comparing scores with human labels, or should policy rely on different evidence/severity criteria?
• What happens after two failed candidates: resumable stop, member-approved traditional fallback, or manual support review?
• What per-tour spend/time ceiling and raw-video retention window should production enforce?

Recommended starting assumptions are 1.5-second sampling, three concurrent reviews, a stable crop, and one quality regeneration. These are proposals for the experiment, not approved production policy.

Sources and related work

The companion research note records current primary-source findings and model caveats. The existing Tours v2 work hub remains the record for prior reliability work and links to this feature plan. Its older production findings were not refreshed here.

• OpenRouter image inputs
• OpenRouter video inputs
• OpenRouter structured outputs
• FFmpeg crop filter