Listing Video Resilience: Code & Contract Rollout

What the code says today

This plan is grounded in the current repository contracts and the previous production failure analysis. It deliberately keeps the existing lifecycle values because completion currently controls credit settlement, result ownership, download, UI readiness, and email delivery.

Target architecture

State ownership rule

attempt failure  -> append event + classifier disposition
retry scheduled  -> run remains running
asset recovered  -> continue pipeline with same run
usable result    -> completed + quality + warnings
recovery exhausted and unusable -> failed + failure_code + one email

Additive persistence and API contracts

Deploy schema before code that reads or writes it. Keep all additions backward compatible during the rollout.

1. Extend tour_render_runs

type TourRenderCompletionQuality = "standard" | "repaired" | "degraded";

type TourRenderWarning = {
  code: TourRenderWarningCode;
  stage: TourRenderStage;
  severity: "info" | "warning";
  customerVisible: boolean;
  details: Record<string, string | number | boolean | null>;
};

2. Add tour_render_provider_jobs

This is the durable boundary between “the provider may have accepted money/work” and “AiM has imported an asset.” It should be service-role only: provider job IDs and diagnostics are operational data, not a browser contract.

tour_render_provider_jobs
  id, run_id, project_id, scene_id?
  stage, provider, operation_key
  request_fingerprint_hash
  provider_job_id, polling_url?
  status: submitted | in_progress | completed | imported | failed
  result_asset_id?
  provider_metadata jsonb
  last_polled_at, created_at, updated_at

unique (run_id, operation_key)
unique (provider, provider_job_id)

Security: enable RLS and create no user-facing policy. Use the service role through repository methods. Do not persist signed output URLs longer than necessary; refresh them from provider status when possible.

3. Separate fresh generation from recovery reuse

// Always checked first for the current run, regardless of fresh-render options.
findRunRecoveryAsset({ runId, projectId, kind, sceneId, fingerprintHash })

// Checked only when the customer's cross-run reuse option allows it.
findReusableAsset({ projectId, kind, fingerprintHash })

4. Normalize failures before customer mapping

type RenderFailureDisposition =
  | { kind: "retry_same_machine"; retryAt?: Date }
  | { kind: "retry_larger_machine" }
  | { kind: "repair"; strategy: RenderRepairStrategy }
  | { kind: "customer_action"; code: TourRenderFailureCode }
  | { kind: "terminal"; code: TourRenderFailureCode };

classifyTourRenderFailure({ error, stage, attempt, checkpoint }): RenderFailureDisposition

The same low-level code may have different dispositions by stage and checkpoint. A timeout before a provider accepted a job is retryable; an ambiguous timeout immediately after create requires reconciliation before another submission.

5. Backward-compatible response

{
  "status": "completed",
  "result": { "assetId": "...", "quality": "degraded", "warnings": [...] },
  "error": null
}

{
  "status": "failed",
  "result": null,
  "error": { "code": "PROVIDER_CREDITS_EXHAUSTED", "message": "..." }
}

Older clients continue to work from lifecycle status and result asset. New clients can surface quality and safe recovery guidance.

Provider checkpoint and resume contracts

Operation key examples

scene-video:{runId}:{sceneId}:{fingerprintHash}
avatar-video:{runId}:{avatarFingerprintHash}
voiceover:{runId}:{voiceFingerprintHash}
joined-scenes:{runId}:{joinedFingerprintHash}
final-video:{runId}:{finalFingerprintHash}

Provider checkpoint writes and asset records must be idempotent under these stable operation identities. A unique-constraint collision should load the winner, not fail the render.

Trigger.dev retry and terminal ownership

Trigger.dev supports task-level idempotency and caching successful child results across parent retries. It also detects OOM from Node, the overall process, and child processes such as FFmpeg, and can retry on a larger machine. Those capabilities become safe after the persistence work above.

Why a watchdog is mandatory

Trigger.dev documents that onFailure fires only after normal retries are exhausted and does not fire for crashed, system-failure, or cancelled statuses. It also documents that max-duration termination skips cleanup, onSuccess, and onFailure. Therefore email and terminal state cannot depend exclusively on task hooks.

watchdog every 5 minutes
  query queued/running runs older than stage-specific threshold
  acquire advisory/row lease
  inspect Trigger root status and provider checkpoint
  active/waiting          -> leave alone
  Trigger complete        -> reconcile persisted final asset and mark completed
  crashed/system/timeout  -> trigger idempotent recovery coordinator
  recovery exhausted      -> finalize failure + enqueue one email
  unknown                 -> event + alert; do not guess terminal state

Planner validation and media salvage

Proposed timeline policy

Extract this work into focused modules such as final-render/timeline-policy.ts, final-render/timeline-repair.ts, and their tests. Do not add more responsibility to the current 1,100-line final-render.ts.

Dependency-ordered implementation slices

Each slice is deployable and reversible. The ordering is a safety contract: later behavior must not be enabled before its prerequisite persistence and observability exist.

1. Contract foundation and shadow classification

   Add warning/failure/quality TypeScript types under lib/tours/rendering/resilience/. Implement a pure classifier and structured events in shadow mode; do not alter retry or terminal behavior.

   Exit Known catalog errors classify deterministically in unit tests; unknown errors remain unknown/internal, never guessed as customer-caused.

2. Additive database and repository contracts

   Add run quality/warnings/failure code plus tour_render_provider_jobs. Update repository selects/mappers/types and response contracts with safe defaults. Add SQL constraint/RLS tests and mapper tests.

   Exit Old and new rows both read successfully; no UI behavior changes; provider job table is inaccessible to user sessions.

3. Centralize terminalization and align Trigger status

   Create finalizeTourRenderFailure() with compare-and-set semantics and an email idempotency key. Remove terminal writes from scene/avatar/final children. Make the root explicit maxAttempts: 1 and throw after genuine terminal failure.

   Exit Attempt failures leave the run active; one terminal transition and one failure email occur under concurrent calls; Trigger and DB terminal states agree for ordinary failures.

4. Same-run asset recovery and child idempotency

   Add findRunRecoveryAsset() and call it before every paid or expensive stage. Add stable Trigger child idempotency keys to scene, avatar, joined-scenes, final-video, and email work. Keep existing cross-run fresh-render behavior unchanged.

   Exit Retrying a run reuses its own completed scene/assets even when reuseExistingAssets:false; a previous run is never borrowed in fresh mode.

5. Provider job checkpoint and resume

   Refactor OpenRouter scenes to submit/checkpoint/poll/import. Add immediate HeyGen checkpoint callbacks and resume paths for avatar and combined narration. Treat unique-key races as load-existing.

   Exit Injected crash after provider submission resumes by provider job ID without a second create call; injected crash after asset import reuses the same-run asset.

6. Enable classified retries and OOM escalation

   Use task catchError for skip/retry decisions and retry.outOfMemory.machine for scene, avatar, and final assembly. Limit retries by operation and provider; do not retry permanent credit/auth/unsupported-input failures.

   Exit OOM fault injection retries on the intended larger machine and proves zero duplicate provider creates. Permanent failures skip retry immediately.

7. Plan repair and media salvage

   Add pre-spend semantic validation plus one bounded LLM repair. Extract FFmpeg timeline policy/repair, return structured warnings, and complete usable repaired/degraded results.

   Exit Unsupported plans never reach providers; small and moderate media mismatches yield validated outputs; unusable outputs still fail honestly.

8. Watchdog, customer surfaces, and production canary

   Add scheduled reconciliation with leases and Trigger status checks. Add failure email code-to-copy mapping, degraded Listing-page notice, support context, and metrics. Roll out provider resume, retries, and salvage behind independent flags.

   Exit Crashed/system/max-duration runs are reconciled; no duplicate emails; canary metrics meet the gates below before full enablement.

Verification and failure-injection matrix

Test layers

Rollout controls and production gates

Promotion gates

Customer and support contract

Customer-facing messages should communicate outcome, responsibility, action, and money/credit implication—without raw stack traces, provider payloads, model responses, or internal machine details.

Email dedupe should be a durable operation key such as render-failed-email:{runId}. If email delivery itself fails, the render remains terminal and the notification task retries independently.

Decisions to lock before coding

Research sources and related documents

• AiM Tours Listing Video Resilience & Recovery — governing product policy.
• AiM Tours Listing Video Failure Catalog — known thrown errors and customer mapping.
• Trigger.dev Machines and OOM — OOM detection, machine presets, and larger-machine retries.
• Trigger.dev Idempotency — task-level dedupe and parent-retry child protection.
• Trigger.dev Durable Execution — checkpoint/resume and cached child results.
• Trigger.dev Errors & Retrying — catchError, conditional retry, and skip behavior.
• Trigger.dev Task Lifecycle — retry defaults and hook coverage limits.
• Trigger.dev Max Duration — lifecycle hooks skipped on forced duration termination.
• OpenRouter Video Generation API — asynchronous create/status/content workflow.
• HeyGen Create Video — create response and video job identity.
• HeyGen Video Status — status polling and output retrieval.

This document proposes contracts and rollout order. It does not represent code already deployed.