Listing Video Resilience & Recovery

Why this work is needed

The Listing Video Failure Catalog inventories the known failure contracts. Production review and code tracing show that the remaining risk is not only missing error codes. The pipeline needs a consistent disposition policy: when to retry, when to repair, when to reuse purchased assets, when to deliver a degraded result, and when failure is the only honest outcome.

Separate lifecycle status from output quality

Keep the existing lifecycle contract stable. Attach a second, additive quality classification and structured warnings to completed renders.

Recommended warning record

Warnings should be structured facts, not prose-only logs. A completed render can carry zero or more records such as:

{
  "code": "AVATAR_ENDED_BEFORE_TIMELINE",
  "stage": "final_assembly",
  "differenceSeconds": 1.42,
  "repair": "FADE_AVATAR_CONTINUE_SCENES",
  "customerVisible": false
}

Disposition framework

Every failure-producing boundary should answer the same questions in the same order.

Failure disposition matrix

This is the working policy. Individual error codes in the catalog should map to one of these dispositions.

Final-media salvage policy

Narration is the canonical timeline because truncating spoken content is usually the most damaging result. Duration difference alone should not be a hard-failure threshold.

Repair invalid plans before expensive generation

The script-planning prompt already expresses duration and word-count limits, but silent numeric clamping can hide a semantic mismatch: the text may still be too long for the clamped Scene. The image-to-video provider also has its own hard duration limit. The validator must operate on the raw plan and on the final provider request duration, including Transition handles.

Repair-call contract

• Preserve every Scene id and ordering.
• Return the same strict JSON schema.
• Change only invalid Scenes unless a total-timeline constraint requires more.
• Include machine-readable issue codes and actual/allowed values.
• Validate spoken read-time, not only the declared number.
• Validate provider duration after incoming/outgoing Transition handles.
• Allow deterministic rounding only inside a documented numeric increment.
• Fail before provider generation if the repaired plan remains invalid.

Retry, resume, and OOM strategy

Retries belong at the smallest stage that can recover without repeating unrelated work or repurchasing provider output.

Attempt versus terminal ownership

A child task may report attempt telemetry, but it should not mark the whole render run terminally failed while its retry policy can still recover. One coordinator—or a watchdog when the coordinator dies—owns the terminal transition and failure-email enqueue.

Same-run recovery versus fresh customer renders

Recovery reuse must be scoped to the same render run. A customer choosing “Generate” for a fresh version should continue to receive fresh provider generation. A worker retry within that run should reuse every successfully generated asset and provider job.

Customer communication policy

Expose information when it helps the customer resolve the problem. Keep implementation diagnostics internal.

Implementation roadmap

Build the disposition foundation before adding isolated retries. Otherwise retries, failure emails, and database status can disagree.

1. Outcome and warning foundation

   Add a quality classification and structured warning records without changing the existing completed lifecycle semantics. Define one terminal-state owner.

2. Pre-spend plan validation and repair

   Validate raw LLM output, read-time, Transition handles, and provider limits. Add one constrained repair pass and make silent provider-duration caps unreachable.

3. Final-media salvage

   Probe narration, joined scenes, Presenter, and output; apply direction-aware trim/extend/fade; re-probe; record repaired or degraded quality.

4. Provider checkpoints and same-run reuse

   Persist OpenRouter and HeyGen job identities immediately, resume polling/import, and ensure retries never repurchase completed work.

5. Stage-local retry and OOM escalation

   Classify transient errors, add bounded backoff, retry FFmpeg OOM on larger machines, and remove attempt-level terminal database writes.

6. Watchdog and reconciliation

   Detect stale runs, inspect child/provider/asset state, resume or reconcile completion, then mark true failure and enqueue notification if recovery is impossible.

7. Customer and support surfaces

   Verify failure email in a controlled environment, add provider-credit guidance, show degraded-output notices, and provide a safe support reference.

8. Operational feedback loop

   Report frequency, provider spend exposure, recovery rate, duplicate-job prevention, degraded-delivery rate, and terminal failures by normalized disposition.

Required production-shaped validation

Unit tests alone are insufficient for process death, provider resumption, and FFmpeg timing behavior. Each high-risk policy needs a failure-injection scenario.

• Invalid duration plan is repaired before TTS or video generation.
• Still-invalid repaired plan stops with zero downstream provider calls.
• Malformed model JSON receives only the bounded configured retries.
• Provider credits exhaustion is never retried automatically.
• Scene provider completes, worker OOMs, and retry does not submit a second paid job.
• Presenter provider completes, worker restarts, and existing video id resumes.
• Final FFmpeg OOM retries on a larger machine using persisted assets.
• Scene timeline longer than narration is trimmed and re-probed.
• Scene timeline shorter than narration is extended without truncating speech.
• Presenter ends early and the output is delivered with a warning.
• Corrupt final output remains a terminal failure.
• Final asset exists but completion update fails; reconciler marks it completed.
• No failure email is sent during a recoverable attempt.
• Exactly one failure email is sent after terminal exhaustion.
• Ready email and download work for standard, repaired, and degraded completions.

Success measures

Decision log and open calibration