Genace
Sign inGet started

generation job dead letter queue: replay the next action, not every POST

A dead-letter queue identifies work your consumer could not process successfully. It does not prove that a generation was never accepted. A message can arrive there after the remote render succeeded but local persistence or delivery failed. Replay should begin with the durable operation record and choose the next appropriate action for that phase.

Build an operator review record

Capture the message ID, application operation ID, action type, attempt count, last safe error category, and known remote job ID. Keep secrets and signed URLs out of the queue body where possible. A message should refer to restricted durable data rather than duplicate a complete prompt and credential context.

Before redrive, compare the message with current application state. The operation may already have completed through another observer. Stale queue data must not overwrite a newer terminal result or republish an asset that is already available.

prepared + confirmed no dispatch -> review allowance, then dispatch
accepted + known job ID -> observe that job
succeeded + copy incomplete -> retry artifact delivery
failed + refund unresolved -> inspect accounting
submitting + no reliable outcome -> manual reconciliation
already delivered -> acknowledge as obsolete

This is an application recovery matrix, not Genace’s state enum. Preserve the distinction in dashboards so operators do not assume a local delivery failure changed the remote render.

Restrict replay permissions and scope

Start with one reviewed operation, not a bulk redrive of every message. Check the selected model, specification revision, current user authorization, and remaining budget before any action that can create work. A replay tool should display whether it will read status, copy an existing output, or submit a new generation.

Known job IDs make observation recovery straightforward: GET the same identifier and inspect state, nullable outputs, error, and credit fields. A failed submission envelope can provide error.job_id even though the POST returned HTTP 502. Preserve that reference rather than classifying every unsuccessful POST as unsubmitted.

Do not convert uncertainty into automatic retries

If the previous consumer may have sent the request but lost its response, queue redrive cannot resolve the remote acceptance interval. There is no published creation idempotency support in the generation route. A new render must be an explicit decision after reconciliation, not a default effect of moving a message between queues.

Likewise, manual reconcile-jobs --fail changes the Genace job state without issuing a refund. Do not use a failed state alone as evidence that financial repair finished. For the explicit submission-failure marker, GET can advance pending compensation; unrelated unknown historical jobs need separate investigation.

Verify replay before production use

Create offline fixtures for all recovery phases and assert the selected action. Replay an accepted message twice and confirm no creation request occurs. Replay a delivery failure and verify the original output is reused. Test an obsolete specification and an already-completed operation to ensure both are rejected or acknowledged safely.

Maintain an audit entry for the operator, chosen action, and resulting local revision. That record supports later accounting without filling logs with sensitive payloads.

See queue visibility, failed response handling, and output storage.

Continue with production operations

Where these facts come from