Genace
Sign inGet started

generation queue visibility timeout: lease short actions, not entire renders

A queue visibility timeout controls when a received message may be delivered again. It is not the maximum lifetime of a generation job. A render can continue after a worker’s lease expires, and another worker can then receive the same message. Design the message around a short, recoverable action rather than holding one queue delivery open for the whole render.

Use distinct dispatch and observation messages

Your dispatch message should refer to an application operation, not contain an unconditional instruction to POST. The worker loads the operation and checks whether it already has a remote job ID. If it does, it schedules observation instead. Only a prepared, authorized operation should enter submission.

After acceptance, persist job_id and create an observation task. An observer reads GET /api/v1/jobs/{id} and saves the snapshot. If still active, it schedules another observation under your delay policy. A completed observation action can acknowledge its queue message even though the remote render is not terminal yet.

message action: observe_generation
operation_id: application-operation-42
expected_remote_job_id: saved-job-id
expected_revision: 7

These are your queue fields. Genace does not receive this envelope. The expected revision helps a worker detect stale local instructions after the user changes or replaces an operation.

Size visibility for the actual worker action

Include connection acquisition, the chosen long-poll window, response parsing, and database persistence in the lease budget. A thirty second wait plus networking and storage needs more than a thirty second visibility period. Renew the queue lease when the action legitimately outlasts its initial budget, using the queue’s supported mechanism.

A renewal failure means ownership may be lost. The worker should stop claiming exclusive authority and use conditional database updates to avoid overwriting a newer observer. It must not respond by creating another generation. Queue visibility limits concurrent processing opportunities but is not an external side-effect deduplication guarantee.

Handle submission timeouts differently

A repeated observation message can safely ask about the known job under a bounded read policy. A repeated dispatch message is more delicate if the previous worker lost the response after sending a POST. Mark that operation uncertain and reconcile it rather than treating message redelivery as proof that submission never happened.

If the API returns a known submission failure with error.job_id, save the ID and inspect credit settlement. Refund retries are a separate internal accounting path; they do not grant your queue permission to submit an identical creation request automatically.

Test redelivery under controlled conditions

Use a local queue adapter or isolated test queue. Pause an observer until visibility expires and let another worker receive the message. Confirm both target the same saved job and only the current revision can update local state. Then simulate a dispatch connection loss and verify redelivery enters reconciliation rather than POST again.

Measure queue age, action duration, remote render age, and lease renewal failures separately. Those measurements identify whether backlog comes from workers, network waits, or rendering capacity.

See worker leases, dead-letter recovery, and polling budgets.

Continue with production operations

Where these facts come from