Genace
Sign inGet started

generation worker lease: coordinate ownership with conditional updates

·

A worker lease tells your application which process currently owns a local action. It should have an owner token, an expiration time, and a revision that changes on takeover. A lease is useful for coordinating observers and delivery workers, but expiration does not undo an external generation request that the previous owner already sent.

Claim a specific action atomically

Use a short database transaction to select eligible work and install a lease. Do not keep a transaction open during a long HTTP request. This application-owned table needs a primary-key operation_id, text remote_job_id, next_action and lease_owner, a timestamptz lease_until, and a non-null bigint revision starting at zero. These are proposed application fields, not Genace’s internal schema.

UPDATE app_generation_operations
SET lease_owner = $2,
    lease_until = clock_timestamp() + INTERVAL '45 seconds',
    revision = revision + 1
WHERE operation_id = $1
  AND next_action = 'observe'
  AND remote_job_id IS NOT NULL AND btrim(remote_job_id) <> ''
  AND (lease_until IS NULL OR lease_until <= clock_timestamp())
RETURNING operation_id, remote_job_id, revision;

No returned row means this statement did not acquire the action; a database error is a separate outcome. Missing remote IDs must go to reconciliation, not observation or an automatic POST. Derive and authorize the operation in your application before running this internal worker query.

PostgreSQL’s time functions distinguish transaction start from current wall time. CURRENT_TIMESTAMP stays fixed inside a transaction: a lease that expires after BEGIN can still appear unexpired. clock_timestamp() evaluates current time instead. Keep transactions short anyway, and handle database clock changes operationally; this is not a monotonic distributed clock or a guarantee that authority lasts through commit. The 45-second interval is a local example. Budget network and persistence work and use bounded lock and statement timeouts for contention.

Fence completion and renewal

When saving a result or extending the lease, require the same owner token, revision, action, and an unexpired lease. For renewal, bind the original owner and revision returned by claim; do not fetch a newer revision to impersonate the current worker:

UPDATE app_generation_operations
SET lease_until = clock_timestamp() + INTERVAL '45 seconds'
WHERE operation_id = $1
  AND lease_owner = $2
  AND revision = $3
  AND next_action = 'observe'
  AND lease_until > clock_timestamp()
RETURNING operation_id, revision, lease_until;

Zero returned rows means renewal was denied. Even without a replacement owner, an expired lease must be claimed again under a new revision. A lost renewal response is uncertain ownership, not proof of either renewal or expiry; stop exclusive writes until resolved. Keep diagnostics, but never overwrite the current owner’s state or schedule delivery under an old revision.

For remote status reads, overlapping workers may both observe the same known ID without creating another render. Conditional persistence still matters because a late pending snapshot should not replace a newer terminal snapshot. Model local updates as permitted transitions rather than unconditional last-writer-wins assignments.

Treat dispatch takeover as a separate problem

A lease that expires while a worker is submitting a generation leaves a possible external side effect. Another worker must inspect the operation’s phase and recorded evidence before dispatching. A submitting operation without a reliable response should enter uncertainty review, not automatically become prepared.

Genace’s public creation route does not accept a documented application lease token or idempotency key. Your local fencing token therefore cannot make an already-sent remote POST disappear. Refund idempotency inside Genace protects credit repayment, which is a different operation from generation creation.

Plan graceful shutdown and crash recovery

On shutdown, stop accepting new claims and finish or release known observation work within the process budget. Do not release a dispatch claim as safe-to-retry if its network outcome is unknown. Persist an explicit reconciliation state before leaving when possible.

Use a durable job ID for observation and separate delivery progress from rendering state. A collector can resume GET requests after takeover, while an upload worker can inspect its deterministic object key before copying again.

Acceptance tests

Execute the actual SQL against an isolated database. Begin a transaction, set a lease to expire just after its start, wait past that instant, then confirm claim uses current wall time. Test missing IDs, the wrong action, expired renewal, wrong owner, and a stale revision even when the owner string is reused.

Separately run multiple PostgreSQL connections to exercise lock waits and real worker overlap; sequential claims in a single embedded database do not prove that concurrency behavior. Test conditional result persistence in your own implementation, then pause a dispatcher after a mocked POST and confirm takeover does not submit again. These observation snippets do not implement dispatch, result storage, or queue scheduling.

See PostgreSQL operation storage, Redis coordination, and uncertain submissions.

Continue with production operations

Where these facts come from