Genace
Sign inGet started

generation worker readiness: distinguish a live process from safe dispatch capacity

A generation worker can be running while it should not accept another dispatch. Its database may be unavailable, its operation lease may be lost, or the application may be draining for deployment. Define liveness as process viability and readiness as permission to take the next kind of work. These are application deployment controls, not new Genace API endpoints.

Separate the signals by consequence

Liveness should identify a process that needs restarting, not every temporary upstream problem. Readiness should stop new work when the worker cannot safely claim or persist operations. A startup signal can allow initialization to finish before either decision is enforced by your deployment platform.

For a queue worker, becoming unready does not automatically stop queue consumption unless your application connects that signal to its claim loop. Kubernetes readiness primarily affects traffic routing; your worker code must implement its own pause and drain behavior.

{
  "process": "alive",
  "accepting_dispatch": false,
  "accepting_observation": true,
  "reason": "dispatch_paused",
  "active_operations": 3
}

This is an example internal status response. Keep it protected if it reveals operational details. Dispatch and observation can have different readiness requirements: pausing new paid work may still allow collectors to reconcile known jobs and complete delivery.

Base readiness on local safety prerequisites

Require access to your durable operation ledger, a valid configuration version, available worker capacity, and the ability to enforce your allowance policy. A successful model-list GET can verify gateway authentication, but it does not prove the next provider render will succeed.

Avoid calling paid generation from liveness or readiness probes. Also avoid using a transient upstream timeout as a reason to restart every worker simultaneously. That can destroy useful in-memory context while creating a synchronized retry burst.

Drain accepted work deliberately

On deployment shutdown, stop claiming new dispatch operations, save accepted job IDs, and release or finish short observation actions under a bounded grace period. Preserve uncertain submission outcomes rather than marking them ready for immediate redispatch.

A collector can resume from the durable ID after restart. A delivery worker can inspect the stored artifact manifest. If the process exits during the acceptance-to-persistence interval, keep a reconciliation case instead of pretending shutdown canceled the remote work.

Interpret existing worker statistics correctly

Genace’s polling worker reports a count of selected jobs, processing exceptions, and resulting state distribution. Its failed counter describes processing exceptions, not every job whose terminal state is failed. Likewise, the cron route may return HTTP 200 while recording an internal operational failure, so status code alone is not a sufficient alerting signal.

The configured cron currently runs daily. Do not derive a minute-level recovery guarantee from internal backoff timestamps or older worker comments. Authorized GET observation can also advance known jobs.

Acceptance tests

Simulate a database outage, deployment drain, exhausted local capacity, and an upstream read timeout. Confirm each affects the intended signal and that a paused dispatcher sends no new POST. Restart after a saved acceptance and verify observation resumes. Ensure probes remain bounded and do not expose credentials or create renders.

See non-generating health checks, worker leases, and structured logs.

Continue with production operations

Where these facts come from