Genace

Job state failed and UPSTREAM_FAIL: when the model provider errors

HTTP 500 UPSTREAM_FAIL

Upstream failures surface in two different places depending on when they happen, and the difference decides how you have to handle them.

At submit time: an HTTP error

If the provider rejects the request outright, you get a response with type: "upstream_error". No job was created.

After submit: a terminal job state

More often the job is accepted and fails later. There is no HTTP error to catch — you poll, or receive a webhook, and find:

{
  "job_id": "...",
  "state": "failed",
  "error": { "code": "UPSTREAM_EMPTY_RESULT", "message": "..." }
}

Your try/catch will not see this. A failed job is a successful HTTP response that describes a failure. Code which only handles thrown exceptions treats it as a completed render and carries on with no output — usually surfacing much later as a missing file.

Check state, not just the status code.

The error codes

CodeMeaning
UPSTREAM_EMPTY_RESULTProvider reported COMPLETED but returned no media
UPSTREAM_WEBHOOK_ERRORThe provider’s webhook reported a failure
UPSTREAM_UNKNOWN_STATUSA status value the adapter does not recognise
UPSTREAM_TIMEOUTNever reached a terminal state within the stuck threshold
UPSTREAM_RATE_LIMITEDProvider-side rate limiting
UPSTREAM_INVALID_INPUTProvider rejected the parameters

UPSTREAM_EMPTY_RESULT is the one worth knowing by name. It means the provider said it finished successfully and then handed back nothing — exactly the case a naive integration mishandles, because every status check reported the job as healthy right up to the end.

Credits come back

Every path above refunds in full:

if (update.state === 'failed' || update.state === 'canceled') {
  await refundForJob({ /* ... */ });
}

You are not charged for renders that produced nothing, which also means a retry costs you one render rather than two.

Which of these to retry

  • UPSTREAM_TIMEOUT, UPSTREAM_RATE_LIMITED, UPSTREAM_EMPTY_RESULT — transient. Retry with backoff.
  • UPSTREAM_INVALID_INPUT — your parameters. Retrying the same payload fails the same way. Compare them against the model’s schema; see INVALID_PARAM.
  • UPSTREAM_UNKNOWN_STATUS — a gap in our adapter, usually after an upstream API change. Worth reporting rather than retrying.

upstream_request_id

Included when the provider gave us one. Quote it if you escalate — it is what identifies the call on the provider’s side.

Where these facts come from

  • codebase: src/ai/providers/errors.ts — PROVIDER_ERROR_CODE taxonomy
  • codebase: src/ai/jobs/service.ts — refund on failed/canceled