Job state failed and UPSTREAM_FAIL: when the model provider errors
Upstream failures surface in two different places depending on when they happen, and the difference decides how you have to handle them.
At submit time: an HTTP error
If the provider rejects the request outright, you get a response with
type: "upstream_error". No job was created.
After submit: a terminal job state
More often the job is accepted and fails later. There is no HTTP error to catch — you poll, or receive a webhook, and find:
{
"job_id": "...",
"state": "failed",
"error": { "code": "UPSTREAM_EMPTY_RESULT", "message": "..." }
}
Your try/catch will not see this. A failed job is a successful HTTP response that describes a failure. Code which only handles thrown exceptions treats it as a completed render and carries on with no output — usually surfacing much later as a missing file.
Check state, not just the status code.
The error codes
| Code | Meaning |
|---|---|
UPSTREAM_EMPTY_RESULT | Provider reported COMPLETED but returned no media |
UPSTREAM_WEBHOOK_ERROR | The provider’s webhook reported a failure |
UPSTREAM_UNKNOWN_STATUS | A status value the adapter does not recognise |
UPSTREAM_TIMEOUT | Never reached a terminal state within the stuck threshold |
UPSTREAM_RATE_LIMITED | Provider-side rate limiting |
UPSTREAM_INVALID_INPUT | Provider rejected the parameters |
UPSTREAM_EMPTY_RESULT is the one worth knowing by name. It means the
provider said it finished successfully and then handed back nothing — exactly
the case a naive integration mishandles, because every status check reported
the job as healthy right up to the end.
Credits come back
Every path above refunds in full:
if (update.state === 'failed' || update.state === 'canceled') {
await refundForJob({ /* ... */ });
}
You are not charged for renders that produced nothing, which also means a retry costs you one render rather than two.
Which of these to retry
UPSTREAM_TIMEOUT,UPSTREAM_RATE_LIMITED,UPSTREAM_EMPTY_RESULT— transient. Retry with backoff.UPSTREAM_INVALID_INPUT— your parameters. Retrying the same payload fails the same way. Compare them against the model’s schema; see INVALID_PARAM.UPSTREAM_UNKNOWN_STATUS— a gap in our adapter, usually after an upstream API change. Worth reporting rather than retrying.
upstream_request_id
Included when the provider gave us one. Quote it if you escalate — it is what identifies the call on the provider’s side.
Where these facts come from
- codebase: src/ai/providers/errors.ts — PROVIDER_ERROR_CODE taxonomy
- codebase: src/ai/jobs/service.ts — refund on failed/canceled