Job state failed and UPSTREAM_FAIL: when the model provider errors

An upstream failure can mean a rejected HTTP request, an uncertain submission, or a job that reached a failed terminal state. These outcomes require different recovery steps. This page keeps its existing UPSTREAM_FAIL URL and lookup term, but that label is not a promise that every provider failure returns an error code named UPSTREAM_FAIL. Preserve the actual response before deciding what to do next.

At submit time: an HTTP error

The gateway validates the model and parameters, then reserves credits and creates the queued job in one database transaction. The reservation includes the balance deduction, FIFO credit consumption, and USAGE record. If that transaction fails, the reservation and job insertion roll back together; the provider submission has not been called.

Only a confirmed rejection permits submission-failure compensation. The service persists UPSTREAM_SUBMISSION_FAILED together with submissionOutcome: "rejected", then attempts its refund. Generic transport exceptions and unclassified responses preserve an unknown queued job and its reservation. The confirmed-rejection HTTP response is 502:

{
  "error": {
    "type": "upstream_error",
    "code": "UPSTREAM_SUBMISSION_FAILED",
    "message": "Upstream submission failed. Check the job for credit reconciliation.",
    "job_id": "saved-job-id"
  }
}

Save error.job_id and query GET /api/v1/jobs/{id} using the same account. This is the local job identifier, not an upstream request identifier. The subsequent job snapshot calls it id. A 502 does not by itself prove that the refund has settled; inspect the snapshot and credit history.

Only the provider submission call is covered by this compensation catch. If the provider returned an accepted reference but saving that reference fails, the service does not treat it as a rejected submission and automatically refund it. Other unhandled exceptions can still return HTTP 500 with error.type: "internal_error" and error.code: "INTERNAL". If persisting the submission-failure marker itself fails, the local ID still supports manual investigation, but automatic compensation cannot infer the missing marker. Transport failures can also leave the upstream acceptance outcome uncertain.

Preserve the timestamp, model, application operation identifier, and complete response for reconciliation before generating again. Refund idempotency does not make creation idempotent: an unconditional POST retry may create another job because this endpoint does not publish a submission idempotency contract.

After submit: a terminal job state

After a successful submission, inspect the saved job through polling or SSE. Genace does not currently expose customer callback URLs. The job read can return HTTP 200 with a failed state:

{
  "id": "saved-job-id",
  "state": "failed",
  "outputs": null,
  "error": { "code": "UPSTREAM_EMPTY_RESULT", "message": "..." },
  "credits_actual": 0
}

An HTTP client that only throws for unsuccessful status codes will miss this render failure. The submission response uses job_id; the subsequent job snapshot uses id. Keep that distinction in persistence and logging, and do not dereference outputs[0] before checking the state and output collection.

Check state, not just the status code.

The fal SDK result method returns a wrapper containing data and requestId. The gateway now extracts that data before an image or video adapter inspects media fields. Treating the wrapper itself as the model payload could otherwise misclassify a successful output as empty. This boundary is covered by offline tests for all six registered adapters, not a paid quality benchmark. Historical outcomes and credit balances are not automatically recomputed by this fix.

The error codes

Code	Meaning
UPSTREAM_SUBMISSION_FAILED	Confirmed submission rejection; automatic compensation requires rejection evidence
UPSTREAM_SUBMISSION_UNCERTAIN	Acceptance unknown; keep the ID and reservation without resubmitting
UPSTREAM_EMPTY_RESULT	Provider reported COMPLETED but returned no media
UPSTREAM_WEBHOOK_ERROR	The provider’s webhook reported a failure
UPSTREAM_UNKNOWN_STATUS	A status value the adapter does not recognise
UPSTREAM_TIMEOUT	Reserved timeout code in the provider taxonomy; verify the actual producer before assuming automatic timeout handling
UPSTREAM_RATE_LIMITED	Reserved provider rate-limit code; current adapters do not emit it
UPSTREAM_INVALID_INPUT	Reserved provider input-rejection code; current adapters do not emit it

These are names defined in the provider taxonomy. A defined name does not prove that every failure path emits it. In particular, UPSTREAM_RATE_LIMITED and UPSTREAM_INVALID_INPUT are defined as reserved constants, but current adapters do not emit them. Inspect the actual error.code rather than hard-coding a speculative retry list from an example. Provider webhook errors refer to Genace’s upstream ingestion, not a customer webhook subscription.

Credits come back

The normal job service invokes a full refund of held credits on failed or canceled outcomes and records credits_actual: 0 after settlement. For a persisted confirmed rejection, a temporary refund error leaves the job failed with credits_actual unsettled, normally serialized as null. Its next_poll_at is scheduled with backoff so the worker can retry compensation. An authorized GET of that job can also drive another settlement attempt.

const failed = snapshot.state === 'failed';
const recordedAsZero = snapshot.credits_actual === 0;
const needsReconciliation = failed && !recordedAsZero;

Check the snapshot and credit transaction history before treating a refund as settled. Refunds use a deterministic job-refund:{id} transaction identifier, a locked balance row, an atomic balance increment, and a REFUND record whose remainingAmount is available for later credit consumption. Repeated settlement attempts do not add the same refund again. Internal reconciliation also counts matching legacy refunds, including accumulated partial records, and adds only the difference needed to reach the amount due; this is not a public partial-refund API.

Automatic compensation retries target the explicit UPSTREAM_SUBMISSION_FAILED marker, submissionOutcome: "rejected", and unsettled accounting. Historical markers without that evidence are not automatically assumed refundable. These retries do not submit to the provider again, refund a queued job merely because it is old, or repair every terminal failure indiscriminately. Existing production jobs with unknown queued outcomes were not retroactively rewritten. The configured cron currently runs daily, so its backoff timestamp is not a promise of minute-level refund completion. Use GET to check and advance settlement, or escalate an unresolved case with its local job ID.

A dropped connection is not refund evidence. The manual reconcile-jobs --fail implementation updates the job record without invoking the refund helper, so marking a task failed is not by itself a billing repair.

Which of these to retry

• A failed read or disconnected stream: read the same job again; do not submit a new generation just to recover status.
• UPSTREAM_SUBMISSION_FAILED: save error.job_id, read that job, and verify its credit settlement before deciding whether a new render is appropriate.
• UPSTREAM_EMPTY_RESULT: retain the failed job and request context, verify accounting, and make any new render an explicit budgeted decision. An identical retry is not guaranteed to produce a valid result.
• Invalid parameters: fix the payload using the model schema and INVALID_PARAM guidance before trying again.
• UPSTREAM_UNKNOWN_STATUS: preserve the response and report the adapter mismatch instead of repeatedly submitting work.

Preserve provider identifiers without renaming them

HTTP ApiError responses may contain error.upstream_request_id. Job snapshots serialize their error object unchanged, whose optional field is error.upstreamRequestId. Neither field is guaranteed to exist. Preserve the exact returned shape and your local job ID when escalating; do not invent a provider identifier from the local ID.

For a complete consumer example, read the failed job response guide. For reconnecting and diagnostics, see resuming jobs and observability.

Unknown submission outcomes need reconciliation

Transport failures and malformed acceptance responses use UPSTREAM_SUBMISSION_UNCERTAIN, preserve the queued reservation, and do not trigger another creation request. Nonterminal jobs with an uncertain marker or either missing upstream handle cannot be canceled into a refund: the route returns HTTP 409 SUBMISSION_RECONCILIATION_REQUIRED, including when saving the uncertainty marker failed. Legacy failure codes without rejection evidence need manual investigation, not bulk balance changes. Read the unknown-submission reference.