generation API fault injection: test each side-effect boundary independently

A generic network-error test cannot prove that a generation workflow handles its different failure boundaries. The correct recovery depends on whether credits were reserved, the provider accepted work, the local reference was saved, or a completed output was being delivered. Inject one failure at a time and assert both the expected state and the side effects that must not happen.

Map the execution boundaries first

The current service validates a model and parameters, reserves credits and inserts the job atomically, calls the provider, then saves the returned reference. Only a typed, confirmed provider rejection enters submission-failure compensation; an arbitrary transport exception preserves uncertainty and held credits. A later reference-save failure is not treated as a rejected submission.

reservation fails -> no committed reservation/job, no provider call
confirmed provider rejection -> persist rejection evidence, attempt refund, HTTP 502 with job ID
transport/unknown submit exception -> preserve queued uncertainty and reservation, no automatic retry/refund
failure marker save fails -> preserve incident for manual reconciliation
refund fails -> marked job remains unsettled for retry
accepted reference save fails -> uncertain accepted outcome, no rejection refund assumption
download fails -> local delivery failure, existing render remains unchanged

This matrix describes boundaries to test, not a claim that every distributed uncertainty can be resolved automatically. Keep unknown outcomes visible instead of forcing them into a convenient success or failure label.

Inject through interfaces, not production infrastructure

Use a fake provider and database adapter for service flow tests. Make each dependency fail on a named operation, then inspect call counts and saved state. The repository already uses injected JobDb interfaces and mocked providers in tests. No paid API request is needed to exercise these branches.

For transaction behavior, use an isolated database capable of executing the relevant constraints and rollback semantics. The repository’s ledger fixture uses PGlite with a focused schema. That provides useful SQL transaction coverage but does not establish production multi-connection locking or every production foreign-key behavior.

Assert prohibited side effects explicitly

After a reservation failure, assert provider submission was never called. After an uncertain reference-save failure, assert the rejection-compensation path did not issue an unjustified refund. For an unknown marker-write failure, assert cancellation still returns 409 when either upstream handle is absent. After a confirmed-rejection refund retry, assert the provider was not called again and the ledger did not add another full refund.

For application collectors, inject GET timeout, malformed response, and database snapshot-save failure. Each should retain the known job identifier. For delivery, inject partial bytes and upload rejection while preserving the remote succeeded state.

Test interruption as well as exceptions

An exception is not identical to a process disappearing after a side effect. Pause execution immediately after a mocked acceptance, restart from the saved operation record, and verify the recovery decision. If no ID was durably saved, the operation should remain uncertain rather than dispatch automatically.

Use synthetic operation IDs and local media fixtures. Block external network access in the test harness so a missed mock cannot turn a reliability test into a paid generation.

Acceptance criteria

Document which boundary each test covers and what remains outside its model. Confirm repeat runs start from isolated state, failure injection is deterministic, and logs contain safe correlation fields. A useful suite explains the recovery decision instead of merely proving that an exception was thrown.

See contract fixtures, refund tests, and uncertain submissions.

Unknown submission outcomes need reconciliation

Transport failures and malformed acceptance responses use UPSTREAM_SUBMISSION_UNCERTAIN, preserve the queued reservation, and do not trigger another creation request. Nonterminal jobs with an uncertain marker or either missing upstream handle cannot be canceled into a refund: the route returns HTTP 409 SUBMISSION_RECONCILIATION_REQUIRED, including when saving the uncertainty marker failed. Legacy failure codes without rejection evidence need manual investigation, not bulk balance changes. Read the unknown-submission reference.

Continue with production operations

• AI video API webhook vs polling: which one to use and when
• How to call a video generation API without blocking your agent
• generation operation unique key: deduplicate local intent without inventing API idempotency
• Browse all production operations guides