Genace
Sign inGet started

generation API connection pooling: share transport capacity deliberately

A generation collector can spend most of its time waiting for status responses. Reusing an HTTP client reduces repeated connection setup, but the connection pool is only a transport resource. It does not define how many generation jobs your account may run. Size those budgets independently and make sure a crowd of observers cannot occupy every connection needed for a deliberate submission.

Measure active requests separately from active jobs

A remote job may remain running when no local connection is open. Conversely, several interfaces can observe one job through several simultaneous HTTP requests. Genace’s concurrency check counts queued and running jobs for the user, while status reads skip that active-generation check and still pass authentication and rate limiting.

The current job GET route allows a waiting window clamped between zero and sixty seconds. A thirty second long poll holds a client connection during observation. Pool sizing therefore depends on the number of concurrent collectors, not simply the number of newly submitted prompts per minute.

Use a shared client with explicit limits

This HTTPX configuration is an application example. Create the client once within the worker’s lifecycle, pass it to request functions, and close it during orderly shutdown. The numbers illustrate a policy rather than a recommended account quota.

import httpx

limits = httpx.Limits(max_connections=12, max_keepalive_connections=8)
timeout = httpx.Timeout(connect=5, read=45, write=15, pool=3)

def make_client(api_key: str):
    return httpx.AsyncClient(
        base_url="https://genace.ai/api/v1/",
        headers={"Authorization": f"Bearer {api_key}"},
        limits=limits,
        timeout=timeout,
        follow_redirects=False,
    )

Call paths relative to that trailing-slash base, such as jobs/{id}?wait=30. Rebuilding the client for every polling iteration discards the intended sharing. Creating an unbounded number of worker clients multiplies the effective limits, so document which process owns each pool.

Reserve capacity through application scheduling

Limit your own simultaneous observation tasks with a semaphore or worker allocation. If submissions and long polls need different latency targets, separate their application queues or clients and budget their combined connections. A second pool is not permission to exceed the upstream rate limit.

Pool timeouts mean a request could not acquire local transport capacity. They do not show that the provider rejected a generation. Record that distinction in telemetry. For status reads, reschedule observation of the same saved ID. For a submission whose bytes may already have been sent, preserve an uncertain outcome instead of retrying indiscriminately.

Watch for hidden consumers

Download workers may use the same transport library but contact media hosts rather than the API. Keep their credentials and connection policies separate. Do not attach the Genace Authorization header to a generic client that follows arbitrary output URLs. Also ensure abandoned status responses are consumed or closed so their resources can be released.

Verify under local pressure

Use a local mock that delays GET responses while recording connection counts. Confirm pool acquisition has a bounded deadline and canceled tasks release resources. Run more observers than available connections and verify submissions follow your intended scheduling policy. Count remote jobs separately from sockets in dashboards so a transport bottleneck does not look like a model concurrency rejection.

See HTTPX resource limits, generation concurrency, polling budgets, and deadline propagation.

Continue with code and integration

Where these facts come from