Troubleshooting

Every error carries a code. The code is the diagnosis — start there, not with the status.

{
    "error": {
        "type": "invalid_request_error",
        "code": "alias_unresolvable",
        "message": "The alias \"transcribe\" resolved to no available model.",
        "param": "model",
        "request_id": "req_3a71f0c9d284"
    }
}

request_id is also on every response as X-CogniGate-Request-Id, including responses that failed before anything else ran. It is the one string worth quoting in a bug report.

Authentication

wrong_plane is the one that catches people. A cg- key on /admin/v1, or a cga- key on /v1, is a distinct refusal precisely so it does not read as "bad credential". Check the prefix before you check anything else.

insufficient_scope on tenant creation means the key is tenant-scoped. Only a root admin key — or the bootstrap key — can create or delete tenants.

A tenant-scoped key reaching another tenant's resources gets resource_not_found, not a 403. That is intentional and not a bug: a 403 would confirm the other tenant exists.

Routing

alias_unresolvable almost always means no provider is registered, or the registered provider's catalogue does not advertise the capability the alias requires. It is separated from model_not_found because it is a configuration problem the caller cannot see or fix — a bare 404 would send them looking in the wrong place.

If a provider is registered and the catalogue is still empty, the reason is on GET /v1/health as catalog.error. It is present only when every provider failed and no snapshot exists at all, which is the one case the per-provider rows cannot report, and it names the provider and its own reason. A key the provider refuses is called that in as many words. No key material appears in it.

upstream_exhausted carries an attempts array naming what was tried:

"attempts": [
  {"provider": "openai", "model": "gpt-4o", "failure": "rate_limit", "status": 429},
  {"provider": "local",  "model": "llama-3.1-70b", "failure": "breaker_open"}
]

breaker_open means the gateway never made the call. Check GET /v1/health, which reports every breaker that is not closed and, on the open ones, breaker_until.

Limits and quotas

All four 429s are distinct on purpose: they need four different responses from you. rate_limited wants backoff, concurrency_exceeded wants fewer parallel requests, quota_exceeded and budget_exceeded want a bigger cap or a new window.

X-CogniGate-Quota-State reports the caller's position on every data-plane response: ok, soft-exceeded (past the warning threshold, still served) or hard-exceeded. The approach to a cap is visible before the refusal.

"My quota didn't stop anything"

Two likely causes, both by design:

1. quotas.enforcement is observe. Over-cap requests are served, events are emitted, and headers are set. That is what the mode is for — sizing caps before they start refusing traffic. Switch to on.
2. Metering is asynchronous. Usage is recorded after the response is returned, so a burst of concurrent requests can cross a cap together before any of them has been counted. The cap is enforced on the next request, not retroactively on that burst.

Streaming

upstream_stream_stalled means a stream produced nothing for longer than limits.stream_idle_timeout (60s by default) and was ended rather than left holding a connection open indefinitely.

It arrives as a terminal SSE event, never an HTTP status — the status line went out with the first token, long before the stall. This is the reason a streaming client must read the last event rather than trusting the 200: a truncated answer announces itself in-band or not at all. The same applies to an upstream that dies mid-stream.

Raise the timeout if your provider is genuinely that slow to first token; suspect the provider if it happens often.

Configuration changes that seem not to apply

The gateway holds its configuration in memory, and that has two consequences worth naming before you go looking for a cache bug:

• A restart loses it. Tenants, keys, providers, aliases, routing rules and quotas are gone. Usage survives, because that lives in PostgreSQL.
• Replicas do not share it. Creating a tenant on one gateway does not create it on the others. If you are running more than one behind a load balancer, an admin call landed on exactly one of them.

Neither is a synchronisation failure. Both are the shape described under Scaling.

What is cached with a TTL is the model catalogue (catalog.ttl, 1h). It is invalidated automatically when a provider is added or changed, and POST /admin/v1/catalog/refresh forces it.

The stack itself

docker compose ps                    # all three should be healthy
docker compose logs -f gateway
docker compose logs -f analytics
curl http://localhost:8080/healthz   # unauthenticated liveness

GET /v1/health — with a data-plane key — is the richer view: per-provider reachability and every open breaker.

Usage is missing but requests are working

That is the expected shape of an analytics outage, not a coincidence: the gateway does not wait for the analytics service, buffers what it cannot deliver and drains the backlog when it recovers.

If usage stays missing after analytics is healthy, the likely cause is a mismatched ANALYTICS_TOKEN. The analytics service answers 401, the gateway treats a 4xx as permanent and drops the record after one attempt rather than retrying forever against a rejection that will not change. Look for usage record could not be persisted in the gateway's log.

Analytics will not start

It has no ANALYTICS_TOKEN, or it cannot reach PostgreSQL. It refuses to start without the token rather than publish an open metering API.

Filing a bug

Include the request_id, the code, and the gateway log line for that request id. All three are already on the response.