Routing & Failover

A caller sends {"model": "..."}. Turning that into an upstream call is three steps — resolve the name, build the chain, run the cascade — and each one has a rule about what it will not do.

1. Resolving the name

The name in the request is either a model in the tenant's catalog or one of the tenant's aliases. Every tenant is created with four:

An alias is a constraint, not a pin. It keeps meaning the same thing as providers ship new models, which is the point: a chain hardcoded to gpt-3.5-turbo decays, fast does not.

An alias may carry a pin, a required capability set, a minimum context window, an ordered provider_preference and a cost_tier. Resolution applies them in that order:

1. A pin that resolves wins outright. A pin that no longer exists in the catalog falls through to the constraints rather than 404ing — the alias still describes what the caller wants, and a retired model should not break it.
2. Filter to models that advertise every required capability and meet the minimum context window.
3. Order by provider_preference, then by price: ascending for cheapest and balanced, descending for best. Price is the only quality signal that exists across providers, which is a crude thing for best to mean and an honest one.
4. Break ties on model id, so the same request resolves the same way twice. A cache key depends on it.

A provider not named in provider_preference sorts after every one that is without being excluded. A preference is an ordering, not an allowlist.

curl -X PUT http://localhost:8080/admin/v1/tenants/$TENANT/aliases/summarise \
  -H "Authorization: Bearer $CG_ADMIN" \
  -H "Content-Type: application/json" \
  -d '{
        "capabilities": ["chat"],
        "min_context_window": 128000,
        "provider_preference": ["together", "openai"],
        "cost_tier": "cheapest"
      }'

2. Building the chain

A routing rule maps what the caller asked for to an ordered list of what to try.

curl -X PUT http://localhost:8080/admin/v1/tenants/$TENANT/routing-rules \
  -H "Authorization: Bearer $CG_ADMIN" \
  -H "Content-Type: application/json" \
  -d '{
        "match": "best",
        "chain": ["gpt-4o", "llama-3.3-70b", "fast"]
      }'

match is the model or alias the caller names; chain is tried left to right. Chain entries are themselves models or aliases, so a chain can end in a constraint rather than a specific model — a useful last resort.

With no rule, the chain is one entry: whatever the caller asked for.

Each chain position contributes at most one candidate. An alias that could expand to six models does not become six attempts; it becomes one, and the next attempt comes from the next position. That is what makes the chain the unit of fallback and keeps X-CogniGate-Fallback-Depth meaningful. A position that resolves to nothing at all is dropped rather than failing the request — surviving exactly that is why the chain exists.

A chain longer than routing.max_fallback_depth (default 5) is refused when the rule is written, not silently truncated when it is used.

3. Running the cascade

Key rotation happens inside one candidate

A provider is registered with a pool of keys. A 429 rotates to the next key in that pool and tries again — a per-key quota is not a provider outage, and cascading to a different model over one would be an overreaction. Only when every key in the pool has been rate-limited does the provider count as failing and the cascade move on.

Key rotation does not increment X-CogniGate-Fallback-Depth. The header counts candidates that failed, and a rotation stayed on the same one.

Which key a request starts from is key_strategy on the provider record. round_robin, the default, advances one key per request, so a pool of equivalent keys shares a steady load instead of piling onto the first until it throttles. failover starts every request at the first key and keeps the rest for when it begins refusing, which is what a paid key backed by a free one wants. Neither changes the paragraph above: whatever the starting offset, the whole pool is tried before the cascade moves on.

When the 429 carries a Retry-After, the key it came from is parked for exactly that long and the next request skips straight past it, rather than spending a round trip rediscovering a limit the provider already reported. The window is read from the header and nothing else: a provider that sends no Retry-After rotates exactly as it did before. Both forms the HTTP spec allows are understood, a delay in seconds and an absolute date. Which providers send it, and when, is theirs to decide; nothing here assumes any of them do.

A parked key is skipped, never dropped. If every key in a pool is inside a window the whole pool is tried anyway, because the window is the provider’s estimate of when it will accept traffic again and not a fact about when it will.

The failure that is never retried

A 4xx the request itself caused — a malformed body, an oversized prompt, a content-policy refusal — stops the cascade immediately, and the provider's own status and message are passed through rather than replaced.

This is deliberate and it is the most important rule on this page. The same request would fail identically on every other model in the chain, so cascading would burn the whole chain to be told the same thing five times, and bill the caller for each. A bad request should cost one call.

When everything fails

{
    "error": {
        "type": "api_error",
        "code": "upstream_exhausted",
        "message": "All 3 routing candidates failed.",
        "attempts": [
            {
                "provider": "openai",
                "model": "gpt-4o",
                "failure": "rate_limit",
                "status": 429
            },
            {
                "provider": "together",
                "model": "llama-3.3-70b",
                "failure": "server",
                "status": 503
            },
            { "provider": "local", "model": "llama-3.1-70b", "failure": "breaker_open" }
        ]
    }
}

breaker_open is named apart from the provider-reported failures on purpose: it says the gateway never asked. That is what lets an operator tell "the provider is down" from "the gateway has already decided the provider is down".

Only the classification crosses this boundary. The upstream's own error text stays in the gateway's log line, because it can quote a request body and request content does not belong in a response the caller did not send it with.

Circuit breakers

The breaker exists to stop the next several hundred requests from each paying a full timeout against a provider that is known to be down. It cannot save the request that is already failing.

Its unit of isolation is tenant / provider / model:

• Per model, so one model's outage does not take that provider's others out of rotation.
• Per tenant, because provider names are tenant-scoped. Two tenants that each call their upstream primary are naming different accounts, and one tenant's bad credential must not stop the other's traffic.

routing:
    breaker:
        error_threshold: 5 # failures...
        window: 30s # ...inside this window opens it
        open_duration: 60s # ...for this long

Failures older than window are dropped, so a slow trickle of errors spread over hours never accumulates into a trip.

Closed → traffic flows. Open → candidates are skipped with zero upstream calls. After open_duration, half-open → exactly one request is let through as a probe while everyone else keeps waiting.

A successful probe closes the breaker outright rather than decrementing a counter: the provider has demonstrably recovered. A failed probe re-opens immediately — one confirmation is enough, and spending another five requests to re-prove what the probe just showed would be the point of the breaker, missed.

Every transition emits a breaker.opened or breaker.closed event, and GET /v1/health reports every breaker that is not closed, with breaker_until on the open ones so a dashboard can say how long the outage has left to run.

What the caller sees

A cascade is invisible to the client otherwise: a request that fell back twice returns 200 with the same body shape as one that did not.