# llm-gateway

> One API in front of multiple model vendors. Send requests in the OpenAI Chat Completions,
> Anthropic Messages, or OpenAI Responses wire format; the gateway routes to the selected
> provider and answers in the dialect you called.

## Docs

- [skills.md](https://gateway.teamdev.ai/skills.md): how to connect a client, and what the gateway can do.
- [OpenAPI specification](https://gateway.teamdev.ai/openapi.yaml): the machine-readable contract — every route, schema, and error shape.
- [Field fidelity matrix](https://gateway.teamdev.ai/fidelity.md): every request field of every dialect, carried or declined with the written reason, plus the request shapes each model author's wire refuses. Generated from the gateway's own code, never hand-written.

## User documentation

- API and SDK integration: https://gateway.teamdev.ai/docs#api-integration
- Agent installation, SSO, key issuance (prefer a workload identity wherever the environment can present an OIDC token, and a static key only where it cannot; a member creates their own key or identity with no unit grant, charged to the one of their units they pick, by default their primary unit; a service account's key needs one of its administrators or the unit's head or a deputy; organization administrators list and revoke any key) and model routing: https://gateway.teamdev.ai/docs#agent-usage
- Model selection and SDK examples: https://gateway.teamdev.ai/skills.md#api-integration
- Complete machine-readable setup: https://gateway.teamdev.ai/skills.md#agent-and-mcp-setup
- One-shot local commands on macOS: [member-run](https://github.com/devsisters/llm-gateway/blob/main/docs/clients/member-run.md) signs in through the browser and supplies an expiring unit key on stdin.
- GitHub Actions workload identity: [setup and UUID requirements](https://gateway.teamdev.ai/skills.md#github-actions-workload-identity) and the [shared OIDC Action](https://github.com/devsisters/ai-tech-shared-library/blob/actions/llm-gateway-oidc/v1/actions/llm-gateway-oidc/README.md).
- Guided setup: llm-gateway-setup in Common Tools from devsisters-plugins; gateway MCP tools remain in llm-gateway@teamdev.

### Claude Code plugin

published version: 0.5.5

Install:

```sh
claude plugin marketplace add https://gateway.teamdev.ai/marketplace.json && claude plugin install llm-gateway@teamdev --scope user
```

Update:

```sh
claude plugin marketplace update teamdev && claude plugin update llm-gateway@teamdev
```

Invoke the installed skills explicitly as `/llm-gateway:api`, `/llm-gateway:migrate`, `/llm-gateway:keys`, `/llm-gateway:usage`, `/llm-gateway:workload-identities` or `/llm-gateway:console`. The `llm-gateway` MCP server acts with your existing console permissions, including workspace and service administration, after a browser sign-in. An installation from before 0.2.0 came from the `llm-gateway` marketplace: run `claude plugin marketplace remove llm-gateway`, then install again.

### Codex plugin

Install:

```sh
codex plugin marketplace add https://gateway.teamdev.ai/plugins.git && codex plugin add llm-gateway@teamdev
```

Update:

```sh
codex plugin marketplace upgrade teamdev
```

Invoke the installed skills explicitly as `$llm-gateway:api`, `$llm-gateway:migrate`, `$llm-gateway:keys`, `$llm-gateway:usage`, `$llm-gateway:workload-identities` or `$llm-gateway:console`. Sign the MCP server in with `codex mcp login llm-gateway`. An installation from before 0.2.0 came from the `llm-gateway` marketplace: run `codex plugin marketplace remove llm-gateway`, then add it again.

## Conventions

- **Auth**: `Authorization: Bearer lf_...`. `POST /v1/messages` also accepts `x-api-key: lf_...`.
- **Base URL**: OpenAI and Responses clients take `https://gateway.teamdev.ai/v1`; Anthropic clients take `https://gateway.teamdev.ai`.
- **Errors**: `{"error": {"message", "type", "code?"}}`, except `/v1/messages`, which uses the Anthropic envelope. Retry 429, retryable 503, 504, and a 502 coded `upstream_unavailable`. Do not retry `probe_spend_unavailable`. Disable automatic retries for workload probes. `X-Request-Id` is on every response.
- **Usage**: normalized to OpenAI semantics — `prompt_tokens` includes cache reads and writes, `completion_tokens` includes reasoning tokens, and `model` names what actually served the turn.
- **Dry run**: `X-Llm-Gateway-Dry-Run: 1` returns the request the gateway would have sent upstream, without dispatching or charging. Its envelope's `fields` array annotates each top-level request field with its observed fate — `expressed` (naming the upstream keys it shapes), `no_effect`, `required`, `declined` (with the recorded reason), or `unrecognized`. Runtime cache preparation is omitted: system instructions remain inline, and notes identify the unmaterialized cache reference; no cache resource is created or resolved.

## Endpoints

- POST /v1/workload-identity/token: alpha OIDC exchange for a short-lived client key; requires the workload identity feature flag and an approved EKS, Keycloak or GitHub Actions issuer.
  For GitHub Actions, set `workload_identity` to the UUID of the registered identity.
  The display name is invalid.

- `POST /v1/chat/completions`: chat completions (OpenAI dialect), streaming with SSE comment heartbeats supported.
- `POST /v1/messages`: messages (Anthropic dialect), streaming with native ping events supported.
- Operator-approved synthetic workload probes can use a spend envelope on unary Chat Completions, Messages, and Responses requests.
- `POST /v1/messages/count_tokens`: estimate an Anthropic Messages request's input tokens.
- `POST /v1/responses`: responses (OpenAI Responses dialect, stateless subset), streaming with SSE comment heartbeats supported.
- `GET /v1/responses`: WebSocket upgrade for the same dialect.
- `POST /v1/responses/compact`: compact a Responses conversation into a replayable item.
- `POST /v1/images/generations`: generate one inline base64 image.
- `POST /v1/images/edits`: edit source images with a text instruction.
- `GET /v1/models`: list the models and aliases you can call.
- `POST /v1/embeddings`: text embeddings.
- `POST /v1/decisions`: typed predicate, choice and score questions answered with probabilities; not streamed.
- `POST /v1/audio/speech`: text-to-speech with per-character timestamps (`timestamps: true` required); not streamed.
- `POST /v1/files`, `GET /v1/files`, `GET`/`DELETE /v1/files/{id}`, `GET /v1/files/{id}/content`: upload and manage files.
- `POST /v1/batches`: submit asynchronous batch jobs; `GET /v1/batches`, `GET /v1/batches/{id}`, `POST /v1/batches/{id}/cancel`.