Loading…
Product Visit
Visit tomorrow's products today
Re-engineer LLM prompts to spend the fewest tokens that still answer. A governing gateway with auth, budgets, caching, and streaming. Works with Anthropic, OpenAI, Gemini, and any OpenAI-compatible provider.
Re-engineer LLM prompts to spend the fewest tokens that still answer — a governing gateway with auth, budgets, caching, and streaming that works with Anthropic, OpenAI, Gemini, and any OpenAI-compatible provider. Plus a live A/B, saved constraint presets, and a batch triage that flags prompts which never needed a model.
llm · tokens · cost-optimization · ai-gateway · llmops · openai · anthropic · gemini · streaming · prompt-engineering
A fixed one-screen dashboard (no page scroll — panels scroll internally). The top bar carries the mode tabs and pricing/scale inputs; below sit three panels:
Switch to Batch mode and this becomes a triage table that flags deterministic prompts. (The image is a layout mockup, not a screenshot — run the app for the live, interactive version.)
fetch)npm install cp .env.example .env # then edit .env (see "Providers" below)
The server is provider-agnostic. Set these in .env:
| Variable | Purpose |
|---|---|
LLM_PROVIDER |
anthropic or openai (openai = any OpenAI-compatible API) |
LLM_API_KEY |
your key (kept server-side; never sent to the browser) |
LLM_MODEL |
model id your key can access |
LLM_BASE_URL |
optional; override to point at Gemini, Groq, OpenRouter, Ollama, etc. |
The frontend never changes: it POSTs a normalized request to /api/messages,
and the adapter in server/providers.js translates to/from the chosen provider
and returns a normalized response. To add a native provider, edit that one file.
Almost every provider now offers an OpenAI-compatible API, so it works by setting a base URL and model — no code change. Only providers you want to hit on their own native protocol need an adapter.
| Provider | Supported? | How |
|---|---|---|
| Anthropic | Yes | native (LLM_PROVIDER=anthropic, or gateway /v1/messages) |
| OpenAI | Yes | LLM_PROVIDER=openai |
| Gemini | Yes | OpenAI-compat base URL (below) |
| Groq, OpenRouter, Together, Mistral, Fireworks, DeepSeek, xAI/Grok | Yes | OpenAI-compat base URL |
| Local — Ollama, vLLM, llama.cpp | Yes | OpenAI-compat base URL |
| Azure OpenAI | Mostly | OpenAI-compat, but needs a different auth header + api-version |
| Cohere / Bedrock / Vertex native protocols | Not yet | would need a native adapter in server/providers.js |
All OpenAI-compatible providers use LLM_PROVIDER=openai:
# OpenAI LLM_MODEL=gpt-4o-mini # Gemini (Google AI Studio) — NOTE: no trailing slash (the code appends /chat/completions) LLM_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai LLM_MODEL=gemini-2.5-flash # use a model your key can access # Groq LLM_BASE_URL=https://api.groq.com/openai/v1 # OpenRouter LLM_BASE_URL=https://openrouter.ai/api/v1 # Local Ollama LLM_BASE_URL=http://localhost:11434/v1 LLM_API_KEY=ollama
Notes:
.../v1beta/openai/ because the OpenAI SDK appends chat/completions. This server appends /chat/completions, so drop the trailing slash or you'll get a double-slash 404.max_completion_tokens — set OPENAI_MAX_TOKENS_PARAM=max_completion_tokens if you hit that error.npm run check:live confirms the wiring end to end.generateContent) is not supported. If your app uses the Google GenAI SDK's native API rather than its OpenAI-compat mode, either switch that client to OpenAI-compat mode or add a native Gemini adapter (a body-mapper plus a :streamGenerateContent passthrough, parallel to the Anthropic adapter).npm run dev
/api to it)npm run build # outputs dist/ npm start # Express serves dist/ AND /api on PORT (default 8787)
Confirm the key and model are wired before you ship:
npm run check # hits a running server's /api/health npm run check:start # boots the server itself, checks, stops npm run check:live # also sends a tiny real prompt (costs a few tokens)
Check a deployed instance instead of localhost:
HEALTH_URL=https://your-app.example.com/api/health npm run check
It verifies the health endpoint responds, the key is present, and a model and
base URL are configured; --live additionally confirms a real round-trip.
Exit code is 0 on success, 1 on any failure — so it drops straight into CI.
Beyond the analysis UI, the server is also a governing gateway. Point your
existing SDK's base_url at it and every call is governed automatically —
constraints injected, max_tokens capped, deterministic prompts optionally
blocked — then streamed straight back. Your app code barely changes.
Native endpoints:
| Endpoint | Protocol | Upstream |
|---|---|---|
POST /v1/messages |
Anthropic | Anthropic (or ANTHROPIC_BASE_URL) |
POST /v1/chat/completions |
OpenAI-compatible | OpenAI / OPENAI_BASE_URL |
GET /v1/governor/info |
— | lists policies + wiring (no secrets) |
Per-request control via headers:
| Header | Effect |
|---|---|
x-governor-policy: terse |
pick a named policy (default, terse, strict, passthrough) |
x-governor-detect: block |
override detect mode (off / warn / block) |
x-governor: off |
bypass governance for this call |
Policies live in server/policies.js (instructions, maxTokensCap,
forceTemperature, detect). Edit or add your own.
# OpenAI SDK
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8787/v1", api_key="unused")
client.chat.completions.create( model="gpt-4o-mini", stream=True, messages=[{"role": "user", "content": "..."}], extra_headers={"x-governor-policy": "terse"},
)
# Anthropic SDK
from anthropic import Anthropic
client = Anthropic(base_url="http://localhost:8787", api_key="unused")
with client.messages.stream(model="claude-sonnet-5", max_tokens=1024, messages=[{"role": "user", "content": "..."}]) as s: for text in s.text_stream: print(text, end="")
(The gateway supplies the real key from its env; the SDK's api_key is ignored.)
stream: true is passed through as Server-Sent Events, unmodified, with
backpressure preserved. The gateway sniffs a copy only to log token usage —
it never alters the bytes your client receives. Try it:
node examples/stream.mjs # OpenAI-protocol streaming demo
curl -N http://localhost:8787/v1/chat/completions \ -H 'content-type: application/json' -H 'x-governor-policy: terse' \ -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"hi"}]}'
strict policy (or x-governor-detect: block) blocks deterministic prompts; it returns a provider-shaped 400 with a suggested script. The detector is a heuristic — run it in warn first.max_tokens is the only hard lever — instructions are advisory, the cap is enforced.[usage] … line per call. For OpenAI streaming, set stream_options.include_usage=true (the demo does) or usage will read as zero on streams./api/messages is for).GATEWAY_TIMEOUT_MS, GATEWAY_RETRIES). The timeout covers connect/headers, not the stream body, so long streams aren't cut off.REDIS_URL (and npm i redis) to share budgets and cache across replicas; otherwise state is per-process./v1/chat/completions: set OPENAI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai (no trailing slash) and OPENAI_API_KEY=<gemini key>, then use a gemini-* model. See the Providers table for the full compatibility list.The gateway is also a cost-control plane. All three are optional and layer onto
the same /v1/* endpoints (the app's own /api/messages is unaffected).
Auth. Define callers, each with a bearer key and optional pinned policy and
budget, in gateway.callers.json (see gateway.callers.example.json):
{ "callers": [ { "id": "web-app", "key": "gw_live_...", "policy": "terse", "budget": { "tokens": 2000000, "usd": 20 } }, { "id": "worker", "key": "env:WORKER_KEY", "budget": { "tokens": 500000 } }
] }
Callers send Authorization: Bearer <key>. A key value of env:VAR is pulled
from the environment (so it can come from an injected secret). With no callers
configured the gateway runs open (anonymous) — fine for local, not for
production. A caller's pinned policy is authoritative: clients can't loosen it
with x-governor-* headers.
Budgets. Per-caller daily token and/or USD caps (UTC reset). When a caller is
over, the next call gets 429 with Retry-After. Cache hits and the current
count are free. USD caps need a pricing file (gateway.pricing.example.json →
GATEWAY_PRICING_FILE); token caps need nothing. Check usage:
curl -H 'Authorization: Bearer gw_live_...' http://localhost:8787/v1/governor/usage
Enforcement is soft (checked before each call, recorded after), and the ledger is
in-memory — single-instance. For multiple replicas, back server/budget.js with
a shared store (e.g. Redis).
Cache. Exact-match cache for non-streaming, deterministic calls
(temperature <= GATEWAY_CACHE_MAX_TEMP, default 0). A hit returns the stored
response with x-governor-cache: hit, skips the provider, and doesn't charge
budget. Sampled (higher-temperature) and streaming calls are never cached, so you
never get a stale non-deterministic answer. Bypass per request with
x-governor-cache: off. Introspect the whole plane at GET /v1/governor/info.
npm test
Dependency-free (node --test) coverage of the control plane — auth token
checks, budget enforcement, and cache determinism/keying — runnable without a
provider key. npm run check / check:live validate a running server's wiring.
docker build -t token-governor . docker run -p 8787:8787 \ -e LLM_PROVIDER=anthropic -e LLM_API_KEY=sk-... -e LLM_MODEL=claude-sonnet-5 \ token-governor
The API key is never hardcoded — it comes from the environment. There are two ways to source it from a managed secret store:
1. Platform injection (recommended, no code). Let your platform fetch the secret and inject it as an env var at deploy time. Nothing in the app changes:
--set-secrets=LLM_API_KEY=anthropic-key:latestsecrets from AWS Secrets Manager or SSM2. In-app fetch (built in). Point any *_API_KEY at a managed secret using a
scheme, and the server resolves it at startup (see server/secrets.js):
