Billing products

Your AI bill can be quietly wrong. Providers shouldn't get to grade their own bills. Today, the company that charges you also decides what counts as a failure, what gets credited, and keeps the only detailed records. inferock-bench puts an independent, per-call receipt of what you were billed — and what failed — in your own hands. We built inferock-bench because we kept paying for answers that died mid-sentence, and nobody could tell us where the money went. Quickstart · Test your loss · What can go wrong · Key boundary · Docs Use it when you need to audit an AI/LLM bill, measure Claude or GPT token usage locally, or answer "was I billed for a failed API call?" It is a local LLM cost-tracking proxy for four measured provider planes: OpenAI, Anthropic, Gemini Developer API, and pinned OpenRouter endpoints spanning meta-llama, deepseek, mistral, moonshot/kimi, z-ai/glm, and qwen on observed hosts. Everything else is extensible-by-design, not measured today. It does not declare every mismatch an OpenAI overcharge or Anthropic billing error; it preserves token, cost, retry, and failure evidence so billing-integrity questions can be checked. Common cases it can help you inspect: a failed or timed-out request that still has usage, token counts that do not match the visible output, retries that may have amplified cost, and latency or model-version changes that need a trail. It cannot cap provider spend across calls it never sees, and it cannot explain traffic that bypassed the local proxy. The receipt headline Receipt word Plain-English meaning spent provider spend observed by the run for priced calls it saw. money loss bill-bounded dollar loss The Inferock Standard can tie to observed spend or charge evidence. time loss real wait or downtime measured as time, never added to dollars. invoice-check exposure an invoice-check amount, such as cache discount at risk; it is labeled "verify your invoice" and never summed into money loss. Real measured traffic, not fixture rows. Measured since 2026-07-09, the cumulative public ledger through run15 captured 1,268 measured calls, 565 failures/signals, $7.15 provider spend observed, $0.07 bill-bounded money loss (stored exact: $0.073875), ~2.9 min time loss, and $16.80 invoice-check exposure across 202 cache-discount-at-risk signals. The current-code cumulative receipt watches 12 of 13 surfaces and keeps invoice-check exposure separate from money loss. Run facts: sanitized public run card for 2026-07-09 and run15 public run card for 2026-07-10. The 2026-07-06 0.1.7 card remains published as a historical artifact; the current public receipt presentation ships with 0.1.10. ImportantThe receipt is spend-anchored. The headline is spent $X · money loss $Y · time loss Z · invoice-check exposure $E; bill-bounded money loss and recognition gap never include invoice-check exposure. CACHE_DISCOUNT_AT_RISK is still visible below the headline as a separate detail line that says "verify your invoice" rather than as money loss or a refund claim. Watch it run Share the receipt Quickstart Run it locally. We think you should be able to see exactly what a provider failure cost you, to the cent. Provider keys are not sent to Inferock; attached only to provider requests. Prerequisite: install Node.js 22+ with npm. node --version npm --version If either command is missing, install Node.js 22 or newer first. Run the local benchmark: npx inferock-bench The first run downloads the package and can take a minute or two before printing anything. Leave it running. You see lines like: inferock-bench listening at http://127.0.0.1:4318 Dashboard: http://127.0.0.1:4318/ Config: ~/.inferock-bench/config Save your provider key locally. Easiest path with the server from step 2 still running: open http://127.0.0.1:4318/ and save the provider key in the dashboard. Create that key in your provider account first; use a low-limit or development key while evaluating. It stays local under ~/.inferock-bench/, is saved with owner-only file permissions, and is shown back only in masked form. CLI path, before starting the server or after stopping it: npx inferock-bench setup <provider> The setup prompt hides your key while you type it. On a headless machine, pipe the key from your secret manager into the same command. To see the current supported provider names, run npx inferock-bench status or npx inferock-bench --help. A running server does not reload provider keys written by a separate CLI setup process; restart it after CLI setup. Any traffic you send through the benchmark is real provider usage. Start with a few short prompts and expect a small evaluation spend, controlled by your provider account limit. The built-in npx inferock-bench test flow shows estimated tokens, estimated dollars, and a spend cap before it makes any provider call. Get your local bench key: npx inferock-bench key reveal This prints the local ibl_ bench key to stdout, so it is pipe-friendly. It is a LOCAL-ONLY credential, not your provider key. To copy it to the clipboard instead, run: npx inferock-bench key copy If no clipboard is available, the copy command falls back to printing the key. You can also copy the local bench key from the dashboard. Check what is configured at any point: npx inferock-bench status It shows each provider's configured/masked state, the local store location, server state, and version. For a fast command list or package version without starting the server, use npx inferock-bench --help or npx inferock-bench --version. Point some traffic at the local benchmark. No app yet? Use one of these equal local targets after saving the matching provider key in step 3. Claude Code: npm i -g @anthropic-ai/claude-code ANTHROPIC_BASE_URL=http://127.0.0.1:4318 ANTHROPIC_API_KEY=ibl_your_local_bench_key claude -p "Draft a five-bullet checklist for reviewing an AI invoice." OpenAI SDK: import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.INFEROCK_BENCH_KEY ?? "ibl_your_local_bench_key", baseURL: "http://127.0.0.1:4318/v1", }); await openai.chat.completions.create({ model: "gpt-4o-mini-2024-07-18", messages: [{ role: "user", content: "Draft a five-bullet checklist for reviewing an AI invoice." }], }); Gemini: await fetch("http://127.0.0.1:4318/v1beta/models/gemini-2.5-flash:generateContent", { method: "POST", headers: { authorization: "Bearer " + (process.env.INFEROCK_BENCH_KEY ?? "ibl_your_local_bench_key"), "content-type": "application/json", }, body: JSON.stringify({ contents: [{ role: "user", parts: [{ text: "Draft a five-bullet checklist for reviewing an AI invoice." }] }], }), }); For these commands, the SDK API key is the local ibl_ bench key from step 4. Your provider key is not passed to Claude Code or your app; it is not sent to Inferock and is attached only to provider requests. If you configured OpenRouter, use the provider-specific SDK snippet in the dashboard's Local app connection panel or in the package README. Note: a Claude subscription (OAuth) login is not a supported mechanism for measuring calls. inferock-bench measures metered API traffic only — save a provider API key in the bench and point your SDK or agent at it with the local ibl_ key, as shown above. After the first successful proxied call, the terminal running inferock-bench prints: first call measured ✓ View your receipt from another terminal: npx inferock-bench receipt --compact If you installed inferock-bench globally, inferock-bench receipt --compact works too. This README uses the npx inferock-bench <cmd> form so one-time users are not stranded. Already have an app? Change exactly two SDK settings: apiKey and baseURL. const client = new YourProviderSdk({ apiKey: process.env.INFEROCK_BENCH_KEY ?? "ibl_your_generated_local_key", baseURL: "http://127.0.0.1:4318", }); Some SDKs use /v1 in the base URL. The dashboard shows the exact value for every configured provider; npx inferock-bench init prints OpenAI and Anthropic constructor snippets. Run npx inferock-bench init to detect OpenAI or Anthropic SDK usage and print the exact SDK change. npx inferock-bench init --patch path/to/client.ts --yes patches simple constructors only when it can update both apiKey and baseURL; otherwise it refuses with a clear message. For Gemini or OpenRouter, use the dashboard's Local app connection snippet or the package README's provider example. Run from source git clone https://github.com/inferock/inferock-bench.git cd inferock-bench pnpm install pnpm -r --workspace-concurrency=1 build node apps/inferock-bench/dist/index.js start Test your loss npx inferock-bench test runs the complete coverage battery through your configured provider scope, on your provider key, so the receipt can show what your provider cost you and which loss surfaces the run actually opened. The checked-in measured baseline powers the estimate, so a configured provider key and priced compatible model are enough to reach the consent step. For the exact formulas behind the receipt, see Paid-loss arithmetic. You see the estimated tokens, estimated dollars, model, suite, baseline, pricing source, and spend cap before any provider call is made. The copy states the price plainly: running the complete test set on the selected provider(s) will cost approximately the displayed amount. If you stop there, the command makes zero provider calls. Interactive runs require you to type RUN; automation must pass the displayed hash with --accept-estimate <hash> because --yes alone is not consent to a changed estimate. In the dashboard, open Advanced options, set Test driver to Agent test, then run the test to use a real coding agent. Agent test currently supports OpenAI and Anthropic runs; use the built-in generator for Gemini and OpenRouter coverage. If the pinned local agent is not installed, the dashboard names the exact npm tarballs, versions, SRI checksums, sizes, source URLs, and local install path before downloading. The agent receives only localhost and an ephemeral local ibl_ key, never your provider key. CLI equivalent: npx inferock-bench test --generator agent. The receipt is run-scoped. It reports spent $X · money loss $Y · time loss Z · invoice-check exposure $E, provider-recognized recovery, bill-bounded recognition gap, the separate invoice-check exposure detail line when applicable, and surfaces watched N of M surfaces your selected providers can open (M varies by provider), with every surface labeled watched-clean, signal, or not-openable. A zero only counts when the surface was watched; unopened surfaces are named as coverage debt, not silently claimed clean. For a priced call that fails the standard and is tied to observed spend or charge evidence, money loss is bill-bounded; provider-recognized can still be $0, and the gap is the difference inside that bill-bounded money ledger. The receipt opens with a one-line plain-English guide to spent dollars, bill-bounded money loss, time loss, and invoice-check exposure. A receipt "failure" is a measurement finding, not necessarily your app crashing. A signal is one finding the benchmark saw, such as a token cross-check. CACHE_DISCOUNT_AT_RISK is shown as invoice-check exposure with "verify your invoice" guidance; it is not summed into money loss or recognition gap. Provider-recognized is the part that appears likely to fit the provider's current credit rules; bill-bounded gaps stay visible instead of being hidden. If no provider key is configured, pricing is unknown, or the token baseline is ever absent or bootstrap-only, the CLI and dashboard fail closed and make zero provider calls. The baseline-degraded state is reported as baseline not measured yet: run `npx inferock-bench test --record-baseline` with explicit consent to produce a real per-task token baseline. The method details are in Coverage test methodology. What your provider doesn't tell you The gap is simple: providers give you totals. They usually do not give you the per-call receipt you would need to prove which answer broke, which retry ran, or which token count changed. We do not think you should have to trust a monthly total. We think every broken answer should leave a trail: what happened, how sure we are, and whether the provider would actually recognize the claim. Illustrative mechanism — not measured data. This is the kind of billing blind spot we built inferock-bench to catch on your own traffic.

Billing

Explore categories

How Product Visit works