<h3>Helicone AI Gateway</h3> <p><strong>The fastest, lightest, and easiest-to-integrate AI Gateway on the market.</strong></p> <p><em>Built by the team at <a href="https://helicone.ai" target="_blank" rel="noopener">Helicone</a>, open-sourced for the community.</em></p> <p><a href="https://docs.helicone.ai/ai-gateway/quickstart" target="_blank" rel="noopener">🚀 Quick Start</a> • <a href="https://docs.helicone.ai/ai-gateway/introduction" target="_blank" rel="noopener">📖 Docs</a> • <a href="https://discord.gg/7aSCGCGUeu" target="_blank" rel="noopener">💬 Discord</a> • <a href="https://helicone.ai" target="_blank" rel="noopener">🌐 Website</a></p> <h3>🚆 1 API. 100+ models.</h3> <p><strong>Open-source, lightweight, and built on Rust.</strong></p> <p>Handle hundreds of models and millions of LLM requests with minimal latency and maximum reliability.</p> <p>The NGINX of LLMs.</p> <h3>👩🏻💻 Set up in seconds</h3> <h3>With the cloud hosted AI Gateway</h3> <pre>from openai import OpenAI client = OpenAI( api_key="YOUR_HELICONE_API_KEY", base_url="https://ai-gateway.helicone.ai/ai", ) completion = client.chat.completions.create( model="openai/gpt-4o-mini", # or 100+ models messages=[ { "role": "user", "content": "Hello, how are you?" } ] )</pre> <p><em>-- For custom config, check out our <a href="https://docs.helicone.ai/ai-gateway/config" target="_blank" rel="noopener">configuration guide</a> and the <a href="https://github.com/Helicone/ai-gateway/blob/main/ai-gateway/config/embedded/providers.yaml" target="_blank" rel="noopener">providers we support</a>.</em></p> <h3>Why Helicone AI Gateway?</h3> <h3>🌐 <strong>Unified interface</strong></h3> <p>Request <strong>any LLM provider</strong> using familiar OpenAI syntax. Stop rewriting integrations—use one API for OpenAI, Anthropic, Google, AWS Bedrock, and <a href="https://docs.helicone.ai/ai-gateway/providers" target="_blank" rel="noopener">20+ more providers</a>.</p> <h3>⚡ <strong>Smart provider selection</strong></h3> <p><strong>Smart Routing</strong> to always hit the fastest, cheapest, or most reliable option, and always aware of provider uptimes and your rate limits. Built-in strategies include model-based latency routing (fastest model), provider latency-based P2C + PeakEWMA (fastest provider), weighted distribution (based on model weight), and cost optimization (cheapest option).</p> <h3>💰 <strong>Control your spending</strong></h3> <p><strong>Rate limit</strong> to prevent runaway costs and usage abuse. Set limits per user, team, or globally with support for request counts, token usage, and dollar amounts.</p> <h3>🚀 <strong>Improve performance</strong></h3> <p><strong>Cache responses</strong> to reduce costs and latency by up to 95%. Supports Redis and S3 backends with intelligent cache invalidation.</p> <h3>📊 <strong>Simplified tracing</strong></h3> <p>Monitor performance and debug issues with built-in Helicone integration, plus OpenTelemetry support for <strong>logs, metrics, and traces</strong>.</p> <h3>☁️ <strong>One-click deployment</strong></h3> <p>Use our <a href="https://us.helicone.ai/gateway" target="_blank" rel="noopener">cloud-hosted AI Gateway</a> or deploy it to your own infrastructure in seconds by using <strong>Docker</strong> or following any of our <a href="https://docs.helicone.ai/ai-gateway/deployment/overview" target="_blank" rel="noopener">deployment guides here</a>.</p> Launch.Final.1.1.1.mp4 <h3>⚡ Scalable for production</h3> <table> <thead> <tr> <th>Metric</th> <th>Helicone AI Gateway</th> <th>Typical Setup</th> </tr> </thead> <tbody> <tr> <td><strong>P95 Latency</strong></td> <td><5ms</td> <td>~60-100ms</td> </tr> <tr> <td><strong>Memory Usage</strong></td> <td>~64MB</td> <td>~512MB</td> </tr> <tr> <td><strong>Requests/sec</strong></td> <td>~3,000</td> <td>~500</td> </tr> <tr> <td><strong>Binary Size</strong></td> <td>~30MB</td> <td>~200MB</td> </tr> <tr> <td><strong>Cold Start</strong></td> <td>~100ms</td> <td>~2s</td> </tr> </tbody> </table></markdown-accessiblity-table> <p><em>Note: See <a>benchmarks/README.md</a> for detailed benchmarking methodology and results.</em></p> <h3>🎥 Demo</h3> AI.Gateway.Demo.mp4 <h3>🏗️ How it works</h3> <pre><code>┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Your App │───▶│ Helicone AI │───▶│ LLM Providers │ │ │ │ Gateway │ │ │ │ OpenAI SDK │ │ │ │ • OpenAI │ │ (any language) │ │ • Load Balance │ │ • Anthropic │ │ │ │ • Rate Limit │ │ • AWS Bedrock │ │ │ │ • Cache │ │ • Google Vertex │ │ │ │ • Trace │ │ • 20+ more │ │ │ │ • Fallbacks │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Helicone │ │ Observability │ │ │ │ • Dashboard │ │ • Observability │ │ • Monitoring │ │ • Debugging │ └─────────────────┘ </code></pre> <h3>⚙️ Custom configuration</h3> <h3>Cloud hosted router configuration</h3> <p>For the cloud hosted router, we provide a configuration wizard in the UI to help you setup your router without the need for any YAML engineering.</p> <p>For complete reference of our configuration options, check out our <a href="https://docs.helicone.ai/ai-gateway/config" target="_blank" rel="noopener">configuration reference</a> and the <a href="https://github.com/Helicone/ai-gateway/blob/main/ai-gateway/config/embedded/providers.yaml" target="_blank" rel="noopener">providers we support</a>.</p> <h3>📚 Migration guide</h3> <h3>From OpenAI (Python)</h3> <pre>from openai import OpenAI client = OpenAI( - api_key=os.getenv("OPENAI_API_KEY") + api_key="placeholder-api-key" # Gateway handles API keys + base_url="http://localhost:8080/router/your-router-name" ) response = client.chat.completions.create( - model="gpt-4o-mini", + model="openai/gpt-4o-mini", # or 100+ models messages=[{"role": "user", "content": "Hello!"}] )</pre> <h3>From OpenAI (TypeScript)</h3> <pre>import { OpenAI } from "openai"; const client = new OpenAI({ - apiKey: os.getenv("OPENAI_API_KEY") + apiKey: "placeholder-api-key", // Gateway handles API keys + baseURL: "http://localhost:8080/router/your-router-name", }); const response = await client.chat.completions.create({ - model: "gpt-4o", + model: "openai/gpt-4o", messages: [{ role: "user", content: "Hello from Helicone AI Gateway!" }], });</pre> <h3>Self-host the AI Gateway</h3> <p>The option might be best for you if you are extremely latency sensitive, or want to avoid a cloud offering and would prefer to self host the gateway.</p> <h3>Run the AI Gateway locally</h3> <ol> <li>Set up your <code>.env</code> file with your <code>PROVIDER_API_KEY</code>s</li> </ol> <pre>OPENAI_API_KEY=your_openai_key ANTHROPIC_API_KEY=your_anthropic_key</pre> <ol> <li>Run locally in your terminal</li> </ol> <pre>npx @helicone/ai-gateway@latest</pre> <ol> <li>Make your requests using any OpenAI SDK:</li> </ol> <pre>from openai import OpenAI client = OpenAI( base_url="http://localhost:8080/ai", # Gateway handles API keys, so this only needs to be # set to a valid Helicone API key if authentication is enabled. api_key="placeholder-api-key" ) # Route to any LLM provider through the same interface, we handle the rest. response = client.chat.completions.create( model="anthropic/claude-3-5-sonnet", # Or other 100+ models.. messages=[{"role": "user", "content": "Hello from Helicone AI Gateway!"}] )</pre> <p><strong>That's it.</strong> No new SDKs to learn, no integrations to maintain. Fully-featured and open-sourced.</p> <p><em>-- For custom config, check out our <a href="https://docs.helicone.ai/ai-gateway/config" target="_blank" rel="noopener">configuration guide</a> and the <a href="https://github.com/Helicone/ai-gateway/blob/main/ai-gateway/config/embedded/providers.yaml" target="_blank" rel="noopener">providers we support</a>.</em></p> <h3>Self hosted configuration customization</h3> <p>If you are self hosting the gateway and would like to configure different routing strategies, you may follow the below steps:</p> <h3>1. Set up your environment variables</h3> <p>Include your <code>PROVIDER_API_KEY</code>s in your <code>.env</code> file.</p> <p>If you would like to enable authentication, set the <code>HELICONE_CONTROL_PLANE_API_KEY</code> variable as well.</p> <pre>OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... HELICONE_CONTROL_PLANE_API_KEY=sk-...</pre> <h3>2. Customize your config file</h3> <p><em>Note: This is a sample <code>config.yaml</code> file. Please refer to our <a href="https://docs.helicone.ai/ai-gateway/config" target="_blank" rel="noopener">configuration guide</a> for the full list of options, examples, and defaults.</em> <em>See our <a href="https://github.com/Helicone/ai-gateway/blob/main/ai-gateway/config/embedded/providers.yaml" target="_blank" rel="noopener">full provider list here.</a></em></p> <pre>helicone: # Include your HELICONE_API_KEY in your .env file features: all cache-store: type: in-memory global: # Global settings for all routers cache: directive: "max-age=3600, max-stale=1800" routers: your-router-name: # Single router configuration load-balance: chat: strategy: model-latency models: - openai/gpt-4o-mini - anthropic/claude-3-7-sonnet rate-limit: per-api-key: capacity: 1000 refill-frequency: 1m # 1000 requests per minute</pre> <h3>3. Run with your custom configuration</h3> <pre>npx @helicone/ai-gateway@latest --config config.yaml</pre> <h3>4. Make your requests</h3> <pre>from openai import OpenAI import os helicone_api_key = os.getenv("HELICONE_API_KEY") client = OpenAI( base_url="http://localhost:8080/router/your-router-name", api_key=helicone_api_key ) # Route to any LLM provider through the same interface, we handle the rest. response = client.chat.completions.create( model="anthropic/claude-3-5-sonnet", # Or other 100+ models.. messages=[{"role": "user", "content": "Hello from Helicone AI Gateway!"}] )</pre> <p>For a complete guide on self-hosting options, including Docker deployment, Kubernetes, and cloud platforms, see our <a href="https://docs.helicone.ai/ai-gateway/deployment/overview" target="_blank" rel="noopener">deployment guides</a>.</p> <h3>📚 Resources</h3> <h3>Documentation</h3> <ul> <li>📖 <strong><a href="https://docs.helicone.ai/ai-gateway/introduction" target="_blank" rel="noopener">Full Documentation</a></strong> - Complete guides and API reference</li> <li>🚀 <strong><a href="https://docs.helicone.ai/ai-gateway/quickstart" target="_blank" rel="noopener">Quickstart Guide</a></strong> - Get up and running in 1 minute</li> <li>🔬 <strong><a href="https://docs.helicone.ai/ai-gateway/config" target="_blank" rel="noopener">Advanced Configurations</a></strong> - Configuration reference & examples</li> </ul> <h3>Community</h3> <ul> <li>💬 <strong><a href="https://discord.gg/7aSCGCGUeu" target="_blank" rel="noopener">Discord Server</a></strong> - Our community of passionate AI engineers</li> <li>🐙 <strong><a href="https://github.com/helicone/ai-gateway/discussions" target="_blank" rel="noopener">GitHub Discussions</a></strong> - Q&A and feature requests</li> <li>🐦 <strong><a href="https://twitter.com/helicone_ai" target="_blank" rel="noopener">Twitter</a></strong> - Latest updates and announcements</li> <li>📧 <strong><a href="https://helicone.ai/email-signup" target="_blank" rel="noopener">Newsletter</a></strong> - Tips and tricks to deploying AI applications</li> </ul> <h3>Support</h3> <ul> <li>🎫 <strong><a href="https://github.com/helicone/ai-gateway/issues" target="_blank" rel="noopener">Report bugs</a></strong>: Github issues</li> <li>💼 <strong><a href="https://cal.com/team/helicone/helicone-discovery" target="_blank" rel="noopener">Enterprise Support</a></strong>: Book a discovery call with our team</li> </ul> <h3>📄 License</h3> <p>The Helicone AI Gateway is licensed under the <a>Apache License</a> - see the file for details.</p> <p><strong>Made with ❤️ by <a href="https://helicone.ai" target="_blank" rel="noopener">Helicone</a>.</strong></p> <p><a href="https://helicone.ai" target="_blank" rel="noopener">Website</a> • <a href="https://docs.helicone.ai/ai-gateway/introduction" target="_blank" rel="noopener">Docs</a> • <a href="https://twitter.com/helicone_ai" target="_blank" rel="noopener">Twitter</a> • <a href="https://discord.gg/7aSCGCGUeu" target="_blank" rel="noopener">Discord</a></p>
