Question 1

What is FlexInference?

Accepted Answer

A deadline-aware LLM router across OpenAI, Gemini, and Anthropic. It's OpenAI-compatible, so you keep your SDK and point the base URL at us. Run it two ways: bring your own provider key, or let us hold the keys with Managed. Add a start_within deadline and we race a cheaper tier, then hand you back the model you asked for. The race runs on both now. BYOK is free at any volume, no card.

Question 2

What is start_within?

Accepted Answer

A field every request needs, either in the body or as a default on the key. Set it to default, priority, auto, or a duration like 00h-00m-30s. A duration races a cheaper tier for the model you picked, then escalates to standard if the race loses. That race runs on both BYOK and Managed, for OpenAI and Gemini. On Managed, a claude duration between 3 and 10 minutes races Claude's cheaper tier instead. Can't find cheaper in time? You still get your response, we serve it standard. Using a tool that can't add a field? Put the deadline on the key when you create it, and the body still wins whenever it sends one.

Question 3

How much does FlexInference save?

Accepted Answer

Across all our traffic, the flex race cuts blended cost by about 49%, for roughly 20% more time to first token. You also skip the upcharge other routers slap on default, auto, and priority requests. That upcharge annoyed us, so we don't add it. Your numbers depend on your models, providers, and the deadline you set.

Question 4

What is FlexInference great for?

Accepted Answer

Can we say everything? As a normal router it's free for BYOK and doesn't strip parameters you set to force a success. So when you want to know that what you asked for is what you got, it's great. As for the flex race, you set start_within to a deadline. It's great for tasks that aren't latency sensitive. That can be data processing, evals, LLMs as judges, browser agents, or summarization. Anything that can afford to wait a few seconds for huge cost reductions.

Question 5

What if the flex race doesn't win?

Accepted Answer

You still get your answer. Can't find cheaper inference before your deadline? We escalate to a normal request, before flex writes a single output token, so we never charge you for a losing try. Either way it's one billable event, never two.

Question 6

Which providers and models do you support?

Accepted Answer

The supported OpenAI, Gemini, and Anthropic models, though only the flex-capable ones run the race. A model we do not carry is refused by name with model_not_carried rather than quietly substituted. Pin a route with a provider array, openai, google, or anthropic. Want the same model on your own cloud key? Vertex and Bedrock serve it BYOK, standard tier, no flex. Managed serves OpenAI, Anthropic, and Gemini direct, plus Cloudflare Workers AI for open-source models.

Question 7

BYOK or Managed, which should I pick?

Accepted Answer

Both route the same way. What changes is who holds the keys and how you pay. Pick BYOK if you already have provider keys. You bring the key, model usage bills to your provider, and our serving is free at any volume, with no subscription, allowance or overage. Pick Managed to skip provider signup. We serve it on our own pooled provider credentials and model cost draws from a wallet you load. The flex race runs either way for OpenAI and Gemini, and Claude's race runs on Managed only.

Question 8

Do you have an MCP server?

Accepted Answer

Yes. Point any MCP client, Claude or Cursor, at https://mcp.flexinference.com/mcp. Two docs tools work with no login: search the docs, look up any error code. After a one-time OAuth sign-in, an agent can read your usage and savings, list your API keys, and check billing. Reads only, creating or revoking a key stays in the dashboard. Each of those sits behind its own scope. We never take a raw provider key over MCP, add those on the dashboard.

Question 9

Are your errors built for coding agents?

Accepted Answer

Yes. Errors come back in the shape of the endpoint you called, OpenAI, Anthropic, or Google, so your SDK parses them unchanged. The OpenAI shape adds a machine-readable code, the offending field, and a doc_url. Each message says what broke, the rule you hit, and the exact fix with an example. Usually you just paste it to your agent and it fixes on the first try.

Question 10

Do Anthropic models get the cheaper inference search?

Accepted Answer

Yes, on managed keys. Send a duration start_within between 3 and 10 minutes and we race Claude's cheaper tier against that clock. A win bills you half and a miss serves you standard at full price. Current constraints put that floor at 3 minutes, so it suits backend services and data pipelines rather than chat. No streaming, anthropic direct route only. Send a tier name instead when you need an instant start.

Question 11

Do you support caching?

Accepted Answer

Yes, your provider's own prompt caching, passed straight through and priced just as they price it. Our own semantic caching is coming soon.

Question 12

How do you handle provider failures?

Accepted Answer

Two knobs. Fallback: hand us an ordered list and we move down it when a provider errors or times out. If the first is flex-capable, the race still runs. Retries: set a count like 2, pick linear or exponential backoff, toggle jitter, we handle the timing. On BYOK, Vertex, Bedrock, and Foundry all work as fallbacks today. Managed falls back within its own served providers.

Question 13

Do you support automatic model selection?

Accepted Answer

Not yet, coming soon. The start_within value auto is OpenAI's service tier, not this.

Question 14

What does FlexInference cost?

Accepted Answer

Two models, two ways to pay. BYOK: free, unlimited, no card. You bring your own key, so model usage bills to your provider with zero markup from us, and FlexInference charges nothing at all. Managed: you pay model cost, zero markup, from a wallet you top up. We add one flat 10% fee per top-up. Top up $50, we charge $5. No per-request charge either way.

Question 15

Do I need a credit card?

Accepted Answer

No. BYOK never needs a card: serving is free and unlimited. A card only comes into play if you switch a provider to Managed and top up the wallet that pays for model usage.

Question 16

What are Managed Keys?

Accepted Answer

We run the provider keys so you don't have to. Load your wallet and we serve your traffic on our own pooled provider credentials. You pay model cost with no markup, plus one flat fee when you top up. Ask for a deadline and we race a cheaper tier for the OpenAI or Gemini model you picked, and drop to standard if the race loses. A Claude deadline of 3 to 10 minutes races Claude's cheaper tier the same way. No OpenAI, Anthropic, or Google account needed on your side.

Question 17

What does Managed cost?

Accepted Answer

Model cost passes straight through at the provider's rate, no markup from us. Our only charge is a flat 10% fee when you add funds. Add $50, the fee is $5. No per-request charge.

Question 18

Is there a request limit on Managed?

Accepted Answer

No. Managed requests have no monthly cap and no per-request charge. You pay model cost plus the one top-up fee. When the wallet runs dry Managed pauses with a 402 until you add funds. We warn you first, around $5 left.

Question 19

Whose provider account serves my Managed traffic?

Accepted Answer

Your own. Every Managed workspace gets its own encrypted credential, held by us and shared with no one else. Sharing one provider account across customers is what annoyed us about other routers, so we don't. The one exception is Cloudflare Workers AI, which runs on our in-network account with your traffic tagged to your workspace.

Question 20

Which providers can Managed serve?

Accepted Answer

OpenAI, Anthropic, and Gemini direct, served on our pooled provider credentials. Plus Cloudflare Workers AI for open-source models, run in-network at the edge for low latency. Bedrock serves Claude models on Managed too, and Vertex serves Gemini, each on our pooled credentials for that route.

Question 21

Can you mask sensitive data before it reaches the provider?

Accepted Answer

Yes. Flip on PII masking per request and we scrub emails, phone numbers, card numbers, and more before the request leaves the edge for the provider. It changes only the copy we forward upstream.

Question 22

Can I get my Managed balance back?

Accepted Answer

Yes, the cash. Any cash you haven't spent goes back to your card. Promo credits we granted don't. We spend cash first and only touch credits after, so a refund never hands back a credit.

Question 23

How do promo credits work on Managed?

Accepted Answer

Credits are balance we add to your wallet free, say to get you started. They spend after your cash, and they're not refundable. Everything else about them works like cash.

Question 24

Do you store my requests or responses?

Accepted Answer

Depends on the mode. BYOK, never. Your request passes through memory to be translated and forwarded, then it's gone. No prompt or response content lands in our database, logs, or analytics, just metadata like token counts and cost. Your provider key is encrypted at rest. Managed stores nothing either, until you opt in. Trace storage is off by default. Turn it on and we store the full request and response, encrypted, so you can read them in your dashboard logs. Only your workspace can, decrypted server-side behind your own login. We hold the key, so it's trust-us, not zero-knowledge.

Question 25

Does FlexInference change my model, prompt, or output?

Accepted Answer

No. We never swap your model or silently drop a parameter we cannot serve: an unsupported one is refused by name rather than ignored. The one thing we vary is the tier that serves your model, when that makes the same model cheaper. Audit any request and see for yourself.

一个将成本降低49%的路由器

将成本减半的方法

亲自试用

我们从不更换您的模型，仍能为您节省开支。

无时间预算

有时间预算（flex 竞速）

旨在让您无需在凌晨 2 点调试

300 个城市间 3 毫秒路由。

我们快速且明确地失败。

您现有的客户端无需更改即可工作。

您的代理可自行修复的错误。

了解您能节省多少

自带密钥，或由我们为您管理。

另外三种降低成本的方法

无需学习新的 SDK 语义。

常见问题