Great software wrestles for you
By Aditya Perswal
What does great software do? It wrestles with the hard part so you don't have to.
The CDK from AWS. The primitives of Postgres. The simplicity of Cloudflare. The beauty of PostHog. None of them ask you to hold their complexity. They hold it for you.
We built FlexInference to wrestle with your AI bill.
The cost problem
Cost starts as a line item on an invoice someone skims. A few months later it has its own standing meeting. Eventually it's the reason a good idea doesn't ship.
The feature works, the demo goes well, and the projected bill stops it. Nobody decided any of this. The cost grew while everyone was looking at the product.
And every existing way to cut it makes the product worse somewhere. A cheaper model gives worse answers. A batch API returns answers tomorrow. A trimmed context loses what the model needed to know.
What we built
You tell us how long a request can wait to start: "start_within": "00h-00m-30s". Anywhere from five seconds to ten minutes. We try to start your request on the provider's discounted tier within that time. If it starts in time, you pay about half. If it doesn't, your request runs on the standard tier at full speed and full price.
The model, the answer, and the SDK stay the same. Only the base URL changes.
Across everything running with a deadline now, the median request pays 48% less and gets its first token about 20% later. If a workload can't wait, you don't set a deadline on it.