Fireworks AI is fine with helping companies run AI models in their own datacenters, though VP Rob Ferguson argues against doing so.
Ferguson told The Register in an interview that while Fireworks AI, a managed inference provider, supports on-prem AI, “we’re trying to push people more and more away from it because frankly, it’s just the economics.”
The GPU hardware is just so expensive that if you can’t keep it fully utilized, it’s a waste of money, he contends. Ten thousand GPUs, he said, represent a billion dollars of capital.
“You don’t want those things to be sitting idle,” he said.
For companies with particularly sensitive workloads, Ferguson said, Fireworks is happy to work something out – it offers a service called Sovereign for just that purpose and it has partners like Microsoft that can help. But most inference work probably doesn’t need that level of protection, he said.
“The state of the art for everyone is to have a portfolio of models,” he explained. “In fact, I think most people don’t know that the best model in the world’s not Astra. The best model in the world’s not Fable. The best model is literally a portfolio and we can prove it. You are always better routing between a few models than to do one singular model.”
That’s what Fireworks does through Nexus and its API, alongside competitors like Together AI, Baseten, and DeepInfra. As a managed inference platform, Fireworks AI routes inference tasks to whichever model, frontier model subscription, or API is best suited to handle them.
But the company is not just doing naive routing like an AI gateway. It’s evaluating tasks as they come in and directing them to the optimal model.
“We build custom routers,” Ferguson explained. “Our auto model is continuously updating so that we actually go through and we grade the difficulty of the task and we see if there’re affinities that the models have, like Kimi is slightly better at JavaScript than Python, things like this.”
The company’s competitive edge, he contends, is its crack engineering team, which follows from its founders’ work on PyTorch, a widely used open source machine learning library.
“We’re performance experts both on the training and inference side,” he explained. “People I think don’t understand how important, for example, numerics is. So when you use a model, it’s always called a mixture of experts – an expert is a different machine.
“And so you have all these different machines. It actually matters when you’re training that you go back to precisely the right machine, because if you don’t, the other machine will still answer the question, but it’ll answer in a slightly different way. So we have to know how to build the topology to write back the data to the right space. We have to understand floating point precision [to handle very small values]. We have to be able to do all this where we’re routing through a complex fabric through potentially more than one provider depending on what everything is. And so we’re really systems experts.”
Hyperscalers, he argues, will give you a GPU but then it’s your responsibility to put it to use. Fireworks’ advantage involves being able to maximize utilization at scale while maintaining model precision and speed.
Ferguson said AI keeps coming back to economics.
“People are just trying to figure out more and more things to use AI for and [to answer questions like] just how much more can I send to cached models, how much more can I send to open models, when do I need this specialization?” he said. “So we’re constantly working with customers to help them come up with their own evaluation criteria.”
One reason for this, Ferguson explained, is that frontier model companies are not transparent about their caching rates. He said Fireworks was working with a data science company and was caching about 90 percent of the tasks.
“We looked at [Anthropic’s] Opus and it was caching 30 percent of the time,” he said. “It was doing AI rework constantly. And with AI, rework is by far your biggest cost. If it’s not hitting a cache, then your costs are absolutely insane. And so the token cost is not the real cost. It’s the cost per task. And they’re hiding that from you. And so we always tell our customers to be tracking cost per task, because that’s the real number.”
Another factor that companies should consider in their cost calculations, Ferguson suggested, is the value of their data and whether they’re giving that data away for training.
“I think people grossly underestimate the value of their data,” he said.
Fireworks AI has a zero data-retention policy, “but a lot of times,” Ferguson said, “these other companies [do] not, and you should be getting paid for your data. Nobody really knew at the beginning of this current era that these models are memorizers. But now there’s plenty of evidence that these models are. So if they’re going to be using your data, then you should be getting value for it.”
But it’s not just about cost. Speed, said Ferguson, is underappreciated.
“Speed is the thing that people often don’t know they can ask for,” he said. “It’s such a game changer for me and how I work with AI is so different when the responses come back as fast as I ask them.”
What’s the advantage of model ensembling, as the company refers to its model routing activities? A bill that’s at least 50 percent smaller, Ferguson claims.
“So we say six to eight times cheaper,” he said. “The reality is that because a lot of folks have a preference in their tooling, we kinda guarantee that in the case where you’re still using a combo of closed and open models, we can always guarantee that your bill will drop by half right away. Like we’re sure about that. And then the question is can we just get you to use more stuff?” ®