Artificial intelligence is turning out to be far more expensive than anyone expected, and CFOs at major U.S. companies are now facing a brutal new trade-off: tokens or humans.
That was the picture two enterprise AI CEOs at the center of the buildout described to CNBC this week. Their accounts of what’s happening inside the Fortune 500 paint a sharp picture of the threat that rising costs pose to the AI trade. It’s a risk the market hasn’t yet recognized as it hits record highs and mints new trillion-dollar companies like Micron.
The number one topic for every enterprise right now is overblown AI budgets, Arvind Jain, CEO of enterprise AI company Glean, told CNBC.
“Companies are telling us that their AI budgets are getting exhausted in one month or two months, and these are annual budgets,” he said.
That’s because the cost of AI hasn’t come down the way buyers expected. Rather, it’s gone up. Each new model release from the frontier labs is roughly twice as expensive per token as the one it replaced, putting enterprise AI on what Jain called “an unsustainable path right now.”
“This is the first time ever that I can remember that technology costs the same as people, and you’re making that comparison: choose tech or people,” he said. “We’ve never had that conversation historically, because tech is a fraction of the overall cost of any operating business.”
That growing AI budget, he says, is increasingly coming in lieu of future headcount growth.
Arvind Jain, CEO of Glean, on SaaS Monster stage during day one of Web Summit 2022 at the Altice Arena in Lisbon, Portugal, on Nov. 2, 2022.
Harry Murphy | Sportsfile | Getty Images
Matan Grinberg, CEO of Factory AI, which routes engineering work across every frontier AI model, described the shift as a defined resource allocation problem now playing out inside leadership teams.
“Companies say, hey, if we could optimize one thing, is it the number of employees that we have, or is it the AI spend per employee?” Grinberg said.
Grinberg said companies have moved through three distinct phases in roughly a year. The first involved boards demanding their CEOs do something about AI. Then came so-called tokenmaxxing, or using AI by any means necessary regardless of cost. In the third phase, leadership teams are reassessing their needs when it comes to premium models.
“Do we need to be using Opus-level intelligence for every single task?” Grinberg said. “You just don’t need to.”
Paying more than it pays back
The root of the squeeze is that the technology works but doesn’t yet pay for itself.
“The way AI works today, it’s very powerful, but it’s very inefficient,” Jain said. “The value that AI drives at this point is trailing the cost that businesses are incurring.”
A big part of the problem is inefficiency in picking models. Roughly 95% of enterprise AI usage is still running on the most expensive frontier models, even for tasks that could be handled by cheaper alternatives, Jain said.
There’s a simple fix: routing the easy work to the cheaper tier. Jain said that’s the lowest-hanging fruit.
“You have a 10x savings that you can actually achieve with the right model routing at the front,” he said.
That’s also the pitch behind Factory AI, which automatically sends each task to the model best suited to it. The trick, Grinberg said, is recognizing how rarely a job actually needs the top of the line. He likened the gap between the newest frontier models to two veteran academics.
“Opus 4.7 versus Opus 4.8 is like the difference between a professor who’s been a professor for 13 years versus 15 years,” Grinberg said. “To a lay person, it’s really, really hard to tell the difference.”
The entire AI trade rests on the bet that historic demand will remain, with buyers largely indifferent to cost. But the view from inside the Fortune 500 suggests demand may be far more price-sensitive than the trade assumes.
Read more about what the AI price reckoning means for the valuations of OpenAI and Anthropic, which have built their business models on premium pricing.
