Token-Maxxing and Inference Ops: The New FinOps Frontier

A Head of Product at a major sportswear retailer has a brilliant idea: let’s build an app that sales staff on the shop floor can have on their tablets, and ask their questions there and then, where they serve customers. 

They set about building. In order for the app to answer questions, it needs to have the information from the 2026 Spring/Summer Catalogue, a mammoth manual, let’s say 300k tokens. 

Then the developer loads the huge PDF into the system instructions of the LLM and builds a chat interface over it. 

When the sales assistant in the store asks about Adidas Gazelles, the LLM reads the entire 300k-token book, finds the answer on page 89 and page 607, and delivers a response. The staff loves the POC, and it’s pushed for production.

The app gets rolled out to all stores - the 5000 sales assistants ask it 10 questions a day each. The app is processing 15bn input tokens a day, creating something that never happened when it was a POC: a $75k daily run rate. 

Production environments are where AI budgets expand horribly. A working prototype can become a 75k/day liability overnight. And that’s where the CFO shuts it right down. 

Not only is the hero app dead, but the CFO issues strict guidelines and a new era of token-rationing in the company begins. 

This is not a drill - average enterprise AI budgets expanded from roughly $1.2 million annually in 2024 to about $7 million in 2026. Uber recently became a cautionary tale when its AI spending spiral made headlines for the wrong reasons. And it won't be the last.

Welcome to 2026 and the token cost crisis. 

Why Token Costs Are Suddenly Huge Financial Exposure

What we just saw was classic context stuffing. Brute-forcing that bypasses complex data engineering and is essentially the quickest way to get to a working prototype.

Context stuffing is just one of the symptoms of the bigger problem of token over-consumption due to lazy architecture and bad management practices. This is something organisations need to pay attention to as they transition from having agents in PoC to production environments. 

This is where poor architecture can become wildly expensive and opens up a new era of tech debt (and potentially financial debt!).

In addition to context-stuffing, other token-bonfires include:

Cache Blindness

The same questions/instructions are sent again and again without leveraging middleware or native caching layers. 

Token Amplification

Where agents talk to each other in multi-agent workflows, e.g. agent A sends a 5k token history to agent B who sends 5k back. The context grows quadratically unless there is summarisation or state-pruning. 

Poor state management forces the LLM to re-process large conversation histories with each query in the conversation, while wasting output tokens on heavy JSON schemas. 

‘Token-Maxxing’ 

This is the dubious trend where companies and individuals measure AI productivity by the sheer volume of tokens consumed; it's essentially the ‘lines of code’ KPI for the agentic era. 

It’s the ‘tyranny of metrics’ - something that the developer community is becoming increasingly vocal about, with stories of code-slop and usage becoming a proxy for value.

Sonny Rivera, principal data and AI strategist and former Chief Product Officer of TIFIN, advocates for ‘inference opps’ where good practices keep token burn manageable, and token-maxxing is discouraged. 

The Three Pillars for ‘Inference Finops’

At ThoughtSpot, we believe that ‘inference finops’ is a mandatory engineering discipline to transition from prototype to sustainable production. 

Here are our pillars of ‘inference finops’ in distinct architectural layers:

1. Data Architecture and RAG Standard

Transitioning from massive context windows to clean retrieval-augmented generation (RAG) and using ‘chunking’ to lower the LLM processing burden and get the same answers for a fraction of the costs.

Let’s go back to the earlier example of the sports store. If, instead of rushing to production, the engineering team architect a RAG pipeline. They take the Spring/Summer Catalogue and chop it up into bits - a process called ‘chunking’. 

So the Adidas Gazelles become a chunk. The Nike Airs become a chunk, and the return policy becomes a chunk. These chunks are converted into vector embeddings and indexed in a specialized vector store, such as the enterprise PostgreSQL using the vector extension. 

Now, when the sales assistant asks the question, the system acts like a librarian and sends the LLM to the right paragraphs and has it answer based only on those two using maybe 80 tokens. 

The 50k daily queries from all the stores come to a cost of $200 a day (down from $75k!) and the assistant gets the exact same answer. 

2. Model Routing to Move From Data Shortcuts to Compute Efficiency

You don't need the power and glory of Mythos to help you understand why there’s excess inventory of those fuzzy pink jackets in the warehouse. 

Rather than hardwiring an application to a single LLM, an agent harness framework like LangChain or LlamaIndex can intercept the prompt, classify its complexity, and dispatch it to an open-source model like Mistral that’s free to run, or internally hosted small models running on your own cloud instances.

For example, ThoughtSpot acts as an optimised, built-in harness—defaulting to a combination of Opus 4.5 and GPT-5.1 for complex analytical questions and dropping down to Haiku for summarizing an insight, ensuring you use the most cost-efficient LLM for the task at hand. And yes, customers can also bring their own LLM. 

So if we take the retail example again, if the router saw a prompt that asked it to do something simple like run a search, it would route to a more affordable model. 

If the prompt was more complex and required reasoning, e.g., ‘which of these shoes are better for a customer with plantar fasciitis, it would route it to something like Opus 4.5.

A piece of insight for CDAIOS: We tend to focus on the price per 1k tokens to decide what the most cost-effective model is. Databricks proved this completely wrong in their recent article here - they found that models with a cheaper list price, when run over end-to-end tasks, actually ended up being more expensive due to the way they work and consume.

3. Semantic Caching 

Semantic caching, unlike traditional caching, caches based on meaning rather than exact character matches. So it not only remembers the exact question, it remembers similar questions too. 

So, ‘what is Q2 revenue?’ and “what is this quarter's revenue?’ are treated as the same question. And you pay only once. 

Again, there is a mode of action that relies on interception from specialist tools; instead of sending the raw string to the database, it’s converted and scored, and a pre-stored LLM response from the database is sent back. 

Implemented at the enterprise API gateway level, specialist middleware intercepts the prompt, converts it to a vector to score against historical queries, and can match to a previous, pre-stored response (if there is one) without incurring token spend.

There is a misconception that native query caching (in, for example, Anthropic and OpenAI models) negates the need for semantic caching. This is not correct. Semantic caching and native prompt caching are different tools doing different jobs. 

Native caching stores the static prefix (for example, in the sports store example, the 300k token catalogue) so you get a ‘discount’ on input tokens when you call the model. The user's questions are not cached. You need semantic caching for this.

It’s worth remembering that semantic caching handles unstructured text data well but struggles with numbers. Spotter Semantics uses a tokenised search hard-linked to the database schema, which is highly optimised for query caching, resulting in up to 90% lower token spend. 

The Metadata Moat and Predictable Search Architectures

Good practices around metadata can turn a 100k token query into a 1k token query. Good metadata (data about data) attached to your documents means that an LLM has to scan fewer documents to get to an answer - it can hone in on relevant documents rather than have to scan many docs trying to find a ‘needle in a haystack.’ 

As an example in the sporting goods store, let’s consider the question ‘what’s our returns policy on Nike Air if the customer lost the receipt?’ 

Without good metadata, the LLM has to scan everything that mentions returns, receipts, Nike, etc. With metadata pre-filtering (e.g., Region: UK, channel = in-store, doc type = returns policy, etc.) lets the LLM zone in on the right docs using far fewer tokens.

ThoughtSpot’s architecture shows us a way to do this: Spotter’s NLQ analytics does not use an LLM for text-SQL generation; instead, it uses a tokenised search. 

Because ThoughtSpot’s tokens are linked to a physical database schema, this mode of action is highly deterministic; it produces a very efficient SQL that yields a much lower and more predictable token cost.

A Procurement Blindspot the CDAIO Must Shed Light On

At ThoughtSpot, we know from many frank conversations with our customer CDAIOS, that in many organizations, procurement functions are negotiating contracts with cost bases that are rapidly changing. 

It will be the job of the CDAIO to be clear about these potential risks to the board and the wider organisation. We expect that tokens will eventually just come under the cost control of the CFO, the same way other things do, like travel spend. 

The CDAIO will need to help establish this and ideally, get a handle on the framework before the CFO comes looking for it!

In fact, EY’s recent white paper advocates for hiring a Head of Agent Economics. You may not need to take quite such a big step, but certainly CDAIOs need to be fully on top of this space. 

The Inference Finops Playbook: A Checklist for CDAIOs

• Audit the sandbox - understand where dev teams are context-stuffing in pilots and get a clear view of where this could potentially turn into production liabilities.

• Set architectural standards that are requirements for production. Mandate chunking architectures and production deployment standards.

• Implement a routing framework to ensure the right model is used for the right task.

• Establish a partnership between the CDAIO, CFO, and procurement to ensure contracts are vetted for token liabilities before they are signed off.

Token-Maxxing: Frequently Asked Questions

1. What Is Token-Maxxing, and Is It Actually a Good Thing?

No. Despite the name, token-maxxing isn't a strategy; it's a symptom. It's what happens when teams (or leadership) start treating raw AI token consumption as a proxy for productivity or value, the same trap "lines of code" was for developers a decade ago. High token usage can just as easily mean bloated context, redundant calls, and lazy architecture as it can mean real output.

2. How Do I Know If My Company Is Token-Maxxing Without Realizing It?

Pull your production invoices and compare them against sandbox costs. If your engineering team can't explain the gap, you already know. Then ask how they plan to fix it: "use it less" is not the right call, because this isn't a volume problem. Start with a sandbox audit. Most token-maxxing traces back to a pilot that went to production before the architecture got a second look.

3. What's the Difference Between Token-Maxxing and Inference FinOps?

Token-maxxing measures success by volume consumed. Inference FinOps measures success by outcome per token spent, and treats token burn as an engineering discipline, not a productivity metric. One rewards the behavior that inflates your bill; the other is the fix for it.

4. Who Should Own Inference Cost Governance: Engineering, Finance, or the CDAIO?

Right now, usually nobody. Token spend has grown fast enough that it needs the same kind of ownership travel or cloud spend already has, and the CDAIO is best positioned to bridge the technical architecture side with the CFO and procurement before contracts get signed, not after the bill arrives.

5. Can I Fix Token-Maxxing Without Ripping Out My Existing AI Stack?

Yes. This is architectural discipline, not a rebuild. Moving from context-stuffing to RAG and chunking, adding model routing so simple queries don't hit your most expensive model, and layering in semantic caching can all be retrofitted onto an existing stack without starting over.