Strategy & Transformation
How to Calculate ROI for AI and Software Modernization
A practical, finance-ready framework for comparing AI and modernization investment with the current-state cost, realistic benefits, delivery risk and time to value.
operationalwrocław--:-- cet
∕ insights / llm-cost-optimization.md
LLM Cost Optimization
LLM cost optimization guide: find the real drivers of your AI spend and cut them without hurting quality, security or reliability.
>_article.meta

Your AI feature shipped in the spring. By autumn it's the most-used thing in the product, the support team leans on it daily, and the inference invoice has turned into a line item your finance lead forwards with a question mark next to it. Nothing looks broken on the dashboard. Revenue just grew slower than the spend.
That gap is the whole problem. Teams measure LLM spend per token while the business runs on outcomes. A setting that halves your token price and doubles your retries has made the product more expensive, not cheaper. Cost per successful outcome is the only number that tells the truth, and every lever below gets judged against it.
The way out isn't one clever trick. It's a sequence: find where the money leaks, fix a quality floor you refuse to cross, then pull levers in order of size while a scorecard watches quality and spend together.
Think about a delivery company. The price per parcel matters far less than how many van trips it takes to land each parcel. A cheap courier who needs three attempts per address costs more than a slightly pricier one who gets it right first time. Inference bills behave the same way, which is why a token-price dashboard can show steady improvement while the monthly total climbs.

Long prompts are the quiet one. A system prompt that grew across eight sprints, plus five retrieved chunks where two would do, plus a paragraph of formatting instructions, gets billed on every call. Verbose outputs are the twin: an answer that could be three lines arrives as nine, and output tokens are usually the expensive half of the transaction.
Then there's the premium model doing grunt work. Classification, extraction, routing decisions, sentiment labels. Work a smaller tier handles, billed at flagship rates because nobody revisited the config after the prototype shipped.
| Cost driver | Where it hides | First lever | Warning sign |
|---|---|---|---|
| Token volume | Long system prompts, bloated context, verbose responses | Trim prompts and cap output length | Tokens per request rising with no feature change |
| Model tier | Premium model handling classification or extraction | Route simple tasks to smaller models | Premium model serving most requests |
| Retries and agent loops | Failed tool calls, repeated reasoning steps | Set step limits and fix the failing tool | Cost per completed task rising while success rate stays flat |
| Cache misses | Identical or near-identical questions sent again | Prefix and semantic caching with freshness rules | Low cache hit rate on repeated questions |
| Context length and KV cache | Long documents and large batches held in memory | Context trimming and KV-cache compression | Latency and GPU count rising with concurrency |
| Human review and rework | Wrong outputs sent to people for correction | Better evaluations and guardrails | Escalation rate growing month on month |
| Idle or over-powered hardware | Private GPUs running at full power around the clock | Power caps and right-sized batches | Low utilization on owned GPUs |
Attribution comes before optimization. Tag every request with feature, model, tenant, token counts, retry cause and cache status, the way a serious agent observability practice wires tracing into the gateway instead of bolting it on later. Without those tags, one route getting cheaper and another getting busier average out into a figure nobody can act on.
Start here: rank the seven drivers by their share of last month's spend, then work down the list. The most familiar driver is rarely the largest one.
Cost is a per-outcome question wearing a per-token disguise. Fix the driver that owns the biggest slice, re-measure, then move to the next.
Say a SaaS team swaps the model behind its ticket-reply drafting feature for a cheaper one. The provider invoice drops by a third inside a week, and the result goes into the monthly review as a win. Then the numbers that matter land: agents accept fewer drafts, rewrite more of them, and escalate the tricky ones to a human. Cost per resolved ticket went up.
Cheaper tokens are an input, not an outcome. A request gets more expensive through retries, tool calls, failed agent trajectories, human review, monitoring and quality regressions that create rework, as the Snowflake analysis of AI cost optimization sets out. The same source notes that semantic caching can cut API calls by up to 68.8% in evaluated query categories, which is a real saving and also an incomplete one. Those savings mean little unless the stale-answer rate and false-match rate sit next to them. Ask for both numbers before you accept any caching win.
Add every layer: model tokens, tool calls, retries, human review time, monitoring, infrastructure. Divide by successful outcomes in the same window. A resolved ticket, a completed workflow, an accepted recommendation, a paid conversion. That's the denominator, and it's the number your CFO will trust more than a token chart.
| Metric | What it reveals | Healthy direction |
|---|---|---|
| Cost per successful outcome | Real unit economics per resolved ticket or completed workflow | Falling while success rate holds |
| Success rate | Whether cheaper settings still get the job done | Stable or rising |
| p95 latency | Slowdowns that drive retries and user drop-off | Within the agreed latency target |
| Escalation or human-review rate | Hidden labour cost of weaker outputs | Flat or falling |
| Retries per task | Waste from loops and failed calls | Falling |
| Cache hit rate and stale-answer rate | Whether caching saves calls without serving wrong answers | High hit rate with low stale-answer rate |
| Joules per generated token | Energy cost of self-hosted inference at a given power cap | Falling at the chosen power cap |
Set the quality floor before you test anything. Pick the accuracy, latency and escalation thresholds you won't cross, write them down, and circulate them. Any proposal that reports tokens saved and nothing else gets sent back for the missing half.
The test I apply: if a cost change can't be shown on the scorecard at the end of the month, it hasn't been proven. It's been hoped for.
Review it monthly. Never accept a saving the scorecard can't confirm, and never accept a quality drop the scorecard does confirm, no matter how good the invoice looks.
The price of capability has collapsed, which changes the decision entirely. Stanford's 2025 AI Index measured the cost of querying models at GPT-3.5-level performance, defined as 64.8% accuracy on MMLU, at $20.00 per million tokens in November 2022. By October 2024, Gemini-1.5-Flash-8B reached the same benchmark level at about $0.07 per million tokens, a reduction of more than 280 times in roughly 18 months. The same report found inference prices fell between 9 and 900 times per year, depending on the task.
So the question isn't which model is best. It's which model is the cheapest one that clears your quality bar for this task today. A tier decision made last year is probably too expensive for you now, and the efficiency curve keeps moving.
| Task type | Starting tier | Fall back to stronger model when |
|---|---|---|
| Classification | Smaller tier | Accuracy misses the quality floor on edge cases |
| Extraction | Smaller tier | Required fields are missing or malformed in the evaluation set |
| Summarization and drafting | Smaller tier | Reviewers reject or heavily edit the outputs |
| Multi-step reasoning and agent planning | Stronger model | Not applicable, starts on the stronger tier |
| High-stakes outputs | Stronger model with human approval | Never routed down without sign-off |
The routing rule is boring and it works. Try the smaller model first, score the output against your evaluation set, fall back only when the check fails. Keep validation, lookups and formatting in ordinary code, because a deterministic function that costs nothing shouldn't become a model call that costs money and fails occasionally.
Routing isn't free either. It adds fallback logic, evaluation runs and monitoring, and a cheap model that needs two retries to get there can cost more per outcome than the expensive one did. Compare API spend against the monthly cost of GPUs you'd run yourself at your real utilization before you commit, since self-hosting only wins when traffic stays steady and high enough to keep hardware busy.
We tell clients plainly when a custom model or a local LLM isn't needed. Most business outcomes are reached with existing models, good data access and well-designed workflows, and we use AI where it creates measurable value while keeping conventional software where that's simpler, safer or cheaper. A custom model earns its budget with evidence from your own evaluation set, not with ambition.
A support bot answers the same five questions a few hundred times a day: password reset, invoice copy, delivery timing, integration limits, cancellation policy. Every one of those answers gets recomputed and paid for again. That's the easiest money in the whole system, and it's also where teams get burned.
Start with prefix caching. If your system prompt, tool schemas or document header are stable across calls, that shared prefix can be reused instead of reprocessed. Recent systems research on inference efficiency makes the limits clear: as context windows and model sizes grow, inference is limited less by arithmetic and more by moving data around, including model weights and the KV cache, and prefix caching cuts repeated computation sharply only when traffic carries a sufficiently stable shared prefix. Shuffle your system prompt on every request and the hit rate collapses.
Semantic caching goes further and returns a stored answer for a similar question. That's where false matches and stale responses enter.
Prompt hygiene is the unglamorous sibling of caching. Remove instructions that duplicate each other, retrieve fewer and better chunks, cap the maximum output tokens, and move tool definitions out of the context when the current turn doesn't need them.
Then look at batching. Nightly enrichment, report generation, bulk tagging, backfills. None of that needs an interactive lane, so run it in lower-priority batch windows where throughput matters and latency doesn't.
Weekly habit: sample a handful of cache hits per layer and have a person read them. Hit rate tells you the cache is working. The sample tells you it's still correct.
Retries hide better than any other driver. Each failed call looks like a normal line on the invoice, so the bill grows and nobody can point at the cause. Instrument first: tag every retry and tool call with its trigger, whether that's a timeout, a rate limit, a malformed response or the agent re-running a reasoning step it already finished.
Three fixes, in this order.
The warning sign is a rising cost per completed task with a flat success rate. That combination means you're paying for loops, not results. Give the number a named owner, because a loop limit that trims useful reasoning shows up as falling success, and only the scorecard tells you which side of the trade you landed on.
For teams running private or self-hosted models, the GPU bill arrives whether requests are useful or not. Cost control depends on keeping memory traffic, energy use and latency aligned with the workload. During decoding, models move data between memory and compute, and a 2024 IEEE study of Llama 2-7B inference identifies memory-bound decoding as a major source of latency and energy waste.
Weight-only quantization stores parameters at lower precision, reducing memory traffic. In the study's aggressive weight-only case, energy consumption fell by 4.6 times, and the main bottleneck shifted from fetching weights to the attention mechanism. That is a meaningful infrastructure gain only if the model still produces successful outcomes on your tasks. Validate quality against representative requests before changing production settings.
Power caps offer a different trade-off. Experiments on LLaMA models across NVIDIA V100 and A100 GPUs, documented in this power-capping study of LLM serving, found that lowering GPU power from 250 W to 175 W increased average inference time by 6.7% while reducing total energy use by 23.21%. Batch jobs and requests with relaxed latency targets can often tolerate that exchange. Live chat may not.
| Lever | Reported effect | Trade-off | Best fit |
|---|---|---|---|
| Weight quantization | Lower energy use in an aggressive weight-only case study on Llama 2-7B | Quality must be validated on your own tasks | Memory-bound decoding workloads |
| Power capping | Lower energy use with longer inference time in tests across V100 and A100 GPUs | Slower individual requests | Batch jobs and relaxed-latency traffic |
| KV-cache quantization or pruning | Can improve throughput under high concurrency and long context | Quality varies by workload | Long-context, lower-risk workloads |
| Combined efficiency techniques | Quantization, power caps and cache compression can compound energy savings; measure the result on your hardware | Results depend on batch size, hardware and framework | Systems with enough traffic to benchmark |
| Right-sized batching | More concurrent sequences per accelerator | Higher tail latency under load | Steady, throughput-heavy workloads |
With high concurrency and long context, KV-cache memory can limit throughput more than model weights. Cache compression may therefore outperform weight quantization for those workloads. A workload-aware benchmark of KV-cache compression methods found that compression ratio alone did not predict end-to-end results. SnapKV delivered strong long-context throughput, KIVI4 held quality more steadily, and CaM was highly sensitive to workload shape.
Benchmark short chat, retrieval-augmented generation and long-document flows separately. Track quality alongside p95 time-to-first-token and sustained throughput. A faster configuration that generates more incorrect answers, triggers escalations or requires rework may cost more per successful outcome, even if it uses less energy per token.
Measure joules per generated token at different power caps and quantization levels, using representative prompt lengths and batch sizes. Then choose the lowest-cost configuration that meets both your latency target and quality floor. Energy per token is a useful diagnostic, not the final business measure: compare infrastructure cost with the number of tasks completed successfully.
Hardware ownership is another cost decision. Steady, high-volume traffic may justify owned GPUs, while spiky or exploratory workloads may be better served elsewhere. Use this guide to compare local LLMs versus cloud AI before committing to private infrastructure.

A model swap that saves money and lets a wrong answer reach a customer buys you rework, churn and possibly a compliance conversation. The saving shows up this quarter. The cost shows up next quarter, usually with interest.
Cheap is not the same as safe, and it's worth being explicit about the order of operations. NIST's Generative AI Risk Management Profile recommends a test plan and a response policy before developing highly capable models, with periodic evaluation for misuse risks, and it notes that generative AI use may require additional human review, tracking, documentation and management oversight. Read that as a delivery constraint, not paperwork.
An evaluation set per workload. Every model, prompt, retrieval or routing change passes through it before it reaches production. No exceptions for "just a config tweak", because config tweaks are how quality drifts.
Security boundaries and access control for models and data. Know which model can see which data, which tenant can reach which context, and what leaves your environment when you call an external API. If you're changing retrieval design as part of a cost cut, our guide to RAG versus fine-tuning covers where the trade-offs land.
Human approval for high-impact actions, with explicit exception handling written down. What happens when the reviewer is unavailable at 2 a.m. should be a documented decision, not an improvised one.
Auditability of decisions, prompts and actions. If you can't reconstruct what the system was told and what it did, you can't debug a cost spike or defend a customer outcome.
Evaluation, monitoring and model operations after go-live. Cost, quality, refusal rates and escalations all get watched in production, with a rollback policy agreed before release rather than negotiated during an incident.
Here's the video walkthrough of how we think about production-grade AI delivery.
Watch for the three warning signs that a cost change went too far: escalations rising in the weeks after the change, quality drifting for a subset of users while the average looks fine, and a cost drop nobody can explain from the configuration. Accountability sits with a named senior owner who answers for cost, quality and security together, not three people who each own a slice.

That monthly review belongs in an operating rhythm, not a one-off project. The Improve area of Managed Technology Services exists for exactly this: planned releases, integrations and automation, plus performance and cloud-cost optimization as ongoing work rather than an annual scramble.
Bring your spend export and your top two features to a free 30-minute scoping call, and we'll tell you what should be optimized first, what should be automated, and what should be left alone.
The issue is rarely the model itself. They have an unattributed cost problem, a missing quality floor, and nobody accountable for the number that matters. Rite NRG builds production AI systems with senior architects who own architecture, data, security, testing and cost together, from agentic workflows and private deployments to the monitoring that keeps spend honest long after launch. Visit Rite NRG and tell us which feature is growing faster than its revenue, and we'll tell you which lever to pull first.
/ about the author
Written by the RITE NRG editorial team — the architects, engineers and delivery leads who build and operate AI-era software for our clients.
Strategy & Transformation
A practical, finance-ready framework for comparing AI and modernization investment with the current-state cost, realistic benefits, delivery risk and time to value.
Technology Talent & Teams
A decision-maker's guide to IT partnership models: what each model actually transfers to the partner, what it costs beyond the rate card, and how to match the model to your situation.
AI Consulting & Automation
A practical framework for choosing and deploying transport or logistics AI agents around real exceptions, reliable data and controlled operational authority.
More guidance: all insights articles
Tell us about the system or idea behind this article. We tell you honestly if it is a fit — and what a fixed-scope delivery could look like.