Your AI Bill Is 10x What It Should Be (Here's How to Fix It)

Your AI Bill Is 10x What It Should Be

I review the AI architecture of small and mid-size businesses as part of my consulting work. The most common finding: they are spending 10-50x more than necessary on LLM API costs. Not because the technology is expensive, but because nobody optimized the implementation after the prototype worked.

A typical example: a chatbot that sends the entire conversation history (including the system prompt) with every message. The system prompt is 2,000 tokens. The conversation averages 10 messages. By message 10, each API call includes 2,000 (system) + 8,000 (history) = 10,000 input tokens, even though only the last 200 tokens are the new message. At Claude Sonnet pricing, that is $0.03 per message instead of $0.006 — 5x the necessary cost.

The Model Routing Pattern

The single biggest cost reduction technique: do not use the same model for everything. Most conversations do not need your most expensive model.

In the AI systems I build, every request goes through a model router that classifies the query complexity and routes to the appropriate model:

• Simple lookups (hours, location, basic FAQs): Claude Haiku ($0.25/MTok input). These are 60-70% of all queries.
• Standard conversations (product questions, support, general chat): Claude Sonnet ($3/MTok input). About 25-30% of queries.
• Complex reasoning (multi-step comparisons, nuanced objection handling, technical troubleshooting): Claude Opus ($15/MTok input). Under 5% of queries.

The router itself is a Haiku call that costs ~$0.0001 per classification. A system processing 1,000 queries per day saves roughly $60/day ($1,800/month) by routing 70% of queries to Haiku instead of sending everything to Sonnet.

Prompt Caching

If your system prompt is longer than 1,024 tokens and you are making more than a handful of calls per hour, prompt caching cuts your input costs by up to 90% on the cached portion. Anthropic’s prompt caching charges 10% of the normal input price for cached tokens.

A 3,000-token system prompt sent 100 times per hour:

• Without caching: 300,000 input tokens/hour × $3/MTok = $0.90/hour = $648/month
• With caching: 3,000 tokens at full price once + 297,000 at 10% = $0.098/hour = $71/month

That is an 89% cost reduction from a single configuration change. I have seen teams spend months optimizing their prompts to be shorter when they could have just enabled caching and saved more.

Context Window Management

The conversation history management pattern I use in production:

1. Keep the last 6 messages in full. This gives the LLM enough context to understand the current topic.
2. Summarize older messages. Messages 7-20 get compressed into a 200-token summary: "Customer asked about MacBook Air pricing, discussed M1 vs M2 differences, expressed interest in the $549 M2 model."
3. Drop messages beyond 20. If the conversation has gone 20 messages without resolution, the AI should be escalating, not accumulating context.

This pattern keeps the context window under 4,000 tokens regardless of conversation length, versus the naive approach that grows linearly and can hit 30,000+ tokens for a long support conversation.

Caching Responses for Common Questions

If 30% of your support questions are "what are your hours?", you are paying the LLM to generate the same answer hundreds of times per month. Response caching eliminates this waste.

I use a two-tier cache backed by Cloudflare KV:

• Exact match cache: Hash the user’s message. If an identical message was asked in the last 24 hours, return the cached response. Hit rate: 15-20% for support chatbots.
• Semantic match cache: Embed the message and check for high-similarity matches (cosine > 0.95) in a lightweight vector index. "What time do you close?" and "When do you close?" get the same cached response. Hit rate: additional 10-15%.

Combined, these caches eliminate 25-35% of LLM calls entirely. Each cache hit costs essentially nothing (a KV read is <$0.0000005) versus $0.003-0.03 for an LLM call.

Streaming vs. Batch

Streaming responses (server-sent events) cost the same as batch responses in API pricing, but they dramatically improve perceived performance. A response that starts appearing in 200ms feels instant, even if the full response takes 3 seconds to generate.

More importantly, streaming enables early termination. If the user sends a new message while the AI is still generating, you can cancel the current generation and start processing the new input. Without streaming, you pay for the full generation even if the user has already moved on.

In chatbot deployments, I measure 5-8% of generations being interrupted by user follow-ups. That is 5-8% cost savings from a UX improvement that also makes the product better.

The Self-Hosted Escape Hatch

For businesses processing more than 10,000 queries per day, the API cost curve starts to favor self-hosted models. I run a 27B parameter Qwen model on consumer GPUs that handles the "simple lookup" tier at effectively $0 per query after hardware costs.

The math: a used RTX 3060 12GB costs $200. It runs a quantized 27B model at 20 tokens/second, handling ~2,000 queries per day. At API pricing, those 2,000 daily queries would cost $1-5/day ($30-150/month). The GPU pays for itself in 1-6 months, then every query is free.

I do not recommend self-hosting for everyone. The breakeven point is roughly 5,000+ queries per day for simple queries, or 1,000+ per day for queries that would otherwise hit Opus. Below that volume, the API is cheaper when you factor in maintenance time.

Real Cost Audit Results

I recently audited the AI costs for a client running a customer support chatbot on Claude Sonnet:

• Before optimization: $340/month (12,000 queries, all Sonnet, full history, no caching)
• After model routing: $145/month (68% routed to Haiku)
• After prompt caching: $92/month (3,200-token system prompt cached)
• After context management: $71/month (6-message window + summary)
• After response caching: $52/month (28% cache hit rate)

Total reduction: $340 → $52/month (85% savings). Same quality of responses. Same customer satisfaction scores. The optimizations took two days to implement.

The Optimization Checklist

If you are running an LLM-powered system in production and have not done these, you are overspending:

1. Enable prompt caching if your provider supports it. 5-minute change, 50-90% savings on system prompt costs.
2. Implement model routing. Use Haiku/GPT-4o-mini for simple queries. One day of work, 40-60% total savings.
3. Cap your context window. Summarize old messages, drop ancient ones. Half a day of work, 20-40% savings on long conversations.
4. Add response caching for frequently asked questions. One day of work, 15-30% reduction in total API calls.
5. Monitor and iterate. Log every API call with its cost. Review weekly. The 80/20 rule applies: a handful of expensive query patterns account for most of your spend.

These five steps, applied in order, typically reduce AI infrastructure costs by 70-90%. The remaining cost is the irreducible minimum: genuinely novel queries that need the full power of a frontier model.

Related Articles