Opus, Sonnet or Haiku: a cost & latency model

“Just use the biggest model” is a fine strategy until you have a P&L. When agents run your operation thousands of times a day, model selection stops being a quality question and becomes an economics question. Here’s the framework we actually use.

Three axes, not one

Every agent step gets scored on three axes before it’s assigned a model:

• Complexity — does this need genuine reasoning, or is it classification / extraction / routing?
• Latency budget — is a human waiting on this turn, or is it a background workflow step?
• Cost at volume — multiply the per-call cost by the real monthly call count, not the demo count.

reasoning-heavy + human waiting      → Opus
reasoning-heavy + background         → Sonnet
routing / extraction / high-volume   → Haiku

The cheap model is a force multiplier

The biggest unit-economics win wasn’t tuning Opus prompts — it was moving the long tail of classification and extraction work to Haiku, and using a larger model only as an escalation target when Haiku’s confidence is low. Most steps in a real agent system are not the hard ones.

Pick the smallest model that clears the bar, and design an explicit escalation path for when it doesn’t. Reliability comes from the fallback, not the ceiling.

Don’t forget the failure budget

Model choice is incomplete without rate-limit handling, retries with backoff, and a latency budget per turn. A correct answer that arrives after the workflow timed out is a wrong answer. We model this the same way we model cost: as a number per step, monitored in Datadog, with alerts when a domain drifts over budget.