A cost and engineering decision framework for choosing between hosted LLM APIs (Anthropic, OpenAI, Gemini) and self-hosted inference (Ollama, vLLM, on-premise GPU clusters).
Monthly inference spend is below the five-figure threshold
Workload is dominated by frontier-tier reasoning that open-weights models cannot match
You need access to the latest models within days of release
Your team does not have GPU infrastructure operational experience
You are early stage and operational simplicity beats cost optimisation
Monthly inference spend has crossed five figures and is dominated by repetitive small-model calls
Data residency or sovereignty requirements rule out hosted vendors
You need predictable latency that does not degrade during provider load spikes
You are running a workload type (classification, extraction, embedding generation, summarisation) where mid-tier open-weights models match frontier-tier API quality at a fraction of the cost
Your team has the operational muscle to own GPU infrastructure — or is willing to build it
Step 1 — Profile your workload. What share of calls are frontier-tier reasoning vs mid-tier extraction / classification / summarisation? The answer almost always pushes toward a hybrid: hosted for frontier, self-hosted for the long tail.
Step 2 — Model true cost-per-request including amortised infrastructure for the self-hosted side. Include GPU lease, ops engineering time, evaluation harness time, and the cost of staying on the open-weights model release cadence.
Step 3 — Run a 50-prompt eval on the open-weights model you are considering against the hosted equivalent for your specific workload. Most teams discover the quality gap is smaller than they assumed for non-reasoning tasks.
Step 4 — Project cost and operational load forward to 100% of expected scale. The break-even is rarely where intuition puts it.
Step 5 — Default to hybrid above the break-even threshold. Single-strategy lock-in (all hosted or all self-hosted) almost always overspends or under-scales.
The cloud-versus-on-premise decision is rarely a clean either-or above scale. The teams Skopa audits at the higher end of inference spend almost always end up running a hybrid: frontier-tier hosted APIs for the genuinely hard reasoning, self-hosted Ollama or vLLM clusters for the long tail of repetitive calls. The discipline that decides whether self-hosted economics actually pay off is GPU utilisation — clusters provisioned for peak load and running at 15% average utilisation almost always lose to hosted APIs. Right-sized clusters with autoscaling and request queuing almost always beat them above the break-even threshold.