Private AI in 2026: Self-Hosted LLMs vs Cloud APIs — Cost, Privacy, and When Each Makes Sense

Two years ago, "private AI" meant a research project. In 2026 it means a decision most mid-sized companies face on their first serious AI build: send prompts to a frontier model through an API, or run an open-weight model — Llama, Mistral, Qwen, Gemma, DeepSeek — on infrastructure you control. Vendors on both sides have an incentive to make it sound simple. It is not, but it is tractable. This guide gives you the actual cost math at three usage levels, separates real privacy constraints from imagined ones, and describes the hybrid architecture that roughly 70% of our AI development clients land on.

First, what "sending data to OpenAI" actually means in 2026

The consumer ChatGPT you used for free in 2023 trained on conversations by default. The business APIs from OpenAI, Anthropic, and Google do not: API inputs are not used for training, are retained for a limited abuse-monitoring window (typically 30 days, with zero-retention options available), and are covered by data processing agreements. Through Azure OpenAI, AWS Bedrock, and Google Vertex AI you additionally get region pinning, a BAA for healthcare, and the same compliance certifications your cloud already has. For most companies, "we can't use cloud AI" turns out to mean "we can't use the consumer product with a personal login" — which is correct, and is solved by using the enterprise path, not by buying GPUs. The rest of this article is for the cases where that is not enough.

When self-hosting is genuinely required

• Data cannot leave your network by policy or law: classified or defense work, some government contracts, certain banking regulators, and companies whose customer contracts explicitly forbid third-party processing.
• Data residency the cloud vendors do not cover: a specific country or an air-gapped facility.
• Volume where the API bill exceeds a server: above roughly 500 million to 1 billion tokens a month on a mid-sized model, dedicated hardware becomes cheaper — the math is below.
• Latency or availability you must control: factory floors, medical devices, offline field use, or a product whose core feature cannot depend on a third party's status page.
• Model ownership as a product moat: you fine-tune on proprietary data and do not want the weights, or the behavior, anywhere else.

If none of these apply, self-hosting is a preference, not a requirement — a legitimate one, but you should know you are paying for it.

The 2026 cost math at three levels

Small: an internal assistant, ~20 million tokens a month

Cloud API on a mid-tier model (roughly $0.15–$0.60 per million tokens): $5–$15 a month. On a frontier model: $50–$200. Self-hosted: the cheapest viable option is a single GPU cloud instance (one L4 or A10-class card) running a 7B–14B open model at $300–$600 a month, or a used workstation with a consumer GPU on-premise for $2k–$4k up front plus power and someone to maintain it. At this level self-hosting costs 20–50× more than the API. Do it for privacy, never for savings.

Medium: customer-facing product, ~300 million tokens a month

Cloud API mid-tier: $50–$200 a month; frontier: $750–$3,000; with routing between them, typically $300–$800. Self-hosted: one or two A100/H100-class GPUs with a serving stack (vLLM or similar) at $2,000–$4,500 a month on cloud GPU instances, less with reserved pricing. Add $1k–$2k a month of engineering time for monitoring, updates, and model upgrades. Still more expensive than the API for most workloads, but within the range where privacy or latency justify it.

Large: platform scale, 1 billion+ tokens a month

Cloud API mid-tier: $150–$600 a month — yes, still cheap on small models; frontier models at this volume: $2,500–$10,000+. Self-hosted: a 4–8 GPU node at $6,000–$15,000 a month serves this with a 70B-class model and headroom, and the per-token cost falls below frontier API pricing. This is where "we should run our own" becomes a finance decision as well as a privacy one — and where a dedicated ML/platform engineer is non-negotiable.

Open models closed most of the quality gap in 2025. They did not close the operations gap. The API bill you avoid becomes an engineer you hire.

— Rocket Systems Team

Quality: how far behind are open models?

For the tasks most businesses actually run — classification, extraction, summarization, retrieval-augmented Q&A over company documents, drafting in a defined format — current 8B–70B open models perform within a few points of frontier APIs when the prompt and retrieval are well engineered, and a fine-tuned small model often beats a generic large one on a narrow task. The gap remains real for long multi-step reasoning, complex agentic tool use, rare languages, and anything where you cannot afford the model to be occasionally wrong without a check. Practically: open models are excellent inside a pipeline with evaluation around them, and riskier as an open-ended assistant. Our RAG vs fine-tuning guide covers how to get that quality out of them.

The hidden costs of running your own

• People: someone must own serving, scaling, GPU drivers, model upgrades, and security patching. Budget 0.25–1 FTE depending on scale; via AI engineer staff augmentation from $15/hour this is the most common way our clients cover it.
• Evaluation: every model swap needs a regression run against real cases. Cloud users get this pressure too, but self-hosters swap more often chasing improvements.
• Security of the box itself: a GPU server with your entire document corpus in a vector database is now the most sensitive machine you run. Network isolation, secrets, and access logs apply.
• Capacity planning: APIs absorb spikes; your two GPUs do not. Either over-provision or accept queueing at peak.
• Falling behind: the API gets better every quarter with no work on your side. Your self-hosted model gets better only when someone upgrades it.

The hybrid most companies end up with

Route by sensitivity and difficulty. A small open model — self-hosted or on a private cloud endpoint — handles the high-volume, sensitive, or simple traffic: PII redaction, classification, extraction, first-pass drafting over internal documents. Anything that survives redaction and needs frontier reasoning goes to an enterprise API endpoint (Azure OpenAI, Bedrock, Vertex) in your region under a DPA or BAA. Sensitive data never reaches the frontier model in raw form, the frontier model does the hard 20%, and the bill is a fraction of frontier-for-everything. The architecture is the same one we use for regulated builds under HIPAA-compliant AI, and it is model-agnostic by construction — when a better open or closed model appears, you swap a config value and rerun the evaluation.

What a private AI build costs to set up

• Enterprise cloud endpoint setup (region pinning, DPA/BAA, zero retention, logging, gateway with SSO): $2.5k–$5k — often part of a first AI automation project.
• Self-hosted single-model stack (vLLM or equivalent, one GPU node, monitoring, an internal API, an eval harness): $6k–$12k plus infrastructure.
• Hybrid routing architecture with redaction, model router, two endpoints, and evaluation: $10k–$25k, 6–10 weeks, depending on how many workflows it serves.
• Fine-tuning a small model on your task: $4k–$15k including dataset curation, which is most of the work.

A decision rule you can apply in one meeting

• If your constraint is "no consumer AI tools with company data" → enterprise API endpoints via your cloud provider. No GPUs.
• If your constraint is regulatory residency or a BAA → enterprise endpoints in-region first; self-host only for what those cannot cover.
• If your constraint is "data cannot leave our network, full stop" → self-host, budget 0.5 FTE, keep the model small and the task narrow.
• If your motivation is cost → you are probably below the break-even; check the token math above before buying hardware.
• If you are unsure → build the hybrid. It costs little more than either extreme and keeps every option open.