The model choice was the easy part

Every board conversation about AI in 2026 starts the same way: which model do we use? It is the wrong question. As we’ve covered elsewhere, the frontier models — Claude, ChatGPT, Gemini — are now close enough in raw capability that the outcome you actually ship is decided by integration, not by brand. The value is in connecting an LLM to your data, your tools and your compliance rules, with a system that keeps working when the model of the month changes.

This piece is the short version of what we tell clients when they ask, “how do we bring AI into our business without breaking it?” Three ways to bring the model to your data, a plain decision table, the privacy rules that decide the architecture, and the cost reality once you move past a proof of concept.

Three ways to connect an LLM to your data

1. Prompting (with tools). The lightest path: you send the LLM a well-structured prompt, optionally with function or tool calls, and it produces an answer. Cheap, fast to build, no data storage. Excellent for one-off analysis, drafting, and workflows where the model doesn’t need to know anything private. Poor at anything requiring live company knowledge — the model only knows what you paste into the prompt each time.

2. Retrieval-Augmented Generation (RAG). The 2026 default for enterprises. Your documents are chunked, embedded and stored in a vector index. At query time, the system retrieves the most relevant passages and passes them to the LLM as context. The model answers from your data, with citations, and no retraining is required. Sensitive content stays in your controlled store; you can update, remove or restrict it without touching the model. Well-built RAG systems layer hybrid search (vector plus keyword), a reranker, permission inheritance from source systems, and audit logs for every response.

3. Fine-tuning. You adapt the model’s weights on curated examples so it internalises your style, terminology or a narrow task. Right when behaviour needs to be consistent and repetitive at very high volume — a classifier, a fixed customer-support tone, a domain-specific extraction task. Wrong when the underlying knowledge changes weekly, or when the training data itself is sensitive. Fine-tuning bakes information into the model, which is exactly what you don’t want for regulated data.

The honest 2026 answer for most enterprises is hybrid: RAG carries the live knowledge; a light fine-tune (or a good system prompt) locks the behaviour. Databricks, Azure OpenAI, AWS Bedrock and Vertex AI all support this pattern.

Data privacy is now an architecture question

Compliance stopped being a checkbox in mid-2026. Full enforcement of the EU AI Act begins in August 2026, adding transparency, risk-classification and documentation duties on top of existing regimes — GDPR in the EU, KVKK in Türkiye, HIPAA and SOX in the US, PCI DSS wherever you take a card. That directly shapes the architecture. Three questions have to be answered before you write any code.

Where does the data live? Residency matters. If your contracts require EU-only processing, use an EU region on Azure OpenAI, Bedrock or Vertex AI, or self-host. If clients demand full isolation, air-gapped deployments running open-weight models (Llama, Mistral, Qwen) via Ollama, vLLM or Onyx are a real option in 2026 — UC San Diego runs one such stack for over 37,000 users.

Is it used for training? The three consumer flagships have different defaults; enterprise API tiers generally do not train on your data, but read every contract. Any integration that touches client information must have this in writing.

Can you prove what happened? Audit logs, source citations and permission inheritance are no longer nice-to-haves. If a regulator asks which documents informed a specific answer six months ago, your RAG pipeline must be able to point at them.

Self-host or API? A practical rule

The right answer is almost never all-or-nothing. Use hosted APIs (Claude, GPT, Gemini) for high-quality reasoning tasks where speed to market matters and the data is either public or contractually protected. Use self-hosted open-weight models for anything that must never leave your infrastructure — regulated data, national-security work, deeply confidential client material. Route each request to the right destination through a thin gateway you own, so you can move traffic between providers as prices, capability and law change — because all three change quickly.

The cost reality at scale

A prompt-only prototype costs almost nothing. RAG is more: continuous vector-database hosting, embedding costs, and larger prompts (retrieved passages plus question) which grow the per-call token bill. Fine-tuning has a real upfront cost — GPU time, training data curation, an MLOps loop — and every model refresh forces you to redo it. Once query volume rises, the interesting number is cost per correct answer, not cost per token. A cheap model that needs three retries or a human review is not cheap. In our own builds, a well-tuned RAG on a mid-tier model routinely beats a raw premium-model call on both cost and accuracy, once the retrieval layer is doing its job.

What we tell our clients

We build AI integrations the way we build any other enterprise system: API-first, modular, and boring where it needs to be boring. A typical DZDSoft delivery has four layers — a gateway that decides which model handles which request, a RAG layer against your indexed content with SSO and permission inheritance, an optional light fine-tune where behaviour has to be consistent, and observability and audit end-to-end. Everything speaks JSON over HTTPS, so a model or vector store can be swapped without a rewrite. That is how you get value from AI now, and still have a system you own next year — when the vendor names and prices have all moved on.