Let’s talk some more about how building AI products in 2025 means making hard choices about your infrastructure stack.
Seen too many startups either drown in infra complexity or burn cash on hyperscaler invoices, so since last year, I’ve started mapping the different “AI stack” adoption patterns I see in the wild.
Here’s five of them. TL;DR:
- “Just Works”
- Self-Hosted, Privacy-First
- Hybrid Pragmatist
- Fully reproducible Academic/Scientific
- “ChatGPT-for-X” Virality
Unless you’re Google-scale, your architecture is less about frontier-model wizardry and more about trade-offs — cost, latency, compliance, lock-in. I’ve added two more constraints : privacy, and operational complexity, because I’m putting this together with teams of my scale in mind.
1. The “Just Works” Stack
The Pattern: Managed services, minimal ops, pay-as-you-scale
The Sweet Spot: Founders who need to validate an AI product quickly — prototypes, PMF testing, SaaS MVPs.
This is where most teams start, and for good reason. Wire up OpenAI’s API to Pinecone, throw in some Firecrawl for data ingestion, deploy to Vercel — you’ve got a functional RAG pipeline running in an afternoon, that won’t go down during that critical demo day traffic spike.
The architecture:
- LLM: API-first (OpenAI, Claude). You’re buying top-tier reasoning quality, multimodal support, and tool use with zero ops overhead.
- Vector store: Fully managed like Pinecone or Weaviate. Simple REST/gRPC, SLA-backed uptime, metadata filtering out of the box.
- Data ingestion: Firecrawl, Apify actors, or similar. Wire them up via LangChain/LlamaIndex connectors for ETL into your vector DB.
- Hosting: Serverless functions (Vercel, Netlify). DX optimized, scales well for that first demo-day traffic spike.
This one is dead simple. You can wire up a RAG pipeline in hours: API calls → embeddings → vector inserts → serverless endpoint. Predictable, boring APIs that ‘just work’, generous free tiers and SDK coverage make this stack ideal for dipping your toe in the AI pond, demos, hackathons, or fundraising decks.
Most enterprises default here — even big corps with dedicated ML teams end up on Azure OpenAI or AWS Bedrock to avoid infrastructure headaches.
The catch, however, is a killer. The cost scaling will kill anyone but the biggest players.
At small scale (or just for weekend trials), usage-based pricing feels like magic — even $20 in API credits gets you a really nice, working demo. But as traffic grows, every token your users send is a cost you pay. Long-form prompts, multi-turn chats, and context-heavy RAG queries all balloon token usage. Unlike compute you own (where marginal cost drops as you scale, FYI), API pricing remains constant. If you’re not modelling token economics early, you will get blindsided when your infra bill outpaces MRR.
Also, there’s the vendor lock-in.
- Embeddings are model-specific. If you start with OpenAI’s
text-embedding-3-smalland later want to switch to something open source (say, Mistral embeddings), you can’t just port vectors. You need to re-embed your entire corpus. - If your app logic assumes OpenAI-style function calling or Claude-style JSON mode, you’re going to have to refactor for another LLM API, which is non-trivial.
Tips: So scaling is going to be difficult here. Benchmark your average tokens per query before you launch. A single RAG pipeline can easily hit 2k–4k tokens per user query (prompt + context + completion). Most founders underestimate how verbose real-world user prompts are compared to test queries.
Finally, Firecrawl/Apify might work for demos and light ETL, but they’re “best effort.” Anti-bot systems, CAPTCHAs, geo-blocking, and rate-limits will break your pipelines in prod. At scale you’ll need robust data acquisition (there is a better solution to this part — we’ll get to that in a second.)
2. The Fully Self-Hosted, Privacy-First Stack
The Pattern: On-premises or private cloud. Full data control, nothing leaves the perimeter.
The Sweet Spot: Regulated industries — healthcare, finance, government — or any team where compliance and data sovereignty trump convenience.
This is where you go when you can’t rely on managed APIs. Forget all the “just works” endpoints, now you’re prototyping locally with Ollama and scaling out with vLLM on Kubernetes when it’s time for production. Every embedding sits in a Qdrant or Weaviate DB that you run. Every doc is ingested through internal processors or crawlers, no external calls.
Something to clarify: this isn’t about open source. You can run proprietary/enterprise software, but it has to run on your infrastructure. Everything lives in your cluster, your VPC, or your metal.
The architecture:
- LLM: Self-hosted via vLLM (Ollama for dev), scaled with Kubernetes or similar for production. OSS models (LLaMA, Mistral, DeepSeek, Qwen etc.) are common, but proprietary inference stacks are 100% fine as long as they run in your VPC.
- Vector store: Qdrant or Weaviate, self-hosted. Both give you high-performance ANN search with metadata filters, but scaling/sharding is now your job. Enterprise/hybrid editions are acceptable if deployed inside your perimeter.
- Data ingestion: Airgapped document processors and internal crawlers. Zero external API calls.
- Hosting: Docker Compose for dev, Kubernetes or bare metal for production. Many teams also use NVIDIA Triton for model serving at scale. Monitoring, GPU orchestration, and upgrades are all on you.
This stack exists for one reason: full sovereignty. Every query, every log, every corpus is yours and stays inside your perimeter. You get an audit trail your compliance team will love, and you decide when to roll model updates, how to tune your indexes, and what GPUs you deploy on. No one dictates what you can or can’t do.
The catch is that you’ve now signed yourself up to be both an infra company and an AI company.
Model updates? That’s on you. Qwen or Gemma ship a new release? You’re rebuilding fine-tunes and redeploying inference stacks. Scaling inference across GPUs means scheduling headaches, and A100 spot prices can swing 10x in a single week. MLOps? You’d better have a real team on it yesterday.
Unlike the “Just Works” stack where you bleed money linearly with tokens, here you bleed ops hours and capex. OSS dogma doesn’t save you here because your pain is operational, not philosophical.
Tips: I don’t have any. If you’re even considering this, you know exactly how good you are — and this article isn’t for you 😅
This is the “compliance or bust” route. It’s the one your security team will push for, but it’s also the one that eats entire engineering teams just to keep the lights on. If you need this stack, you’ll know it already. If you don’t, you’ll regret picking it.
3. The Hybrid
The Pattern: The balanced, pragmatic stack. Mix of hosted and local components. Outsource what’s boring and own what’s critical.
The Sweet Spot: Product teams that want control over specific parts (privacy, cost, latency) while still leaning on managed services to avoid reinventing the wheel. This stack represents where the industry is actually heading in 2025 and beyond. Not the extremes of “all-API” or “all-local,” but intelligent hybrid architectures that optimize for both cost and capability.
This sits smack dab in the middle of the two extremes above. You’ll run local models when it saves you money or keeps data in-house, and lean on frontier APIs when quality matters more than cost. You’ll keep your vectors close, but push cold workloads to managed stores so you don’t have to babysit indexes at scale. You’ll still scrape, crawl, and ingest — but now with hardened pipelines that actually work under load.
The architecture:
- LLM: Local inference (Ollama for prototyping, vLLM for production) handles baseline workloads with capability-based fallback to APIs (OpenAI/Anthropic) for complex reasoning, advanced code generation, or nuanced analysis you know your model can’t handle well (Gemma 3 is suboptimal for code, for example.)
- Vector store: Qdrant or Chroma in-house as the “source of truth.” Pinecone/Supabase sit downstream as managed extensions — think data tiering, not mirroring. Hot, sensitive data stays in-house; cold/archive indexes live on the cloud.
- Data ingestion: Bright Data’s Scraping Browser for bot-detection bypass, or roll-your-own CDP/Puppeteer pipelines in containers. MCP-style plugin servers are emerging here, too.
- Hosting: Traditional cloud (Azure/AWS/GCP), use routing layers like LiteLLM or Portkey for intelligent model switching and unified APIs.
This stack is about control without the masochism. You keep the parts where vendor lock-in or cost hurts the most, and you outsource the boring undifferentiated heavy lifting.
Press enter or click to view image in full size
Here’s an example of what I mean by that. Remember when I said there was a better solution for data acquisition at scale than Firecrawl/Apify? This is where you can flex it.
Take web data ingestion. One of the biggest operational headaches in AI systems. You could build your own crawler infrastructure, manage proxy rotation, handle bot detection, and maintain browser farms…or you could run something like Bright Data’s MCP server as a containerized service.
This is fully open-source, slots right into your toolchain, available as both a local deployment and a hosted service. Handles anti-bot circumvention and geo-restriction bypassing through a standardized protocol interface, so your local models get structured web data without you needing to maintain scraping infrastructure. You can tune the dials on privacy, performance, and spend, without having to burn the house down.
The downside is you’ve now got multiple deployment patterns running in parallel. Whatever you’re using for observability needs to catch both API-level failures (rate limits, quota exhaustion, latency spikes) and infra-level failures (VRAM saturation, GPU pool congestion, or node restarts). And your pipeline has to failover cleanly between the two — a rate-limited API call should feel no different than a vLLM node choking under load.
The key thing is baking in unified routing early — like ChatGPT and Grok do these days. LiteLLM or Portkey let you do this by normalizing API and local inference behind a single interface — but of course you can roll your own routing.
Pair that with tracing and metrics and you should be good to go.
This hybrid model is the natural convergence point because it optimizes for cost, reliability, and control at the same time.
- Shopify has publicly discussed running smaller models locally for product categorization and search ranking while using GPT for complex merchant support queries. Their scale (millions of products) makes the hybrid approach economically necessary.
- Notion uses local models for basic text processing and formatting, but falls back to OpenAI for complex reasoning in their AI writing assistant.
But most teams don’t start here. They land here — usually after the two above stacks’ ops burn them out.
4. The Academic Research Stack
Pattern: 100% open source, full source code access, zero proprietary dependencies Sweet spot: Research institutions, academia, reproducible AI research
This is a very specific one. The full-on privacy stack is all about where code runs (inside your perimeter), this one is about what the code is (you can see, modify, and redistribute every line).
This is all about extensibility and scientific reproducibility. If you need to swap out a vector similarity function, patch an embedding layer, or publish research that others can fully reproduce, this is your stack. If you can’t fork it, redistribute it, or audit it line by line, it doesn’t make the cut here. When academic/research requirements demand full source code access, this stack is your only option.
The architecture:
- LLM: Hugging Face Transformers with pinned versions and checksums. Training in PyTorch/JAX with full configs committed to git. Ollama for prototyping, but production runs use documented scripts that anyone else can re-run.
- Vector store: Chroma, Qdrant OSS, or FAISS. Have index parameters, distance metrics, and thresholds recorded in experiment configs.
- Data ingestion: ETL pipelines with lineage tracking. Scrapy spiders or BeautifulSoup parsers with fixed seeds and deterministic parsing rules. Every preprocessing step saved or logged so another lab can rehydrate the dataset.
- Experiment management: MLflow (or a community effort like MLOP) for logging hyperparameters, metrics, and artifacts. Every run gets an ID, config, and checksum so experiments can be replayed exactly.
- Hosting: You want deterministic builds, reproducible environments: requirements.txt with pinned versions, Dockerfiles with fixed base images, container registries archived for citation. Infra as code is optional — what matters is that another lab can spin up your exact runtime. Also, document hardware and driver versions — results can drift otherwise.
Why not Privacy-First? That one tolerates proprietary enterprise software as long as it runs inside your perimeter. This rejects anything closed-source. The dividing line is reproducibility: if someone else can’t independently verify your results, the research isn’t valid.
Press enter or click to view image in full size
When this stack makes sense:
- Publishing reproducible papers where every component must be replicable.
- Grant funding requires reproducibility. NSF and NIH both mandate open science practices.
- Building open infrastructure that other researchers will extend.
- Avoiding licensing risks that can (and do!) invalidate long-term research.
- Experimenting at the algorithmic level — custom similarity functions, novel embedding pipelines, or new model architectures.
The tradeoff: Velocity is slower, tooling is less polished, and you own every bug. But the payoff is integrity: code, data, and results that stand the test of time. And because it’s open, you’re not maintaining it alone — other researchers will debug, optimize, and extend your work, effectively crowdsourcing QA.
Make triple sure you use MMLU or domain-specific datasets and run them after every commit. Re-run with fixed seeds to detect nondeterminism. Remember, unlike SaaS APIs, nobody else is watching your regressions for you.
5. The “ChatGPT-for-X”
Pattern: Edge/serverless infra. Consumer-facing. Latency is king.
Sweet spot: Startups pitching “ChatGPT-for-X” — customer support, legal docs, health advice, edtech tutors. Anywhere a realtime UI and viral adoption matter more than perfect reasoning.
This is the sexiest one. The stack of demos-that-go-viral. It’s the quickest way to make something feel like a next-gen product, even if under the hood it’s more duct tape than deep reasoning. If you’re building a consumer assistant and want the “wow!” before the “how?” this is your stack.
The user types, and something comes back near-instantly. Whether the answer is frontier-grade or not often matters less than whether the value provided to the user feels instant.
The architecture:
- LLM: Distributed inference at the edge — Cloudflare Workers AI (plenty of open models), Replicate endpoints, or similar GPU fleets. You’ll usually get ~70–80% of GPT-5’s quality, but with that crucial <100ms latency.
- Vector store: Edge-friendly — Upstash Vector, Neon Postgres + pgvector, or Cloudflare D1. Most other stacks assume heavy-duty vector infra (Qdrant, Weaviate, Pinecone). This one is about lightweight edge-ready stores. Different tradeoff entirely: smaller, cheaper, faster to spin up.
- Data ingestion: SERP APIs, lightweight crawlers, and aggressive CDN-backed caching. Fresh context patches over weak reasoning.
- Hosting: Cloudflare Workers, Vercel Edge, or Deno Deploy. App logic rides CDNs; inference hops to distributed GPU clusters.
This stack exists because “fast” beats “smart” in consumer AI. A chatbot that answers instantly feels magical, even if it occasionally hallucinates. A chatbot that answers correctly but lags half a second feels broken. That’s why edge GPU fleets, CDN routing, and caching tricks matter here more than model IQ.
The trick is that you’re compensating for weaker reasoning by giving the model fresh context at runtime. Instead of hoping a 7B OSS model “knows” the latest airline refund policy or NBA score (Spoiler: it absolutely won’t), you just fetch it — scrape, cache, and pass it in as context. Solutions like Bright Data’s SERP APIs are designed for exactly these low-latency pulls — both scheduled and on-demand — and when you cache those results at the edge, you get a chatbot that feels smart without actually running a 400B frontier model.
It’s retrieval-augmented generation with a consumer edge focus: cheap, fast, good enough, and infinitely more believable.
The catch is that edge AI is still rough around the edges. Workers AI or Replicate clusters won’t give you the depth of reasoning you’d get from a frontier API like Sonnet, Grok, or GPT-5. “Sub-100ms” is a marketing line — in reality, you’ll see 50–200ms depending on the region and the model, and debugging distributed edge code is painful when logs and metrics are fragmented across PoPs.
Even your vector layer has limits: pgvector on Neon starts creaking once you cross 100K rows unless you throw a lot of tuning at it.
What saves you is ruthless pragmatism. Optimize, optimize, and optimize. Pre-compute embeddings at ingestion to dodge API roundtrips, cache aggressively because CDN bandwidth is cheap, and benchmark latency per region instead of trusting global averages.
TL;DR: Pick the stack you can actually run today.
Most teams don’t sit down and “design” their stack from first principles. They stumble into one, discover its limits, and then evolve (or retreat) as reality sets in. But not all stacks are equally stumble-able: some are natural starting points, while others are dictated by your industry or mission.
Stack 1 (Just Works / API-First): Where 90% of teams begin. Lowest friction, fastest PMF testing. If speed to market is your constraint, this is the obvious choice — and for many startups, it’s also the endgame.
Stack 2 (The “ChatGPT-for-X” / Edge-Optimized): The consumer path. If you’re chasing virality or building UIs where <100ms latency feels magical, this is usually the first pivot after Stack 1. Think of it as the “growth hack” stack: global routing, edge GPUs, aggressive caching.
Stack 3 (Fully Self-Hosted, Privacy-First): You don’t stumble into this one. It’s the default for healthcare, finance, defense, or government — anywhere compliance officers, not product managers, make the call. If data sovereignty is the blocker, this stack isn’t a choice, it’s a mandate.
Stack 4 (The Hybrid Pragmatist): The transitional play. It looks like a cost-saver but comes with ops pain: GPU scheduling, monitoring, failover logic. Teams often underestimate the burden and either collapse back to Stack 1 or, if they have the muscle, graduate into 3 or 5. If cost optimization is your primary driver and you’ve got ops maturity, this is the tradeoff stack.
Stack 5 (Research/Academia): Not evolutionary — 100% intentional. This is the academia and DevTools stack, where scientific reproducibility or vendor independence trumps shipping velocity. If your mission is publishable research or open infra others can extend, this is the stack.
The key is realizing you’re not choosing once. For commercial orgs, the typical arc is: 1 → 4 → 3 for enterprise, or 1 → 2 for consumer apps. For research orgs, it’s straight to 5. Infrastructure choices compound over time, so just pick the stack your team can actually run today.