The AI tech stack: the tools behind every AI product, explained

The stack, from apps down to chips

Every AI product sits on these eight layers. Open any layer to see what it does and the names you'll hear.

1. 8 Apps and assistantsWhat people actually use Mostly closed

   The chatbots, copilots and AI features people use every day, plus tools for building apps without much code.

   Mostly commercial products, built on the layers below.

   Chatbots

   • ChatGPT
   • Claude
   • Gemini
   • Microsoft Copilot

   For builders

   • Cursor
   • Claude Code
   • GitHub Copilot
   • Lovable

   Words to know:ChatbotVibe coding

   The best AI chatbots →

2. 7 Testing and monitoringIs it actually working? Open and closed

   Tools that test AI answers before launch ("evals") and record what happens once real people use it, so teams can spot wrong answers, rising costs and slow replies.

   Open-source tools such as Langfuse sit alongside paid services.

   Evals and monitoring

   • LangSmith
   • Langfuse · open source; part of ClickHouse since 2026
   • Braintrust
   • Weights & Biases

   Words to know:Evals (AI evaluations)BenchmarkGuardrails

3. 6 Agents and orchestrationJoining the pieces together Mostly open source

   Frameworks that connect a model to data, tools and other models: planning steps, calling tools, remembering context and handing work between agents. MCP is the shared standard for plugging tools in.

   Mostly open-source frameworks, often with a paid cloud service on top.

   Frameworks

   • LangChain and LangGraph
   • LlamaIndex
   • CrewAI

   From the model makers

   • OpenAI Agents SDK
   • Claude Agent SDK
   • Google Agent Development Kit

   No-code

   • n8n

   Standard

   • MCP (Model Context Protocol) · run by the Linux Foundation's Agentic AI Foundation

   Words to know:AI framework (PyTorch, TensorFlow, LangChain)AI agentMCP (Model Context Protocol)

   AI agents, explained simply →

4. 5 Memory and dataWhat the AI can look up Open and closed

   Where a company's documents live in a form AI can search by meaning: turned into embeddings and stored in a vector database, ready for RAG.

   Open-source databases and managed cloud services both common.

   Vector databases

   • Pinecone
   • Weaviate
   • Milvus
   • Chroma
   • pgvector for PostgreSQL

   Words to know:EmbeddingVector databaseRAG (retrieval-augmented generation)

   RAG explained →

5. 4 Model accessHow apps reach a model Open and closed

   The ways an app gets answers from a model: paying per use through a model maker's API, through a cloud platform that offers many models, or by running an open model on your own servers.

   Paid APIs and cloud platforms, plus open-source tools for running models yourself.

   Model makers' APIs

   • OpenAI API
   • Claude API
   • Gemini API

   Cloud platforms

   • Amazon Bedrock
   • Microsoft Foundry · formerly Azure AI Foundry
   • Gemini Enterprise Agent Platform · formerly Vertex AI

   Run it yourself

   • vLLM
   • Ollama
   • llama.cpp

   One door to many models

   • OpenRouter

   Words to know:APIInference

   How to run AI on your own computer →

6. 3 ModelsThe "brains" Open and closed

   The trained AI models themselves. Closed models are only available through their maker's app or API; open-weight models can be downloaded, run and adapted by anyone.

   The biggest names are closed, but open-weight models are close behind.

   Closed

   • GPT (OpenAI)
   • Claude (Anthropic)
   • Gemini (Google)
   • Grok (xAI)

   Open-weight

   • Llama (Meta)
   • Gemma (Google)
   • Qwen (Alibaba)
   • DeepSeek
   • Mistral
   • gpt-oss (OpenAI)

   Where open models live

   • Hugging Face

   Words to know:Large language model (LLM)Foundation modelOpen-weight modelFine-tuning

   How AI chatbots work →

7. 2 Frameworks and chip softwareTools for building models Mostly open source

   The software researchers use to build and train models, and the low-level software that lets that work run fast on AI chips.

   The frameworks are open source; NVIDIA's CUDA is free to use but proprietary.

   Training frameworks

   • PyTorch · run by the PyTorch Foundation
   • TensorFlow and Keras
   • JAX

   Chip software

   • CUDA
   • ROCm

   Words to know:AI framework (PyTorch, TensorFlow, LangChain)CUDANeural network

8. 1 Chips and cloudWhere the maths happens Mostly closed

   The specialised chips that train and run AI, and the data centres that house them, mostly rented from big cloud companies.

   Proprietary hardware from a handful of companies.

   AI chips

   • NVIDIA GPUs
   • Google TPUs
   • AMD Instinct
   • AWS Trainium

   Clouds

   • Amazon Web Services
   • Microsoft Azure
   • Google Cloud
   • Specialists such as CoreWeave

   Words to know:GPUHBM (high-bandwidth memory)Data centre

   What powers AI →

One app, layer by layer

An online shop's support chatbot, followed from the customer down to the chips. Every real AI product works this way.

1. 8. Apps and assistants

   A customer types "Where is my order?" into the chat window on an online shop's website.

2. 6. Agents and orchestration

   A LangGraph workflow decides the steps: understand the question, check the order system through an MCP connection, search the help pages, write a reply.

3. 5. Memory and data

   The shop's help pages were turned into embeddings and stored in pgvector, so the right passage on delivery times comes back in milliseconds.

4. 4. Model access

   The reply is written by a model the shop pays for per use, through a cloud platform such as Amazon Bedrock.

5. 3. Models

   Behind that platform is a model such as Claude, Llama or Mistral, which the shop can swap without rebuilding the app.

6. 7. Testing and monitoring

   Before launch, the team ran evals on 200 real past questions; now Langfuse logs every chat so they can spot bad answers.

7. 1. Chips and cloud

   Under all of it, the model runs on GPUs in a cloud provider's data centre, rented by the second.

An illustrative example: real teams pick different tools at each layer, and can change one without rebuilding the rest.

Build, buy or somewhere in between?

Five ways an organisation can use AI, from the easiest to the hardest. Most start at the top and move down only when they need to.

1. Use an AI app

   ChatGPT, Claude, Gemini or Copilot, as they come.

   Good for

   Writing, research and everyday tasks.

   Skills needed

   None

   Your data

   Goes to the provider; business plans add protections.

2. Customise without code

   Custom GPTs, Claude Projects, Gemini Gems, or no-code automations in tools such as n8n.

   Good for

   Repeated tasks, team assistants, simple workflows.

   Skills needed

   A curious power user

   Your data

   Goes to the provider.

3. Build on a model API

   Your own app calling a model API, with RAG, agents and a framework such as LangChain.

   Good for

   Customer-facing products and internal tools.

   Skills needed

   Software developers

   Your data

   Sent to the API provider under a business contract.

4. Run or fine-tune an open model

   Host an open-weight model such as Llama, Qwen or Mistral on your own servers, and adapt it to your data.

   Good for

   Strict privacy, high volumes, special tasks.

   Skills needed

   ML engineers and GPU budget

   Your data

   Stays with you.

5. Train your own model

   Build a foundation model from scratch.

   Good for

   Only the largest companies and labs.

   Skills needed

   Research teams and huge budgets

   Your data

   Stays with you.

Jargon decoder: what they mean in meetings

• "We're on Bedrock" / "We use Foundry"

  They reach their AI models through Amazon's or Microsoft's cloud platform.

  Layer 4: Model access

• "We're building on Llama"

  They use Meta's open-weight model, probably running it themselves.

  Layer 3: Models

• "It's a LangGraph agent"

  The AI's steps and tool use are organised with the LangGraph framework.

  Layer 6: Agents and orchestration

• "It's MCP-compatible"

  AI assistants can plug into it through the standard connector.

  Layer 6: Agents and orchestration

• "We need better evals"

  They want more systematic tests of whether the AI's answers are good.

  Layer 7: Testing and monitoring

• "We store it in a vector DB"

  Documents are saved as embeddings so the AI can search them by meaning.

  Layer 5: Memory and data

• "There's a GPU shortage"

  There aren't enough AI chips to rent or buy for the work they want to do.

  Layer 1: Chips and cloud

• "It only runs on CUDA"

  The software only works on NVIDIA chips.

  Layer 2: Frameworks and chip software

Want to try a piece of the stack yourself? Run an open model on your laptop, or take a free course from our list of the best free AI courses. This page is part of level 3 of the Learn AI path.