How to Run AI Models Locally in 2026 — Complete Setup Guide

Why run AI locally?

There are compelling reasons to run models on your own hardware rather than relying on cloud APIs:

• Privacy — Your prompts and data never leave your machine. Essential for sensitive documents, medical records, legal work, or proprietary code.
• Cost — After the initial hardware investment, inference is free. High-volume use cases can save thousands per month vs. API pricing.
• Offline access — Works on planes, in areas with poor connectivity, or in air-gapped environments.
• Latency — No network round-trip. For interactive applications, local inference often feels faster than API calls.
• Customisation — Full control over model selection, quantisation, system prompts, and parameters. No content filters unless you choose them.

The trade-off: local models are generally less capable than frontier cloud models (GPT-4, Claude Opus). For many tasks — coding, summarisation, Q&A, data extraction — the quality gap is small enough to not matter. For cutting-edge reasoning, you may still want API access.

Hardware requirements

What you need depends on which models you want to run. Here's a practical breakdown:

• Entry level (7B models, Q4) — 16GB system RAM, any modern CPU. Works but slow (~5-10 tokens/second). Suitable for occasional use.
• Good experience (7B models, fast) — Apple Silicon Mac with 16GB+ unified memory (30+ tokens/sec) OR NVIDIA GPU with 8GB VRAM (40+ tokens/sec).
• Best local setup (13-34B models) — 32GB RAM + NVIDIA GPU with 16-24GB VRAM (RTX 4090, A5000) OR Apple M2/M3 Pro/Max with 32-64GB.
• Enthusiast/production (70B models) — 64GB+ RAM with dual GPUs or Apple M2/M3 Ultra, or a single A100/H100.

Apple Silicon is uniquely good for local AI because its unified memory architecture lets the CPU and GPU share memory — a 96GB M2 Ultra can run a 70B model that would require a $10,000+ GPU on Windows/Linux. See our AI hardware guide for detailed comparisons.

Getting started with Ollama

Ollama is the simplest way to run LLMs locally. It handles model downloading, quantisation, and hardware acceleration automatically.

Installation:

• macOS — Download from ollama.com or run: brew install ollama
• Linux — curl -fsSL https://ollama.com/install.sh | sh
• Windows — Download the installer from ollama.com

Running your first model:

ollama run llama3       # Downloads and runs Llama 3 8B
ollama run mistral      # Mistral 7B
ollama run codellama    # Code-focused model
ollama run phi3         # Microsoft's efficient small model

That's it. Ollama downloads the model on first run (~4GB for a 7B Q4 model) and starts an interactive chat. It automatically detects and uses your GPU if available.

Ollama also exposes an API compatible with the OpenAI format, so you can use it with any tool that supports OpenAI's API by pointing it at http://localhost:11434.

Alternative tools: LM Studio and llama.cpp

If Ollama isn't your style, two other excellent options:

LM Studio — A desktop GUI application (macOS, Windows, Linux) that lets you browse, download, and chat with models through a visual interface. Best for users who prefer not to use the terminal. Features include:

• Built-in model browser with search and filters
• Side-by-side model comparison
• Local API server (OpenAI-compatible)
• Adjustable parameters (temperature, context length, etc.)

llama.cpp — The core inference engine that powers Ollama, LM Studio, and many other tools. Use it directly when you need:

• Maximum performance tuning (batch sizes, thread counts, GPU layers)
• Server mode for production deployments
• Integration into C/C++ applications
• Bleeding-edge features before they reach higher-level tools

For choosing between models, see our open-source LLMs guide which ranks models by real community adoption.

Building local AI workflows

Once you have a model running, here are practical ways to use it:

• Private document Q&A — Use a RAG setup (see our RAG guide) to chat with your own documents. Tools like PrivateGPT and AnythingLLM make this turnkey.
• Coding assistant — Run CodeLlama or DeepSeek Coder through Continue.dev (VS Code extension) for a Copilot alternative that works offline. For more coding tools, see our AI coding assistants guide.
• Email/writing assistant — Connect Ollama to your text editor for drafting, summarising, and editing.
• Data processing pipeline — Use the Ollama API to process CSV/JSON files locally — classification, extraction, summarisation at scale with zero API costs.
• Voice assistant — Combine Whisper (speech-to-text) + local LLM + a TTS model for a fully offline voice assistant.