How to Run a Powerful LLM on Just 2GB of VRAM (Step by Step)

When people hear "run a large language model locally," the first thing they picture is a $2,000 NVIDIA RTX 4090. Ask on Reddit or Twitter whether you can run an LLM on your laptop, and someone will invariably reply: "Not worth it unless you have at least 8 GB of VRAM."

That advice was reasonable in 2023. It's dead wrong in 2026.

I'm writing this on a laptop with integrated graphics and 2 GB of available VRAM — and I'm running a model that outperforms Llama 3.1 7B and Gemma 2 7B across most benchmarks. Not a stripped-down, barely-functional demo, either. Real inference. Usable speeds. A model I can actually get work done with.

The model that makes this possible is MiniCPM-O 6B, a 6-billion-parameter multimodal model from OpenBMB that runs on as little as 2 GB of VRAM while matching or beating much larger models. In this guide, I'll walk through everything: why this model is different from everything else out there, how to install it with Ollama in under five minutes, how to add a ChatGPT-style interface with Open WebUI, and what real-world performance actually looks like.

By the end, you'll have a fully private, fully local AI assistant running on hardware most people would dismiss as unusable for LLMs. No cloud. No API key. No monthly bill.

Why Most People Are Wrong About VRAM Requirements

The conventional wisdom about VRAM is simple and, at first glance, mathematically sound. A 7-billion-parameter model at FP16 precision occupies roughly 14 GB of memory — 7 billion parameters × 2 bytes per parameter. Add a few gigabytes for the key-value cache and overhead, and you're looking at 16-18 GB. That's firmly in RTX 4080/4090 territory. Even at 4-bit quantization, you're still at 4-5 GB plus overhead — borderline for a 6 GB card and impossible on 2 GB.

This math is correct for most models. It's also why the "you need 8 GB minimum" advice persists. But it assumes something critical: that the model architecture is a standard dense transformer where every parameter is activated on every token.

That's not how MiniCPM-O works.

Enter MiniCPM-O 6B: The Model That Broke the Rules

MiniCPM-O 6B is a member of the MiniCPM series from OpenBMB and Tsinghua University, built on a philosophy of extreme efficiency. Rather than chasing raw parameter counts, the team focused on parameter utilization — making each of those 6 billion parameters count for as much as possible.

Here's what sets it apart:

• Mixture of Experts (MoE) architecture. The model doesn't activate all 6 billion parameters for every token. It routes each token through a subset of "experts," activating only around 2 billion parameters at any given time. This keeps memory usage low without sacrificing quality — the total knowledge is spread across all the experts, but only the relevant ones fire per token.
• Knowledge distillation. MiniCPM-O was trained with guidance from much larger teacher models, so it learns to approximate the output quality of a 70B+ model while remaining physically small. This is the same technique that made DeepSeek-R1-Distill models so effective.
• Aggressive quantization support. The model ships with INT4 and even INT2 quantization options that are surprisingly coherent, thanks to quantization-aware training during the distillation process.
• Multimodal capabilities. Despite its small size, MiniCPM-O handles vision and text — you can feed it images and ask questions about them. Running a vision-language model on 2 GB of VRAM should be impossible. MiniCPM-O does it anyway.

The benchmark numbers tell the story. On the Open LLM Leaderboard v2, MiniCPM-O 6B scores competitively with — and in several categories beats — both Llama 3.1 7B and Gemma 2 7B, despite being physically smaller and using a fraction of the memory. On MMLU-Pro, a harder variant of the standard knowledge benchmark, MiniCPM-O 6B actually edges ahead of several 7B and 8B models.

Step 1: Install Ollama

Ollama is the easiest way to run local LLMs. It handles model downloads, quantization, GPU acceleration, and exposes a simple REST API — all without requiring you to touch a Python environment or CUDA configuration.

Windows:

1. Go to ollama.com/download/windows
2. Download the installer and run it
3. Ollama will start automatically as a system tray application

macOS:

1. Download from ollama.com/download/mac
2. Drag to Applications and launch

Linux:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it's working by opening a terminal and running:

ollama --version

You should see something like ollama version 0.6.x. If you get a "command not found" error on Linux, you may need to restart your terminal or add Ollama to your PATH.

Step 2: Pull the MiniCPM-O 6B Model

With Ollama installed, pulling the model takes one command:

ollama pull minicpm-o:6b

This downloads the model in its default 4-bit quantized form, which is about 3.5 GB on disk but only uses ~2 GB of VRAM during inference thanks to the MoE architecture. On a typical broadband connection, the download takes 5-10 minutes.

Want the absolute smallest memory footprint? Try the 2-bit variant:

ollama pull minicpm-o:6b-q2_K

This drops memory usage below 1.5 GB and still produces surprisingly coherent output. It's a great option for truly constrained hardware like older laptops or single-board computers (yes, people have run this on a Raspberry Pi 5 — slowly, but it works).

To verify the model is ready, list your installed models:

ollama list

You should see minicpm-o:6b in the output.

Step 3: Run Your First Prompt

Start an interactive chat session from the terminal:

ollama run minicpm-o:6b

The first time you run it, Ollama loads the model into memory — this takes 3-5 seconds on most systems. After that, you'll see a prompt:

>>> 

Try something simple:

>>> Write a Python function that sorts a list using quicksort

You should see tokens appear at a steady pace. On integrated graphics, expect 5-8 tokens per second — not blazing, but fast enough for reading along. On a discrete GPU (even an older GTX 1060), speeds jump to 15-20 tokens per second.

Let's test the multimodal capabilities too. Save an image to your computer, then pass it in:

>>> Describe this image: /path/to/your/photo.jpg

MiniCPM-O will analyze the image and describe what it sees — all running locally on your 2 GB of VRAM.

When you're done, type /bye to exit.

Step 4 (Optional): Add a ChatGPT-Style Interface with Open WebUI

The terminal is functional, but if you want a polished ChatGPT-like experience — chat history, markdown rendering, image uploads, multiple conversations — Open WebUI is the answer.

Install with Docker (recommended):

docker run -d -p 3000:8080 \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

On Windows, replace http://host.docker.internal:11434 with http://localhost:11434 if Docker Desktop is configured for localhost access.

After the container starts, open http://localhost:3000 in your browser. Create an account (the first account becomes the admin), and you'll see MiniCPM-O 6B available in the model selector at the top of the page.

Now you have a fully private, self-hosted ChatGPT alternative running entirely on your hardware. No data leaves your machine.

Real-World Performance: What to Expect

I tested MiniCPM-O 6B (4-bit quantized) on three different hardware configurations to give you realistic expectations:

The key takeaway: memory usage barely changes across hardware tiers. What changes is speed. On integrated graphics, you get 5-7 tokens per second — about the speed of human reading. That's perfectly usable for interactive chat, code generation, and writing assistance. You won't be doing real-time speech-to-speech translation, but for most day-to-day AI use cases, it's more than adequate.

Quality-wise, MiniCPM-O 6B punches well above its weight class:

• General knowledge: Solid. It handles trivia, explanations, and reasoning tasks on par with 7B models. It won't replace Claude Sonnet for complex research, but it nails the common questions people actually ask their local AI.
• Coding: Surprisingly good. It generates correct, idiomatic Python and JavaScript for most standard tasks. It struggles with obscure libraries and complex multi-file architectures — but so do most 7B models.
• Creative writing: Decent. It can produce blog drafts, emails, and short stories. The prose isn't going to win literary awards, but it's perfectly fine for practical writing tasks.
• Vision: Works. It identifies objects, reads text from images, and can answer questions about visual content. Accuracy is lower than larger vision models like LLaVA 13B, but the fact that it works at all on 2 GB of VRAM is remarkable.

Why Small Models Are the Future of Local AI

There's been an arms race toward ever-larger models — 70B, 405B, even trillion-parameter behemoths. That race makes sense for frontier capabilities. But for the vast majority of real-world use cases, smaller, smarter models are the better play. Here's why:

1. Privacy wins every argument. Running a model locally means your data never leaves your machine. No prompts logged on someone else's server. No conversations stored for training. For developers working with proprietary code, journalists working with sensitive sources, or anyone who just values privacy, local is the only serious option.

2. Cost is zero after setup. No API bills. No per-token pricing. No "you've exceeded your rate limit" messages. Once the model is downloaded, you can chat with it 24/7 at no additional cost. Over a year of daily use, that saves hundreds of dollars compared to ChatGPT Plus or Claude Pro subscriptions.

3. Offline capability is underrated. Flights. Remote areas. Power outages. Server outages. A local model works when the internet doesn't. If you've ever been on a deadline when ChatGPT went down (it happens more often than OpenAI admits), you'll appreciate this.

4. Hardware is catching up fast. The efficiency gains we're seeing in models like MiniCPM-O, combined with hardware improvements like Apple's Neural Engine and Intel's Meteor Lake NPU, are shrinking the gap between what's possible locally and what requires a data center.

5. Specialization beats generality. For most tasks, you don't need a model that knows everything about everything. You need one that's good at the specific things you do. Fine-tuning a small model for your domain — legal documents, technical writing, education — can produce results that rival or exceed a general-purpose giant.

MiniCPM-O 6B is part of a broader trend I'm genuinely excited about. In 2024, running a capable LLM on consumer hardware was a novelty. In 2026, it's practical. By 2027, I suspect we'll look back and wonder why we ever thought we needed 24 GB of VRAM to get useful AI.

Common Issues and Fixes

Here are the most common problems people hit and how to resolve them:

"Ollama says 'model not found'." Check the exact model name with ollama list. The tag matters — minicpm-o:6b is different from minicpm-o:latest.

"It's running on CPU and it's painfully slow." Run ollama ps while a prompt is processing. If you see "100% CPU," your GPU isn't being used. On Windows, make sure your GPU drivers are up to date. On Linux, install the NVIDIA Container Toolkit if using Docker, or verify CUDA is installed for native runs.

"I'm running out of memory." Try the 2-bit quantized version (ollama pull minicpm-o:6b-q2_K). Also close other GPU-using applications — browsers with hardware acceleration can eat hundreds of megabytes of VRAM without you realizing it.

"The quality is worse than I expected." Are you on the 2-bit model? Try the 4-bit version first for a baseline. Also, MiniCPM-O benefits from clear, specific prompts — it doesn't handle vague requests as gracefully as much larger models do.

Going Further

Once you have the basics working, here are a few ways to level up your setup:

• Fine-tune for your domain. MiniCPM-O supports LoRA fine-tuning with the llama.cpp project's fine-tuning tools. Train it on your own documents or codebase to create a specialized assistant.
• Run in CI/CD pipelines. Because it's so lightweight, you can run MiniCPM-O in GitHub Actions or other CI environments for automated code review, documentation generation, or test case creation.
• Build RAG applications. Pair MiniCPM-O with a local vector database like ChromaDB or LanceDB. Index your documents, and you have a private, local alternative to NotebookLM that answers questions from your own knowledge base.
• Multi-model setups. Run multiple small models simultaneously on the same machine — MiniCPM-O for chat, a specialized coding model for development, and an embedding model for search. None of them individually exceed 2 GB VRAM.
• Home automation integration. With Ollama's API and Home Assistant, you can build natural language interfaces for smart home control — all processed locally, with no cloud dependency.

If you've been holding off on running local AI because you don't have a flagship GPU, stop waiting. The barrier to entry has never been lower. MiniCPM-O 6B on Ollama takes five minutes to set up and gives you a genuinely useful AI assistant that runs on hardware you already own.

Try it. You might be surprised at what 2 GB of VRAM can actually do.