Want to Get Into Modeling?¶

Local large language model deployment requires navigating a landscape of inference engines, hardware constraints, and quantization formats. This page provides a decision framework for selecting the appropriate tooling stack based on hardware, use case, and performance requirements. It includes a comprehensive comparison of vLLM and Ollama, the two dominant local inference frameworks, based on benchmark data from NVIDIA Blackwell hardware.

Overview¶

Decision Framework¶

flowchart TD
    A["Need local LLM?"] --> B{"Multiple clients?"}
    B -->|Yes| C["vLLM<br/>PagedAttention"]
    B -->|No| D{"NVIDIA 16GB+ VRAM?"}
    D -->|Yes| E["vLLM + TensorRT-LLM"]
    D -->|No| F{"CPU / Apple Silicon?"}
    F -->|Yes| G["Ollama / llama.cpp"]
    F -->|No| H{"Quick setup?"}
    H -->|Yes| G
    H -->|No| I["Compare benchmarks<br/>on this page"]

    style C fill:#2d5a3a,color:#fff
    style G fill:#4a7c59,color:#fff
    style E fill:#2d5a3a,color:#fff

vLLM vs Ollama: Benchmark Analysis¶

The following data is derived from independent benchmarks conducted on NVIDIA Blackwell (RTX 5090) hardware with 96 GB VRAM, testing both vLLM 0.11.0 and Ollama 0.10.5 with the Gemma 4 27B NVFP4 model.

Throughput Benchmarks (requests/second)¶

Latency Benchmarks (time to first token, milliseconds)¶

Memory Efficiency (VRAM usage, GB)¶

Key Findings¶

1. Single-user scenarios: Ollama has lower time-to-first-token (38ms vs 45ms) and simpler setup. For personal desktop use with one client, Ollama is the pragmatic choice.

2. Multi-user scenarios: vLLM's PagedAttention and continuous batching provide 3.5x throughput at batch=32. The gap widens as concurrency increases.

3. KV cache: vLLM's non-contiguous KV cache management uses 2.16x less memory for the same batch size, enabling larger batches or longer contexts.

4. Throughput saturation: vLLM saturates the GPU at batch=32 (94.7 req/s). Ollama saturates at batch=8 (24.1 req/s) and does not scale beyond.

vLLM: Architecture and Configuration¶

vLLM is a high-throughput inference engine featuring PagedAttention for efficient KV-cache memory management. It supports continuous batching, tensor parallelism, and speculative decoding.

Installation¶

# Pre-built wheels (CUDA 12.1+)
pip install vllm

# From source for specific CUDA version
pip install https://wheels.vllm.ai/nightly/cu123/vllm/

# Verify installation
python -c "import vllm; print(vllm.__version__)"

Serving a Model¶

# Single GPU
vllm serve "meta-llama/Llama-3.1-8B-Instruct" \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 4096

# Multi-GPU
vllm serve "meta-llama/Llama-3.1-70B-Instruct" \
    --tensor-parallel-size 4

# With quantization (AWQ)
vllm serve "TheBloke/Llama-2-7B-AWQ" \
    --quantization awq \
    --dtype half

torch.compile Optimization¶

import vllm
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    compilation_config={
        "level": 3,  # Max optimization
        "backend": "inductor",
        "mode": "reduce-overhead",
        "fullgraph": True,
    }
)

Ollama: Architecture and Configuration¶

Ollama provides a single-binary deployment model with built-in model management and a REST API. It uses llama.cpp as the inference backend.

Installation¶

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download installer from ollama.com

Running a Model¶

# Pull and run
ollama pull llama3.1:8b
ollama run llama3.1:8b

# Start API server
ollama serve

# API endpoint: http://localhost:11434
# OpenAI-compatible endpoint available

Configuration¶

# Modelfile for customization
cat > Modelfile << 'EOF'
FROM llama3.1:8b
PARAMETER temperature 0.7
PARAMETER top_p 0.9
SYSTEM You are a helpful coding assistant.
EOF

ollama create mymodel -f Modelfile
ollama run mymodel

Quantization Formats¶

NVFP4 (NVIDIA Blackwell)¶

NVFP4 is a 4-bit floating-point format introduced with NVIDIA Blackwell (RTX 50-series, H200). It uses 2-bit exponent and 1-bit mantissa with a shared scale factor per 16-element block, providing higher quality than INT4 quantization at the same memory footprint.

# vLLM with NVFP4 (requires Blackwell GPU)
llm = LLM(
    model="nvidia/Llama-3.1-8B-NVFP4",
    quantization="nvfp4",
    dtype="float16"
)

Hardware Requirements¶

CPU Inference with llama.cpp¶

# Download quantized model
wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf

# Run inference
./main -m llama-2-7b.Q4_K_M.gguf -n 256 --temp 0.8 -p "The capital of France is"

# With OpenAI-compatible server
./server -m llama-2-7b.Q4_K_M.gguf --host 0.0.0.0 --port 8080

AI Agent Stack¶

The full stack for local AI development:

Terminal Harness Tools¶

IDE Agents¶

Redirect Proxies¶

Local Setup Guides¶

NVIDIA Free AI Coding Agent¶

Community guides for setting up local AI coding agents on NVIDIA hardware:

DeepWiki References¶

Related Deep Hole¶

• Allen Kuo: vLLM or Ollama on Blackwell? Benchmarks, Landmines, and What Agents Actually Need — Benchmark data source
• Allen Kuo: Gemma 4 on vLLM vs Ollama Benchmarks on a 96 GB Blackwell GPU — Head-to-head comparison
• Allen Kuo: vLLM on Blackwell with TurboQuant KV Cache — Windows vLLM setup
• Julsimon: What to Buy for Local LLMs April 2026 — Hardware recommendations
• vLLM Documentation — Official inference engine documentation
• Ollama GitHub — Source code and releases
• Local AI Wiki — Community LLM testing
• Hermes Agent Documentation — Nous Research agent framework