Building a Multi-GPU Inference Stack on Consumer Hardware

Cloud inference APIs charge $0.01-0.06 per thousand tokens. If you are running production AI systems that process thousands of requests per day — sales agents, voice AI, content generation, autonomous operators — those costs add up to $50-200/month. I wanted zero ongoing cost, full data privacy, and no rate limits.

So I built a multi-GPU inference server on consumer hardware. Two GPUs — a Tesla M40 24GB and an RTX 3060 12GB — running Qwen3.8-27B at 20 tokens per second inside a Proxmox virtual machine. Total hardware cost: under $300. Monthly operating cost: electricity only. This article covers the full build, the benchmarks, and every gotcha I encountered mixing GPU generations.

The Hardware

The GPU selection was driven by one constraint: fit a 27B parameter model entirely in VRAM across both cards. The model at Q2_K_P quantization is 20.61 GiB. I needed at least 23 GiB of combined VRAM to leave headroom for KV cache and context.

• Tesla M40 24GB — Maxwell architecture (2015), $60-80 on eBay. No display output, no video encode/decode, no FP16 tensor cores. What it does have: 24 GB of GDDR5 VRAM and a 288 GB/s memory bandwidth. For inference workloads that are memory-bandwidth-bound, this is the cheapest VRAM per dollar you can buy.
• RTX 3060 12GB — Ampere architecture (2021), ~$200 used. 12 GB GDDR6, 360 GB/s bandwidth, FP16 tensor cores. Roughly 3x the compute throughput of the M40 per CUDA core, plus hardware acceleration for quantized operations.

Combined: 36 GB of VRAM for under $300. For comparison, an RTX 4090 with 24 GB costs $1,600+, and an A100 with 80 GB costs $10,000+. The multi-GPU approach trades simplicity for cost efficiency.

GPU Passthrough in Proxmox

Both GPUs sit in a Dell PowerEdge server running Proxmox VE as the hypervisor. The inference workload runs in a Linux VM (VM127) with both GPUs passed through via VFIO. Getting GPU passthrough working on mixed-generation NVIDIA cards has specific requirements:

Step 1: Enable IOMMU

IOMMU (Input-Output Memory Management Unit) is required for PCI passthrough. On Intel systems, this means enabling VT-d in the BIOS and adding the kernel parameter:

# /etc/default/grub — Proxmox host
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# After editing:
update-grub
reboot

The iommu=pt (passthrough mode) is critical for performance. Without it, all DMA transactions go through the IOMMU translation layer, adding latency to every memory access. With pt, only devices assigned to VMs use IOMMU; the host's devices bypass it entirely.

Step 2: Bind GPUs to VFIO

The GPUs must be claimed by the vfio-pci driver on the host instead of the NVIDIA driver. This happens at boot via device IDs:

# /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:17fd,10de:2504
# 10de:17fd = Tesla M40
# 10de:2504 = RTX 3060

# /etc/modules — load vfio modules early
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# Blacklist NVIDIA drivers on host
# /etc/modprobe.d/blacklist-nvidia.conf
blacklist nouveau
blacklist nvidia
blacklist nvidia_drm

After a reboot, lspci -nnk should show both GPUs bound to vfio-pci instead of nvidia. If a GPU is in the same IOMMU group as other devices (common on consumer motherboards), you may need to pass through the entire group or use the ACS override patch.

Step 3: VM Configuration

The VM must boot in UEFI mode (not legacy BIOS) for GPU passthrough to work reliably. In the Proxmox VM config:

# /etc/pve/qemu-server/127.conf (relevant lines)
bios: ovmf
machine: q35
cpu: host
hostpci0: 0000:41:00,pcie=1    # Tesla M40
hostpci1: 0000:01:00,pcie=1    # RTX 3060
memory: 32768
balloon: 0                      # Disable ballooning for GPU workloads

Memory ballooning must be disabled. GPU workloads allocate VRAM through the driver, and if Proxmox tries to reclaim memory from the VM via ballooning, the GPU driver can crash or the inference process can OOM.

The Inference Stack: llama.cpp

I use llama.cpp for inference because it handles multi-GPU setups natively, supports every quantization format, and has zero Python dependencies. The critical configuration for mixed-generation GPUs:

# Launch command for Qwen3.8-27B on M40 + 3060
./llama-server \
  --model Qwen3.8-27B-128K-Q2_K_P.gguf \
  --host 0.0.0.0 --port 8090 \
  --ctx-size 65536 \
  --parallel 1 \
  --tensor-split 1,0 \
  --split-mode layer \
  --flash-attn \
  --threads 8 \
  --cont-batching

The two flags that matter most:

--tensor-split 1,0 — This controls how model layers are distributed across GPUs. The values are proportional weights, not percentages. 1,0 means "put everything on GPU 0 (the 3060), nothing on GPU 1 (the M40)." This sounds counterintuitive — why have two GPUs if you are not using one? The answer is that the M40 holds the overflow. The 3060 gets all layers it can fit, and layers that do not fit spill to the M40. The reason to put the faster GPU first is that it processes most of the layers during generation. The M40's slower compute only applies to the overflow layers.

--split-mode layer — This must be layer, never row. Layer splitting means each GPU processes complete transformer layers. Row splitting means each layer is split across GPUs, which requires inter-GPU communication on every layer. On PCIe (not NVLink), row splitting destroys performance because the PCIe bus becomes the bottleneck. I tested both: layer splitting gave 19.97 tok/s. Row splitting gave 6.2 tok/s. Never use row splitting on PCIe.

Benchmarks: Real Numbers

All benchmarks are prompt processing + generation on the same hardware (VM127, 8 vCPUs, 32GB RAM, M40 + 3060):

Model: Qwen3.8-27B-128K-Q2_K_P (20.61 GiB)
Context: 65536 tokens
Parallelism: 1

GPU Config              | tok/s  | Notes
3060 primary (1,0)      | 19.97  | Best - fast GPU processes most layers
M40 primary (0,1)       | 14.31  | Slower - M40 bottlenecks generation
Even split (1,1)        | 16.82  | M40 slows down its half
3060 only               | OOM    | 12GB not enough for 20.61 GiB model
M40 only                | 11.24  | Fits but Maxwell is slow
--parallel 4 (1,0)      | 34.53  | 1.73x with batch, higher latency per request

The key takeaway: GPU order matters enormously. Putting the faster GPU first in --tensor-split gave a 40% speed improvement over the reverse order. This is because during autoregressive generation, each token is processed sequentially through every layer. The GPU holding the majority of layers dominates the per-token latency. Putting the fast GPU there and letting the slow GPU handle overflow is optimal.

With --parallel 4, throughput nearly doubles to 34.53 tok/s because llama.cpp can batch multiple requests and process them simultaneously. The tradeoff is higher latency per individual request, which matters for interactive use cases like the voice AI agent where time-to-first-token is critical. For batch workloads like content generation by the autonomous operator, parallel 4 is strictly better.

Quantization Tradeoffs

The model uses Q2_K_P quantization — an aggressive 2-bit quantization that compresses the 27B model from ~54 GiB (FP16) to 20.61 GiB. This is the tradeoff that makes the build possible on 36 GB of combined VRAM.

How much quality do you lose at Q2_K? Honestly, for the workloads I run — sales agent responses, receptionist conversations, content drafts — the difference between Q2_K and Q4_K_M is marginal. The model still follows instructions, handles context, and generates coherent responses. Where Q2_K falls short is in reasoning-heavy tasks: multi-step math, complex code generation, or tasks requiring precise factual recall. For those, I route to Claude via API.

If I had more VRAM, I would prefer Q4_K_M (the sweet spot for quality vs. size), but it would push the model to ~30 GiB, which would not fit with enough KV cache headroom for 65K context. The constraint is the hardware, and Q2_K is the best trade within that constraint.

Running as a Production Service

The inference server runs as a systemd service with automatic restart and health monitoring:

# /etc/systemd/system/qwen-llama.service
[Unit]
Description=Qwen 27B LLM Inference Server
After=network.target

[Service]
Type=simple
User=llm
WorkingDirectory=/opt/llama.cpp
ExecStart=/opt/llama.cpp/llama-server \
  --model /models/Qwen3.8-27B-128K-Q2_K_P.gguf \
  --host 0.0.0.0 --port 8090 \
  --ctx-size 65536 --parallel 1 \
  --tensor-split 1,0 --split-mode layer \
  --flash-attn --threads 8 --cont-batching
Restart=always
RestartSec=10
Environment=CUDA_VISIBLE_DEVICES=0,1

[Install]
WantedBy=multi-user.target

The health monitoring is handled by the autonomous operator, which pings the /health endpoint every hour and restarts the service if it fails to respond. In six months of operation, the server has crashed exactly twice — both times due to Proxmox host memory pressure (an unrelated VM was leaking memory). The inference process itself has been rock solid.

Cost Comparison: Self-Hosted vs. Cloud

The math that justifies this build:

Self-hosted (one-time + electricity):

• Tesla M40 24GB: $70 (eBay)
• RTX 3060 12GB: $200 (used)
• Electricity (server already running): ~$15/month incremental for GPU load
• Year 1 total: $450. Year 2: $180.

Cloud API (Claude Haiku for equivalent workload):

• ~10,000 requests/month at ~500 tokens each: $50-100/month
• Year 1 total: $600-1,200. Year 2: $600-1,200.

The self-hosted stack breaks even in 3-6 months and saves $400-1,000 per year thereafter. More importantly, it has zero rate limits, zero vendor dependency, and complete data privacy — every prompt and response stays on hardware I physically control.

The tradeoff is operational complexity. When the cloud API goes down, that is Anthropic's problem. When my GPU server goes down, it is mine. The autonomous health monitoring and systemd auto-restart mitigate this, but there is no escaping the fact that self-hosting means self-managing.

What I Would Tell Someone Building This

Buy the M40 first. At $60-80 for 24 GB of VRAM, there is no cheaper way to experiment with large models. It is slow, but it works. You can run a 13B model comfortably on a single M40, and that is enough to prototype most applications. Add the faster GPU later when you need production speed.

Do not chase NVLink. NVLink is the high-speed GPU interconnect used in data centers. Consumer cards do not have it, and the M40 does not support it with Ampere cards regardless. PCIe is fine for layer-split inference. The bandwidth bottleneck only matters for row splitting, which you should not use anyway.

UEFI boot is non-negotiable. Legacy BIOS boot with GPU passthrough is unreliable at best and broken at worst. OVMF (the UEFI firmware for Proxmox VMs) handles GPU initialization correctly. If you are running a VM with GPU passthrough on legacy BIOS, switch to UEFI before debugging anything else.

Disable memory ballooning. This one cost me two days of debugging. The GPU driver allocates VRAM through the VM's memory space. If Proxmox reclaims memory via ballooning, the driver loses its allocations and the inference process crashes with cryptic CUDA errors. Set balloon: 0 in the VM config and move on.

The model ceiling is real. With 36 GB of combined VRAM, the practical ceiling is ~36B parameters at Q4 quantization. Larger models require more aggressive quantization, which degrades quality faster than the extra parameters improve it. For my workloads, 27B at Q2_K is the sweet spot. If I needed a larger model, I would add another M40 ($70) rather than replace either card.

Related Articles