Businesses are moving toward self-hosting for a few concrete reasons:
- Keeping proprietary data off public AI systems
- Predictable, flat infrastructure costs instead of per-token bills
- Reducing dependence on third-party uptime, pricing, and policy changes
- Supporting private workflows tied to internal databases and documents
This article walks through a decision framework: calculating VRAM and RAM needs, comparing hardware tiers, planning supporting infrastructure, and accounting for security and growth.
Key Takeaways
- VRAM is usually your first bottleneck for LLM inference; system RAM covers the rest of the stack.
- Quantization can shrink a model's footprint dramatically, with quality and compatibility trade-offs.
- A single-GPU box works for testing; production and multi-user workloads need multi-GPU capacity and headroom.
- Plan beyond hardware: networking, access control, and encryption shape the final deployment.
What Determines Self-Hosted AI Hardware Requirements?
Start with model size and GPU VRAM
The starting formula is simple: weight memory equals parameter count multiplied by bytes per parameter. Hugging Face's model memory estimator puts an 80B model at 148.56 GB in float16, 74.28 GB in int8, and 37.14 GB in int4 — a huge swing based purely on precision choice.
The downloaded model file still isn't the full story. NVIDIA's 2025 technical blog notes that Llama 3 70B needs roughly 140 GB in FP16 for weights alone, and a single user running a 128K context window can add another 40 GB of KV cache on top.
That's the trap: a model that "fits" on paper can still choke on real inference because of:
- Context window and KV cache overhead
- Runtime buffers and embeddings
- Concurrent request handling
System RAM is more than a backup for VRAM
RAM isn't just there in case the GPU runs out of room. It supports the OS, CPU-based inference, tokenization, vector databases, document processing, and any containers or services running alongside your model.
vLLM's offload documentation shows this trade-off clearly: 10 GiB of CPU offload can make a 24 GB GPU behave like a 34 GB one for loading purposes, but it demands fast CPU-GPU bandwidth since weights shuttle back and forth during inference.
Rough starting points:
- 32 GB — small GPU-resident quantized model, light testing
- 64 GB — offloading or a vector database in play
- 128 GB+ — serious 30B–70B offload or RAG workloads
- 256 GB+ — multiple concurrent users, large document corpora
Loading a model is not the same as serving it responsively alongside a RAG pipeline and a database connector.
Account for quantization, context length, and concurrency
| Format | What it means | Memory impact |
|---|---|---|
| FP16/BF16 | ~2 bytes per weight | Highest memory, reference quality |
| 8-bit | Half the memory of FP16 | Good quality baseline |
| 5-bit | GGUF-style middle ground | Varies by exact variant |
| 4-bit | About a quarter of FP32 | Smallest footprint, more quality risk |
llama.cpp's own benchmarks on Llama 3.1 8B show Q4_K_M at 4.58 GiB running 71.93 tokens/sec, versus Q8_0 at 7.95 GiB running 50.93 tokens/sec — smaller isn't always slower. But llama.cpp also warns that accuracy loss should be measured with perplexity or KL divergence, not assumed.
Longer prompts, bigger document sets, and more simultaneous users all multiply KV cache demand. Use the Hugging Face estimator for weights, then load-test your actual runtime with representative traffic before locking in a hardware order.
Match hardware to latency and throughput expectations
Match the box to the job:
- Interactive chat — low latency; plan on GPU
- Batch summarization or embedding — delay-tolerant; CPU-only can work
- Document extraction or coding help — GPU preferred as volume rises
- Image generation — GPU-bound in nearly every case
CPU-only inference can handle low-volume, non-interactive processing just fine. It's a poor fit for a team-facing chatbot with several people hitting it at once. Test with your actual prompts and documents rather than trusting theoretical benchmarks alone.
Check accelerator and software compatibility
GPU memory is only half the equation. Drivers, CUDA/ROCm versions, OS support, and runtime compatibility all matter just as much:
- Ollama — broadest support: NVIDIA, AMD ROCm 7, Apple Metal, Vulkan
- vLLM — strongest on Linux with CUDA or ROCm; Apple support is a community plugin, no native Windows
- llama.cpp / LocalAI — covers CUDA, ROCm, Metal, Vulkan, and Intel SYCL
Before buying anything, confirm your intended runtime actually supports your target hardware. Skipping that check is a common source of delay once gear is on the floor. With AI-ABW deployments, hardware, OS, network configuration, and database access are confirmed as a formal pre-deployment step so the stack matches the environment before go-live.
2026 Hardware Tiers for Self-Hosted AI
CPU-only and low-power starter systems
Small quantized models can run without a discrete GPU for experimentation, classification, or low-volume internal tools. Trade-offs include slower response times, smaller context windows, and limited concurrent users, but also lower energy draw and simpler setup.
Single-GPU developer and small-team systems
A practical single-GPU setup needs:
- A mid-range VRAM class GPU (commonly 16–24 GB)
- 32–64 GB system RAM
- Solid NVMe storage
- A PSU sized with headroom for the card's draw
This tier suits prototyping, a private assistant, or one department's knowledge base, not unrestricted company-wide access.
Prosumer systems with higher-memory GPUs
A higher-memory consumer or workstation card opens up larger quantized models, longer context windows, and more simultaneous requests.
The NVIDIA RTX 5090 ships with 32 GB GDDR7 and draws 575 W, requiring a 1,000 W system PSU. The AMD Radeon PRO W7900 offers 48 GB ECC memory at 295 W but needs triple-slot clearance.
More VRAM doesn't automatically mean better results. A poorly matched model, quantization format, or cooling setup can still bottleneck performance.
Multi-GPU and high-memory workstation systems
Distributing a model across GPUs helps with capacity (fitting bigger models) or throughput (serving more users). Those are different goals, and they need different configurations.
NVLink offers up to 900 GB/s for tensor-parallel traffic versus 128 GB/s over standard PCIe Gen5. VRAM still does not become additive automatically; the inference engine has to implement tensor or pipeline parallelism.
Practical considerations:
- Chassis space for multiple large cards
- Power delivery sized for combined draw
- Matched GPU types (as LocalAI recommends for best results)
- Driver and monitoring overhead
Enterprise and production infrastructure
High-concurrency or very large-model deployments call for data-center-class accelerators such as the H100 (80–94 GB, up to 700 W configurable) or L40S (48 GB, 350 W). Plan for redundant systems, fast networking, and shared storage alongside the GPUs.
At this scale, compare a self-built stack with dedicated servers, colocation, or a private self-hosted AI platform on infrastructure you control. If maintenance, staffing, and support complexity outweigh the control benefits, choose a managed private deployment path instead of building every layer in-house.
Supporting Hardware and Infrastructure Beyond the GPU
CPU, motherboard, PCIe, and expansion planning
CPU cores, PCIe lane availability, and motherboard slot spacing all affect multi-GPU deployment and future upgrades. Before buying, check:
- Physical GPU dimensions against case and motherboard clearance
- Available PCIe lanes for planned GPU count
- CPU performance sufficient to feed GPUs without bottlenecking
- Room for future expansion
Storage, networking, and data protection
Fast NVMe storage speeds up model loading and switching. Size capacity for multiple model versions plus backups.
Keep inference traffic private and recoverable:
- Keep model endpoints off the open internet
- Prefer private networking for internal AI services
- Back up configurations, prompts, indexes, and business documents
Power, cooling, chassis, and physical reliability
GPU power draw, PSU capacity, and airflow all affect stability under sustained load. Used or datacenter GPUs sometimes need special cooling or power connectors not found in standard towers.
For always-on business systems, plan for:
- UPS protection against outages and dirty power
- Dust management and sustained airflow
- Temperature monitoring under continuous load
Match Hardware to Workload and Business Deployment Needs
Define the workload before selecting the server
Build an intake checklist before shopping for hardware:
- Model family, size, and quantization
- Context length and prompts per day
- Simultaneous users and response-time goals
- Document or database sources involved
- Whether fine-tuning is required Ordinary inference, RAG, embeddings, and fine-tuning each create different CPU, GPU, and concurrency demands. Don't size for one workload and assume it covers the rest.
Plan for proof of concept, departmental use, and production
Deployment stage changes the hardware brief:
- Single-user POC: lower capacity, limited uptime needs, informal support
- Departmental server: shared concurrency, clearer response-time targets
- Production: peak capacity, uptime expectations, defined support ownership Size for peak demand, not average load. Model loading and simultaneous users spike resource use in ways averages hide. Start with one model and one use case, then scale gradually.
Choose the software stack alongside the hardware
Beginner-friendly runtimes like Ollama or LM Studio simplify setup. Production-oriented options like vLLM or llama.cpp give more control over concurrency and administration. AI-ABW runs on llama.cpp to keep capable models on real-world business hardware rather than data-center gear. That middle path suits teams that want private inference without managing bare-metal infrastructure themselves.
Treat privacy and compliance as architecture requirements
Self-hosting keeps prompts and documents inside your environment. It does not automatically satisfy compliance. You still need:
- Role-based access and identity management
- Encryption in transit and at rest
- Audit logs and retention policies
- Network segmentation and patching Legal, healthcare-adjacent, and other regulated teams feel this gap first: hardware alone does not meet HIPAA or attorney-client privilege obligations. AI-ABW addresses part of the risk by design. Its data layer is read-only, so it cannot change, delete, or add business records, and SQL Server connections use controlled read-only views rather than open access.
Weigh total cost against a private AI platform
Weigh hardware purchase, electricity, cooling, maintenance, and staff time against recurring API costs. Self-hosting tends to be more predictable for steady workloads with high privacy requirements; hosted APIs often make more sense for experimentation or teams without infrastructure capacity. For teams that want self-hosted privacy without designing the full stack, AI-ABW uses a flat, fixed-environment cost model with no token fees or per-query charges. It runs on the customer's own hardware or a dedicated private cloud, with no outbound API calls or shared infrastructure. Info-Power assesses hardware, OS, and network requirements before deployment so customers are not left guessing at sizing.
Conclusion: Choose the Smallest Reliable System That Meets the Real Requirement
The best 2026 hardware choice is the smallest system that reliably meets your target model, response time, concurrency, and growth needs. Oversizing wastes money on power and cooling. Undersizing means frustrated users and a system that can't keep up.
Next step: Define one business use case, test it against a representative model and dataset, and document the performance and controls you actually need. From there, buy hardware, scale gradually, or use a private self-hosted AI platform to run the stack on infrastructure you control.
Frequently Asked Questions
What kind of hardware is needed for a self-hosted LLM?
You'll need a GPU with sufficient VRAM for your model and quantization choice, system RAM for supporting services, adequate storage, and a PSU sized for the GPU's draw. Exact requirements vary by model size, context length, and number of users.
How much RAM do you need to run your own AI?
System RAM is separate from GPU VRAM and supports the OS, offloading, and services like vector databases. A testing setup might need 32 GB, while production RAG deployments often require 128 GB or more.
Can you run a self-hosted AI model without a GPU?
Yes, for smaller quantized models and batch or non-interactive workloads. CPU-only inference struggles with responsiveness for larger models or multiple simultaneous users, where a GPU is generally the better choice.
How much storage does a self-hosted AI server need?
Storage needs to cover the model files, OS, containers, embeddings, datasets, logs, and backups. Leave room for multiple model versions since upgrades rarely mean deleting the old one immediately.
How do you choose a GPU for self-hosted AI in 2026?
There isn't one universal answer — it depends on VRAM needs, workload, software compatibility, power budget, and total cost. Check current specifications for your specific model and runtime before deciding.
Is self-hosted AI cheaper than using an AI API?
It depends on usage volume, privacy requirements, and staffing capacity. Self-hosting tends to pay off with steady, high-volume usage and strict data control needs, while APIs often suit lower-volume or experimental use.
