LLM Hardware Recommendations

This document contains hardware requirements and recommendations for AI components.

For running LLM

Running Gemma 4 26b through vLLM requires a single GPU with sufficient VRAM to hold the selected model. We were running and testing google/gemma-4-26B-A4B-it.

Minimum Required Setup

Component	Requirement
GPUs	1
VRAM	48 GB or more
Example GPU	NVIDIA L40S (48 GB)

Minimal Recommended Setup

Component	Recommendation
GPUs	1-2
VRAM	96 GB or more
GPU	NVIDIA RTX 6000 Pro (96 GB)
GPU	2x NVIDIA A100, H100 (160 GB)

We also ran VLLM alongside TEI, so both LLM, and embeddings model can run on a single machine.

Performance Notes

With a lower-grade GPU (e.g. 48 GB class), inference will work but may be very slow for chatting and interactive use. A minimum-spec setup is better suited for automation with n8n or background tasks where latency is not critical. For responsive interactive chat, use the recommended GPU or higher or use quantized models.

TEI (Text Embeddings Inference)

For TEI hardware requirements, see Embeddings hardware recommendations.