Prepare a GPU server¶

Prepare a GPU host before you deploy vLLM or TEI.

Tested baseline¶

This setup was tested on:

Hardware¶

architecture: x86_64
CPU: 16 cores, example AMD EPYC 7302
RAM: 48 GiB
GPU: 2x NVIDIA RTX A6000

Software¶

operating system: Debian 12.6 or compatible
Docker: 27.0.3 or newer
Docker Compose plugin: 2.28.1 or newer
KVM virtualization

This is a tested reference, not the only supported combination.

Host prerequisites¶

You need:

root access on the server
Docker Engine installed
Docker Compose plugin available as docker compose
an NVIDIA GPU, when the selected model requires GPU acceleration

If Docker is not installed yet, use the official guides:

NVIDIA GPU and container runtime setup¶

For Docker containers to use the NVIDIA GPU, install the drivers, CUDA packages and the NVIDIA Container Toolkit on the host. The commands are for Debian 12 and run as root.

Install required packages¶

apt update && apt install -y gcc ca-certificates curl gnupg net-tools tclsh unzip python3-pip software-properties-common wget

Install Linux headers¶

apt install -y linux-headers-$(uname -r)

Add CUDA keyring¶

cd /tmp
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update

Install CUDA¶

DEBIAN_FRONTEND=noninteractive apt-get -y -q install cuda

Install the NVIDIA Container Toolkit¶

Add the NVIDIA repository and its key:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Install the toolkit, register the NVIDIA runtime in Docker and restart Docker:

apt update
apt install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Check that a container sees the GPU:

docker run --rm --gpus all ubuntu nvidia-smi

The NVIDIA Container Toolkit install guide has the current steps for other distributions.

Verify the host before deploying AI services¶

Before you start vLLM or TEI, check that:

nvidia-smi shows the GPU
Docker is running and docker compose works
the GPU has enough memory for the model

Then continue with the vLLM deployment example and the TEI deployment example. See also Embeddings hardware recommendations.

Security practices¶

When you expose self-hosted AI services:

use strong, unique credentials
restrict open ports with firewall rules
keep the host and packages updated
use HTTPS for external access