Skip to content

Self-Hosted AI Server Setup

Use this guide when the customer runs self-hosted AI components on their own server and the host must be prepared before deploying vLLM or TEI.

This page keeps the host-level preparation in a separate place from the product installation scenarios.

Baseline used for the original setup

The original setup was tested on the following baseline:

Hardware

  • architecture: x86_64
  • CPU: 16 cores, example AMD EPYC 7302
  • RAM: 48 GiB
  • GPU: 2x NVIDIA RTX A6000

Software

  • operating system: Debian 12.6 or compatible
  • Docker: 27.0.3 or newer
  • Docker Compose plugin: 2.28.1 or newer
  • KVM virtualization

Treat this as a tested reference, not as the only supported combination.

Host prerequisites

Before installing self-hosted AI services, make sure you have:

  • root access on the server
  • Docker Engine installed
  • Docker Compose plugin available as docker compose
  • NVIDIA GPU available when the selected model requires GPU acceleration

If Docker is not installed yet, use the official guides:

NVIDIA GPU and container runtime setup

For Docker containers to access the NVIDIA GPU, install the drivers, CUDA packages, and NVIDIA container runtime on the host.

These commands are taken from the original server setup guide and are intended for Debian 12.

Step 1: Install required packages

apt update && apt install -y gcc ca-certificates curl gnupg net-tools tclsh unzip python3-pip software-properties-common wget

Step 2: Install Linux headers

apt install -y linux-headers-$(uname -r)

Step 3: Add CUDA keyring

cd /tmp
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update

Step 4: Install CUDA

DEBIAN_FRONTEND=noninteractive apt-get -y -q install cuda

Step 5: Add NVIDIA signing key

apt-key adv --fetch-keys https://nvidia.github.io/nvidia-container-runtime/gpgkey

Step 6: Add NVIDIA container runtime repository

curl -s -L https://nvidia.github.io/nvidia-container-runtime/debian12/nvidia-container-runtime.list | tee /etc/apt/sources.list.d/nvidia-container-runtime.list

Step 7: Install NVIDIA container runtime

apt update
apt install -y nvidia-container-runtime

Step 8: Restart Docker

systemctl restart docker

Verify the host before deploying AI services

Before starting vLLM or TEI, verify:

  • nvidia-smi detects the GPU
  • Docker is running
  • docker compose is available
  • the server has enough GPU memory for the selected model

Then continue with:

Security practices

When exposing self-hosted AI services, also consider:

  1. Use strong and unique credentials for all exposed services.
  2. Restrict open ports with firewall rules.
  3. Keep the host and installed packages updated.
  4. Use HTTPS for external access.
  5. Apply access control and user management.
  6. Audit the server regularly.
  7. Use encrypted communication where possible.