vLLM deployment example¶

This example runs vLLM with an nginx reverse proxy that checks an API key. For GPU requirements, see LLM hardware recommendations. If TEI runs on the same GPU, lower --gpu-memory-utilization.

Docker Compose setup¶

Directory structure¶

vllm-setup/
├── vllm.env
├── vllm-nginx.conf
└── docker-compose.yaml

Environment configuration (vllm.env)¶

MODEL=google/gemma-4-26B-A4B-it
LOCAL_MODEL_CACHE_DIR=./data
HF_TOKEN=your_huggingface_token

Docker Compose (docker-compose.yaml)¶

services:
  vllm:
    image: vllm/vllm-openai:latest
    pull_policy: always
    container_name: vllm-gemma-4-26b
    expose:
      - 8000
    volumes:
      - ${LOCAL_MODEL_CACHE_DIR:-./data/vllm}:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - HF_HUB_ENABLE_HF_TRANSFER=False
      - NVIDIA_VISIBLE_DEVICES=all
    restart: always
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command:
      - '--model'
      - '${MODEL}'
      - '--max-model-len'
      - '262144'
      - '--tensor-parallel-size'
      - '1'
      - '--gpu-memory-utilization'
      - '0.95'
      - '--max-num-seqs'
      - '32'
      - '--enable-chunked-prefill'

  nginx:
    image: nginx:stable
    container_name: nginx
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./vllm-nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - /etc/certs:/etc/certs:ro
    restart: always

Nginx configuration (vllm-nginx.conf)¶

This configuration requires a bearer token for access from outside. Replace your_api_key and set the same value as VLLM_API_KEY in the Easy8 .env file.

server {
    listen 80;
    server_name llm.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name llm.example.com;

    ssl_certificate /etc/certs/fullchain.pem;
    ssl_certificate_key /etc/certs/privkey.pem;

    location / {
        if ($http_authorization != "Bearer your_api_key") {
            return 401;
        }
        proxy_pass http://vllm:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 600s;
        proxy_connect_timeout 600s;
        proxy_send_timeout 600s;
        proxy_buffering off;
        proxy_cache off;
    }
}

Usage¶

Create a directory for vLLM and copy the files above
Update vllm.env with your HuggingFace token
Start the services:

docker compose --env-file vllm.env up -d

Test the API through nginx:

curl -H "Authorization: Bearer your_api_key" https://llm.example.com/v1/models

Set LLM_URL=https://llm.example.com in the Easy8 .env file.