vLLM Deployment Example

Docker Compose Setup

This example sets up vLLM with an nginx reverse proxy for API key authentication.

Directory Structure

vllm-setup/
├── vllm.env
├── vllm-nginx.conf
└── docker-compose.yaml

Environment Configuration (vllm.env)

MODEL=google/gemma-4-26B-A4B-it
LOCAL_MODEL_CACHE_DIR=./data
HF_TOKEN=your_huggingface_token

Docker Compose (docker-compose.yaml)

services:
  vllm:
    image: vllm/vllm-openai:latest
    pull_policy: always
    container_name: vllm-gemma-4-26b
    expose:
      - 8000
    volumes:
      - ${LOCAL_MODEL_CACHE_DIR:-./data/vllm}:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - HF_HUB_ENABLE_HF_TRANSFER=False
      - NVIDIA_VISIBLE_DEVICES=all
    restart: always
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command:
      - '--model'
      - '${MODEL}'
      - '--max-model-len'
      - '262144'
      - '--tensor-parallel-size'
      - '1'
      - '--gpu-memory-utilization'
      - '0.95'
      - '--max-num-seqs'
      - '32'
      - '--enable-chunked-prefill'

  nginx:
    image: nginx:stable
    container_name: nginx
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./vllm-nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - /etc/certs:/etc/certs:ro
    restart: always

Nginx Configuration (vllm-nginx.conf)

This configuration adds bearer token to authorize access to VLLM from outside.

server {
    listen 80;
    server_name llm.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name llm.example.com;

    ssl_certificate /etc/certs/fullchain.pem;
    ssl_certificate_key /etc/certs/privkey.pem;

    location / {
        if ($http_authorization != "Bearer your_api_key") {
            return 401;
        }
        proxy_pass http://vllm:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 600s;
        proxy_connect_timeout 600s;
        proxy_send_timeout 600s;
        proxy_buffering off;
        proxy_cache off;
    }
}

Usage

Create a directory for your vLLM setup and copy the files above
Update vllm.env with your HuggingFace token
Start the services:

docker compose --env-file vllm.env up -d

Access the API through nginx:

# Direct access to vLLM (no auth)
curl http://localhost:8081/v1/models

# Access through nginx (requires auth)
curl -H "Authorization: Bearer your_api_key" https://ai-llm.example.com/v1/models