TEI deployment example¶

TEI (Text Embeddings Inference) is a HuggingFace server for text embeddings models. Easy8 AI uses it to serve BAAI/bge-m3, the supported self-hosted embeddings model. For GPU requirements, see Embeddings hardware recommendations.

Docker Compose setup¶

This example runs TEI with an nginx reverse proxy that checks an API key. It starts nginx on ports 80 and 443; if vLLM runs on the same host, use one nginx with both server blocks instead of two.

Directory structure¶

tei/
├── tei.env
├── tei-nginx.conf
└── docker-compose.yaml

Environment configuration (tei.env)¶

HUGGING_FACE_HUB_TOKEN=your_huggingface_token
TEI_MODEL_CACHE_DIR=/data/llm/tei-bge-m3

Docker Compose (docker-compose.yaml)¶

services:
  tei:
    image: ghcr.io/huggingface/text-embeddings-inference:89-latest # tag 89 targets compute capability 8.9 (for example L40S); use the tag for your GPU
    pull_policy: always
    container_name: tei-bge-m3
    expose:
      - 80
    volumes:
      - ${TEI_MODEL_CACHE_DIR:-./data/tei}:/data
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}
    restart: always
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command:
      - '--model-id'
      - 'BAAI/bge-m3'
      - '--hf-api-token'
      - '${HUGGING_FACE_HUB_TOKEN}'

  nginx:
    image: nginx:stable
    container_name: nginx
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./tei-nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - /etc/certs:/etc/certs:ro
    restart: always

Nginx configuration (tei-nginx.conf)¶

server {
    listen 80;
    server_name tei.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name tei.example.com;

    ssl_certificate /etc/certs/fullchain.pem;
    ssl_certificate_key /etc/certs/privkey.pem;

    location / {
        if ($http_authorization != "Bearer your_api_key") {
            return 401;
        }
        proxy_pass http://tei:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 600s;
        proxy_connect_timeout 600s;
        proxy_send_timeout 600s;
        proxy_buffering off;
        proxy_cache off;
    }
}

Usage¶

Create a directory for TEI and copy the files above
Update tei.env with your HuggingFace token and replace your_api_key in tei-nginx.conf with your API key
Start the services:

docker compose --env-file tei.env up -d

Test the embeddings API through nginx:

curl -X POST https://tei.example.com/embed \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"inputs": "Your text here"}'

Connect Chain¶

Set the embeddings URL and the key in the Easy8 .env file:

EMBEDDINGS_URL=https://tei.example.com
TEI_API_KEY=your_api_key

Replace tei.example.com with the host name of your TEI deployment.