vLLM and Dome

1. 01

   Add the connection

   Use the provider id openai_compatible and the server's /v1 URL. If you start vLLM with --api-key, give Dome the same key.

   $ dome models add llama-vllm \
     --provider openai_compatible \
     --endpoint https://vllm.internal.example.com/v1 \
     --model meta-llama/Llama-3.3-70B-Instruct \
     --api-key "$VLLM_API_KEY" \
     --attributes '{"hosting":"self"}' \
     --gateway prod-gateway

2. 02

   Give the Gateway a route

   The hosted gateway can't reach inside your network. Publish vLLM on an address it can call.

3. 03

   Pool it with a hosted provider

   Add Llama 3.3 70B on Together AI as a second connection. Your GPUs answer first, and agents ask for llama-3.3-70b.

   $ dome models pool create llama-3.3-70b \
     --failover-max all --gateway prod-gateway
    
   $ dome models pool member add llama-3.3-70b llama-vllm --priority 0
   $ dome models pool member add llama-3.3-70b llama-together --priority 1

Commands and rules tested against a Dome workspace on October 1, 2026. For anything about vLLM itself, see vLLM's documentation.

forbid (principal, action == Dome::Action::"llm:invoke", resource is Dome::LLMModel)
unless { resource has hosting && resource.hosting == "self" };