vLLM and Dome
Your vLLM servers behind one Gateway, with every call governed. vLLM puts open models on your own GPUs behind an OpenAI-compatible server. Connect it to Dome, let Together AI take the overflow, and a rule decides which agents may ever leave your hardware. The Gateway needs a route to the server.
Dome provides model brokeringand routing for vLLM.
Any model vLLM serves
Point Dome's openai_compatible provider at the server's /v1 address. The served model name is the model id.
Overflow to a hosted provider
Pool vLLM with Together AI serving the same Llama. Your GPUs answer first and Together takes what they can't.
Some agents never leave your GPUs
Tag connections by where they run. A rule keeps an agent on self-hosted models only.
Connect vLLMin three steps.
Connect your server, give the Gateway a route to it, and add a hosted fallback.
Add the connection
Use the provider id openai_compatible and the server's /v1 URL. If you start vLLM with --api-key, give Dome the same key.
shelldome models add llama-vllm \ --provider openai_compatible \ --endpoint https://vllm.internal.example.com/v1 \ --model meta-llama/Llama-3.3-70B-Instruct \ --api-key "$VLLM_API_KEY" \ --attributes '{"hosting":"self"}' \ --gateway prod-gatewayGive the Gateway a route
The hosted gateway can't reach inside your network. Publish vLLM on an address it can call.
Pool it with a hosted provider
Add Llama 3.3 70B on Together AI as a second connection. Your GPUs answer first, and agents ask for llama-3.3-70b.
shelldome models pool create llama-3.3-70b \ --failover-max all --gateway prod-gateway dome models pool member add llama-3.3-70b llama-vllm --priority 0dome models pool member add llama-3.3-70b llama-together --priority 1
This agentstays on your GPUs.
Applied to one agent with --agent, this refuses any connection not tagged hosting: self. The contracts agent never reaches the hosted fallback.
cedar
forbid (principal, action == Dome::Action::"llm:invoke", resource is Dome::LLMModel)unless { resource has hosting && resource.hosting == "self" };Bringing it alltogether.
Connecting vLLM to registered agents, tools, and identity in Dome completes a governed agent application.
Acting for
Agent
Control point
GatewayEvery call decided and audited
RulesGuardsQuotas
Models
Common questionsabout vLLM.
More of whatDome works with.
Model
Claude Fable
Fable 5.1 from Anthropic and Amazon Bedrock in one failover pool, open to one group and capped by quota.
MCP server
GitHub
The GitHub MCP server behind the Tool Gateway, with rules that decide each call on its owner and repository.
Identity
Okta
Okta tokens verified on every agent call, so rules and audit name the person each agent acted for.
Runtime
OpenAI Agents SDK
OpenAI Agents SDK agents with their models on the Model Broker and their MCP tools on the Tool Gateway.
Client
Claude Code
Claude Code on a Dome Gateway with per-developer sign-in, rules on every tool call, and audit by name.
Agent service
TinyFish
TinyFish's web agents behind the Tool Gateway, with rules that decide each run on the site it targets.
Talk with ourFDE team.
Or start free, and register your first agent in minutes. Connect it through a gateway, give it rules, and see every call it makes in one audit trail.
Talk with our FDE team
Our engineers design, build and run governed agents for the part of your business you want to change. You see what they deliver.
LLM providers for AI agents: one governed path to every model
Connect a provider once. Its key stays in Dome, and every agent call to it is authorized, metered and audited.