Self-hosted models

Connect your own OpenAI-compatible inference server, such as vLLM, and your people can use its models in Harriet Desktop and cloud workspaces, and optionally in Claude Desktop. Each device talks to your server directly; Harriet only delivers the settings.

Before you start

Plan: self-hosted models are enabled per organization. Contact support to switch them on.
Reachability: Harriet Desktop and Claude Desktop need to reach the server from each laptop (your office network or VPN is fine). Cloud workspaces run in Harriet's cloud, so they can use the server only if it is reachable from the internet over HTTPS.
The key: the API key you enter is delivered to each device and stored in its configuration. Use a key that only unlocks your inference server, and rotate it here if it leaks — devices pick up the new key within about 5 minutes.

Server settings (vLLM)

Harriet always gives the model tools to use, so tool calling must be switched on. Start vLLM with:

--api-key <key> (or the VLLM_API_KEY environment variable)
--enable-auto-tool-choice --tool-call-parser <parser>, choosing the parser for your model family
--max-model-len 65536 or more: Harriet's instructions and tool descriptions take a large share of the context window

Larger models handle long, multi-step tool use far better than small ones.

Connect the server

Open Settings → Model access and find the Self-hosted models card.
Enter the Base URL: your server's OpenAI API root, ending in /v1 (for example https://llm.example.com/v1).
Paste the API key.
Choose Add model for each model and enter its Served name exactly as your server names it (the id in its /v1/models list). Set the Context window to the value your server allows, and turn on Images for models that accept images.
Choose Save. Devices pick up the change within about 5 minutes.

In Harriet Desktop the models appear under Self-hosted in the model picker.

Claude Desktop (third-party mode)

Turn on Use for Claude Desktop (third-party mode) to point Claude Desktop at your server. Claude Desktop calls your server's Anthropic Messages endpoint (/v1/messages), which vLLM serves from version 0.11.1.

⚠️

Claude Desktop can use only one gateway. With this switch on, Claude Desktop offers only your self-hosted models, not the models Harriet provides. Features that depend on Claude models, such as extended thinking and prompt caching, may not work with other models.

What Harriet does not do for these models

Requests go straight from each device to your server, so Harriet does not record their usage, apply budgets, or restrict them by group. Use the Offered switch on each model to control what people can pick.