agent-platform
One OpenAI-compatible endpoint in front of every model, hosted or running on the desk.
Problem
An agent that is useful is an agent wired to several providers, and each one arrives with its own SDK, its own auth and its own idea of what a streamed response looks like. The local runtimes are worse: the code that talks to Ollama on a laptop is not the code that talks to it from inside a container, so the thing that works while developing is not the thing that ships.
Approach
A FastAPI service that serves the orchestration API and an OpenAI-compatible /v1 surface from the same process, so any client already written against OpenAI points at it and needs no branch per provider. Keys are the caller’s, not the server’s — the request carries them through to OpenAI, Anthropic, Gemini, Groq or Mistral — and the local runtimes are reachable from a container because loopback addresses are rewritten to the host at the boundary rather than in every call site. Tools are denied by default and opened by an explicit policy, not by a flag someone forgot. One image runs the backend, the React flow UI, or both.
Outcome
- Seven upstreams — five hosted providers and two local runtimes — behind a single endpoint, with no provider-specific path in the client.
- A prompt can be answered by a model on the same machine: no key issued, nothing leaving the host, the same code as the hosted route.
- Tool execution is off until a policy turns it on, so the dangerous default is the safe one.
- One Docker image, three run modes, SQLite on a named volume — it comes back up as it went down.
The decision worth defending is BYOK. A gateway that stores provider keys is a gateway that has to be trusted, audited and rotated, and it becomes the single most valuable thing on the host. Forwarding the caller’s key costs a header on every request and removes the entire category: there is nothing on the box to steal.
The loopback rewrite is the small detail that comes from having actually run it. Ollama advertises itself on 127.0.0.1, which inside a container means the container. Rewriting it to the host once, at the boundary, is the difference between a local model being a supported route and being a thing that only works when the process happens to be outside Docker.
Tools default to denied for the same reason the payment path in a finance service defaults to no-op: the failure mode of a permissive default is not a bug report, it is an incident.
- 5 hosted + 2 local
- Upstreams behind one API
- 0
- Keys held by the server