Scaling AI Agents as a Service

Requirements

What the service must do

• Run missions. An agent plans, calls tools and streams its reasoning and answer as it goes.
• Run many agents. An agent can hand a sub-task to a specialised sub-agent, and many users run missions at the same time.
• Remember. A session keeps its history, and a mission can save and reload its state.
• Serve many clients. Any client that speaks HTTP and SSE can use the same agents. Today that is a web app, with early desktop and chat-bot prototypes.
• Ask before acting. Risky tools wait for a human to approve them.

How well it must do it

• It scales horizontally. More users means more replicas, not a bigger machine or a rewrite.
• The first token is fast. Users see the agent start working within a second or two, even if the mission takes a minute.
• Cost has a ceiling. No mission, loop or sub-agent can spend without limit.
• Change is cheap. A new model, prompt or tool is a config change or one new file, not a cross-service rewrite.
• Failures stay small. A broken tool, a dead replica or a closed tab affects one mission, not the service.

Edge cases: what goes wrong at scale

Estimating the load

An agent mission is not a normal request. It holds a stream open for its whole length and spends most of that time waiting: for the model to generate, for a tool to answer. So the useful numbers are not requests per second but open streams and tokens per minute. Little's law gives the first one: open streams equal the arrival rate times how long each one lasts.

Play with the defaults and a pattern shows up. A few replicas hold every open stream, because a waiting stream costs almost no CPU. The number that explodes is tokens per minute, and that one is capped by the model provider, not by your servers. So the design has two jobs: keep the agent tier easy to multiply, and treat every token as the scarce resource.

High-level design

The core decision is to split the system into a control plane and an execution plane. The control plane is a gateway that knows about users, sessions, tiers, history and storage. The execution plane is a pool of agent workers that know only how to run a mission. Clients only ever talk to the gateway.

Each tier scales on a different signal, which is the point of the split: you can add agent workers without touching the database, and change how users log in without redeploying a single agent. The map below plays the three paths that matter most.

Scaling the agent tier

Horizontal scaling only works if any worker can serve any request. That means a worker must not be the only place where anything lives. Echo gets there with three rules.

1. Everything a mission needs arrives with it

The gateway resolves the user, the model, the history and the settings, and sends them in one payload. The worker does not look anything up about the user. The only things it fetches are the tenant's active prompt (through the gateway, cached for 60 seconds) and the session's saved mission state, so a brand-new replica can take the next mission with no warm-up.

{  "user_id": "42",  "session_id": "6f1c2a4e-8b7d-4f3a-9c2e-1d5b7a9e0f12",  "tenant_id": "default",  "message": "Compare the last three releases",  "model": "claude-sonnet-...",  "history": [{ "role": "system", "content": "Summary of earlier turns..." }],  "provider_config": {    "type": "anthropic",    "base_url": "https://api.anthropic.com",    "model": "claude-sonnet-...",    "max_context_tokens": 200000,    "api_key": "..."  },  "features": [],  "strategy_version": "nlah:v1"}

2. State goes out, through the gateway

A mission's working state, and a mission paused for approval, are saved through the gateway's internal memory routes. The gateway keeps each in a Redis list keyed by user and session (a paused mission gets its own list per approval), with the expiry the agent asks for: 10 minutes for mission state, 5 for a paused mission. Long-lived data (sessions, messages, prompt versions) lives in Postgres. When a worker dies, nothing is lost that another worker cannot load. This is the STATE_BACKEND=backend setting the production config uses; the default keeps state in the worker's memory, which is only fine for a single replica.

Some state is allowed to stay inside a worker, as long as losing it is harmless: the handle used to abort a running mission, and the marker that says a mission is already running. The rule is not "no state", it is "no state you would miss".

3. Scale on open missions, not CPU

A worker streaming ten missions and a worker streaming two hundred look the same on a CPU graph, because both are mostly waiting. Autoscale the agent tier on the number of open missions per replica, and keep headroom so one replica can die without a queue forming. The gateway scales the same way, on open connections.

Two guards that keep the pool honest

• One turn per session at a time. The gateway takes a per-session lock before each turn: an in-process lock, plus a Redis lock (SET NX with a 5-minute expiry, renewed while the turn streams) so turns stay ordered across gateway replicas. Two quick messages never get the same turn number or overwrite each other.
• No listener, no mission. When the user presses Stop, the gateway cancels the mission from three sides: the agent's cancel endpoint, its own request context, and closing the connection. The cancel call only reaches one agent replica, so closing the connection is the step that always works. When the client just disconnects, the last two are enough, because the closed connection fires the agent's abort handler. A worker never keeps generating tokens for nobody.

A chat turn, end to end

The client sends almost nothing: a message, a session id (left out for a new chat), and optionally a model or mode that overrides the saved defaults. Features, skills and settings come only from the user's saved preferences on the server, so a client cannot ask for a feature it is not allowed to use. Everything else is the gateway's job, done before the mission starts.

1. Who and what tier. The user id comes from the signed JWT, and the tier is looked up by that id and cached in Redis for 60 seconds.
2. Whose session. The session is loaded and its owner compared with the caller.
3. How much history. History is packed newest first into a 50,000-token budget, from at most the last 200 messages. Once a session reaches 100,000 tokens, the gateway pauses the turn and asks the agent to summarise the oldest turns, never the latest 10. Those turns are deleted, and the summary is sent ahead of the history from then on. A background worker does the same for long sessions that have gone idle.
4. Which model. The model id becomes a full provider_config from the user's saved provider settings.
5. One turn at a time. A per-session lock makes turns wait their turn. Inside one Postgres transaction the gateway locks the session row, takes the next turn number, and writes the user message plus an empty assistant placeholder.

Then the worker streams packets back, and the gateway relays them line by line while saving the answer text to the placeholder every 2 seconds. A crash, a cancel or a closed tab leaves an interrupted turn with everything written so far, and a refresh rebuilds the chat from Postgres, minus any turns already folded into the summary.

Multi-agent: delegation

Some missions are better split. A research question might need one agent to search, one to read, and one to write the answer. Echo handles this with a single tool, delegate_task. The parent agent calls it like any other tool, with an agentName, an instruction and a systemPrompt for the sub-agent.

The sub-agent is a new harness in the same process, running its own loop with its own prompt, on the parent's provider and tools. Three choices make this scale well:

• A clean context by default. The child starts with only its instruction, unless the parent sets fork_context to share its history. The child's whole working conversation never enters the parent's context; only its final text comes back, as one tool result. That keeps the parent's prompt short and its cache warm.
• One stream for everything. The child's packets are re-tagged with the parent's mission id and flow out on the parent's stream, so the user watches the sub-agent work, and the gateway needs no new connection or protocol.
• A depth limit. Each child runs one level deeper. At depth 3 the delegate_task tool is removed from the agent's toolbox, so delegation can never recurse without end.

Keeping it fast

Users judge an agent by how quickly it starts and how steadily it moves. Most of the speed comes from not doing work twice.

Prompt caching only works if the start of the prompt never moves. That gives a simple rule for everyone who touches the agent:

1. tool definitions   fixed for the whole mission     → cached2. system prompt      fixed for the whole mission     → cached3. year anchor        one line, fixed                 → after this, nothing is cached yet4. history            grows only at the end5. latest turn        new every call                  → never cachedRules: append, never insert. Compact the middle, keep the start.

The anchor is a one-line system message with the current year, so the model does not guess the date. Echo breaks its own rule in one place today: a per-call timestamp sits in the system text, which stops the cache after the tools. Moving it to the end of the prompt is the next fix.

Keeping it under control

An agent decides its own next step, so it can also decide to do something expensive forever. Every mission runs inside a harness that watches the loop and stops it before it hurts. These are Echo's defaults; users can tune most of them in their settings:

The guards also show up on the stream. A state_change packet moves the agent between states such as running, looping, degraded, stalled and aborted, and loop and budget hits also send a system_notice, so the client can show what the agent is doing instead of a spinner that might be lying.

Keeping it maintainable

Maintainability in an agent system mostly means one thing: the parts that change often (models, prompts, tools, strategies) must be cheap to change without touching the parts that do not.

Registries instead of if-statements

• Providers. A small factory maps a provider type to a class: OpenAI, Anthropic, LM Studio and OpenCode Go. A new model API is one class, one registry entry, and its name in the provider lists on the agent, the gateway and the web settings form. The harness and the chat flow do not change.
• Tools. Built-in tools are registered by name, picked up from a definitions folder at boot. MCP servers are discovered at runtime, with their JSON Schemas converted to Zod, and plain REST APIs are described by a name, an endpoint, a method, headers and an input schema.
• Strategies, versioned. The mission loop itself is a strategy with a version, such as nlah:v1 or standard:v1. Each session is pinned to the version it started with, so rolling out v2 never changes a conversation halfway through.

Behaviour as data

Prompts are versioned in a studio on the gateway. Prompt engineers write versions; admins and product managers promote or roll them back. Agents ask the gateway for the active version and cache it for 60 seconds, and a promotion clears that cache, so behaviour changes with no redeploy. Feature flags and per-user harness settings work the same way: they are rows, not code.

Ports and adapters

The agent's code is split into adapter/inbound (HTTP routes), adapter/outbound (calls to the gateway), core (harness, strategies, tools) and infrastructure (providers, MCP and REST transports). Every call to the gateway sits in adapter/outbound/backend (one adapter for state, one for prompts), and the state store is picked by a small factory, so swapping the backend is a new adapter rather than a change across the harness. The split is not perfect yet (about ten core files still import adapters and transports directly), and keeping that direction one-way is the rule worth defending in review.

Contracts at every boundary

Requests are validated with Zod on the agent and typed structs on the gateway, and both services publish OpenAPI specs (served with Scalar at /docs and /api/docs). As long as a service honours its contract, it can be rewritten in another language without the other side noticing.

Security without slowing down

Splitting the system creates trust boundaries, and each one gets its own credential so a leak opens one door, not all of them:

All checks happen inside the services: a signature check, a cached tier lookup, and for API keys one database read. Nothing calls out to an auth service in the hot path. The internal routes are protected once, at the group level, so a route added later cannot be forgotten:

internalGroup := api.Group(routes.V1InternalGroup, middleware.InternalAuthRequired(cfg))memoryGroup := internalGroup.Group("/memory")memoryGroup.Post("/episodic/store", memoryHandler.HandleStoreEpisodic)memoryGroup.Post("/episodic/recall", memoryHandler.HandleGetEpisodic)

Asking a human first

Approval is where a stateless design pays off. When a risky tool needs a yes, the agent saves everything it needs to continue (messages, the pending call, tools, provider settings without the key, cost so far) through the gateway for five minutes, sends an approval packet, and ends the stream. Nothing waits in memory.

The approval can arrive minutes later at any replica. That replica loads the snapshot, runs the approved tool or tells the model it was denied, builds a fresh harness, and streams the rest. If that replica saw the mission cancelled while it waited, it answers 409 and the approval is used up, so a late click cannot bring a stopped mission back to life. Sharing that cancel flag across replicas, for example in Redis, closes the gap completely.

What breaks first

Push the estimator's users slider to the right and walk down this list. It is roughly the order in which a design like this runs out of room, with the fix for each step.

Seeing what agents do

At scale you cannot read every transcript, so the system has to tell you. Echo's agent records each mission as a Langfuse trace: every turn, tool call and model call, with its prompt, tokens and cost. It exports these as OpenTelemetry spans to Langfuse, and the gateway passes the caller's W3C traceparent header through to the agent. The next step is for the gateway to record and export its own spans, so one trace shows where a slow answer actually spent its time.

What we learned

Most of the work in serving agents has nothing to do with the agent. It is deciding where state lives, which tier scales on what, and where to put the ceilings. Once those are fixed, the agent itself becomes the easiest part of the system to change.