Single Agent or Multi-Agent: Four Tests That Decide, and the Router Pattern That Usually Wins

You do not need a multi-agent system to write an email, draft a SQL query or parse a PDF. One well-structured model call does each of those. Yet many agent codebases route a single intent through four or five sub-agents that talk to each other in prose. The result is slow, expensive and hard to debug.

The opposite mistake costs just as much. A branching, multi-tool, approval-gated workflow forced into one prompt works in the demo. It fails as volume grows.

How many agents a system needs is the first architecture decision in an agent project. It sets latency, cost, the blast radius of a mistake, and whether you can write tests at all. The recommendation below is the default we start from: one router, a small number of specialists, and handovers passed as data rather than conversation. Four tests say when to depart from it. Which framework to build on is a separate question, and the framework comparison linked below answers it.

The two ways to get it wrong

The monolith prompt. One model call carries every tool, every persona, every schema and every edge case in a single system prompt. It is cheap and fast while it works. As tools and instructions pile up, the model starts picking the wrong tool. It ignores negative constraints such as "do not quote prices" and drifts from the output schema.

Two published findings explain the decay. Liu and colleagues at Stanford showed it in "Lost in the Middle". Models use information at the start and end of a long input far better than information buried in the middle. Tool selection has the same shape. OpenAI's function-calling guide recommends fewer than 20 functions available at the start of a turn, "for higher accuracy". Anthropic's tool-search documentation states that selection accuracy degrades once you exceed 30 to 50 available tools. The exact threshold depends on your model and your prompt. The direction does not: more tools and more instructions in one prompt means less reliable selection.

[User input] -> [One agent: 15 tools, 4 personas, 3 schemas] -> [Output]
                                  |
                                  v
                  attention drift, wrong tool, schema drift

The conversational swarm. Some frameworks encourage an agent per noun in your business: Researcher, Writer, Editor, Publisher. Each hands off to the next in natural language. In a demo it looks magical. In production it introduces nondeterminism. Agent A can ask Agent B for clarification in dozens of ways. Strict unit tests are hard to write, and every hop adds a full model round trip. The three framework families take different positions on this, and how the three framework families differ is where that choice gets made.

Both mistakes share a root cause: the architecture was chosen before the workflow was written down.

The four tests

The question is not whether the task is complex. It is whether the workflow needs isolated state. If different stages need different tools, prompts or credentials, split. If they share one context, keep one agent or a hard-coded router. Four tests decide.

1. Tool catalogue size. When one agent must choose from a long list, selection accuracy falls. Database reads, CRM writes, payment calls, email and calendar actions in one catalogue is the usual shape. Specialists that each see a handful of tools make fewer wrong calls. The guidance above gives the range. Measure the break point on your own workflow before you decide.

2. Conflicting personas or instructions. "Be a strict legal auditor" and "be a warm, empathetic copywriter" in the same prompt gives mediocre output on both. Conflicting personas belong in separate agents.

3. Credential and schema isolation. The agent that checks a refund policy should not hold the payment gateway's write token. The agent that reads a public knowledge base should not reach the HR database. Splitting here is a security requirement, not an engineering preference. The data minimisation principle in Article 5(1)(c) of the GDPR applies to each AI component as much as to any other system. The Gulf data protection laws take the same line, and the PDPL checklist covers them. If a component does not need access, it should not have it.

4. Execution path shape. A linear path (A, then B, then C) wants a single agent with structured outputs. A plain script with one model step also works. Loops, conditional branches and replanning want a stateful graph such as LangGraph.

What each shape costs

Work with the shape of the costs rather than someone else's benchmark. For a workflow handling N requests a month:

• ▸Single agent. One model round trip per request. Latency is one time to first token plus one generation. Cost is one call's input and output tokens. Failures are hard to localise, because everything happens in one prompt.
• ▸Router plus specialists. One classification call plus one specialist call. Latency and cost roughly double. Failures are easy to localise: either the router chose wrong or the specialist did. This is the default we start from.
• ▸Stateful multi-agent graph. Several calls per request, often with retries and loops. Latency and cost grow with the number of nodes traversed. In return you get isolation, testability, and approval gates at specific nodes.

Write down your target end-to-end latency and your tolerance for errors. Then measure the three shapes on your own data. If users cannot wait through three sequential model calls, the interactive path cannot afford a deep graph. Push the heavy work into an asynchronous job and return a receipt.

A router also changes what a failure costs. In a monolith, a wrong answer could have come from any of fifteen tools or four personas. In a router, the trace shows one classification and one specialist call. Either the router chose wrong, which is a labelled training example, or the specialist did, which is a narrow prompt to fix. That is why the router pattern tends to win on maintenance even when it loses on raw latency.

Deterministic handovers in Python

The most common failure in multi-agent code is the conversational handover. Agent A writes a free-form message, Agent B parses it, something drifts, and the run goes sideways. The fix is to make handovers data, not conversation. Every transition is a validated schema, and the orchestrator decides who runs next. The model never does.

Here is the pattern, with Pydantic for state validation and LiteLLM so each node can use a different model.

import os
from typing import Literal, Optional
from pydantic import BaseModel, Field
from litellm import completion

# Model choice lives in the environment, not the code. Use a small, fast model
# for routing (a small GPT or Claude model) and a larger model from the same
# families where the node has to reason. Pin exact IDs per environment.
ROUTER_MODEL = os.environ["ROUTER_MODEL"]
REASONING_MODEL = os.environ["REASONING_MODEL"]

# Global state schema. Every node reads and mutates this.
class AgentState(BaseModel):
    customer_id: str
    original_query: str
    extracted_intent: Optional[str] = None
    refund_amount: float = 0.0
    verification_status: Literal["pending", "approved", "rejected"] = "pending"
    next_step: Literal["classifier", "billing_agent", "support_agent", "complete"] = "classifier"
    execution_log: list[str] = Field(default_factory=list)

# Structured outputs, so handovers are data rather than prose.
class RoutingDecision(BaseModel):
    intent: str
    target_agent: Literal["billing_agent", "support_agent"]
    reason: str

class BillingAction(BaseModel):
    refund_approved: bool
    calculated_amount: float
    justification: str

def log_transition(state: AgentState, message: str) -> None:
    state.execution_log.append(message)
    print(f"[{state.next_step.upper()}] {message}")

# Node 1: classify intent and pick the next specialist.
def run_classifier(state: AgentState) -> AgentState:
    log_transition(state, "Analyzing customer query for routing.")
    prompt = (
        f"Analyze this customer query: '{state.original_query}'. "
        "Decide if it is a billing issue or a general support issue."
    )
    response = completion(
        model=ROUTER_MODEL,
        messages=[{"role": "user", "content": prompt}],
        response_format=RoutingDecision,
    )
    decision = RoutingDecision.model_validate_json(response.choices[0].message.content)
    state.extracted_intent = decision.intent
    state.next_step = decision.target_agent
    log_transition(state, f"Routed to {decision.target_agent}. Reason: {decision.reason}")
    return state

# Node 2: billing specialist. Only this node holds billing credentials.
def run_billing_agent(state: AgentState) -> AgentState:
    log_transition(state, "Processing billing and refund eligibility.")
    prompt = (
        f"Customer query: {state.original_query}. "
        f"Calculate eligible refund. Customer ID: {state.customer_id}."
    )
    response = completion(
        model=REASONING_MODEL,
        messages=[{"role": "user", "content": prompt}],
        response_format=BillingAction,
    )
    action = BillingAction.model_validate_json(response.choices[0].message.content)
    state.refund_amount = action.calculated_amount
    state.verification_status = "approved" if action.refund_approved else "rejected"
    state.next_step = "support_agent"
    log_transition(
        state,
        f"Refund {state.verification_status}: {state.refund_amount}. Handing to support."
    )
    return state

# Node 3: drafts the customer-facing message. No billing credentials in this context.
def run_support_agent(state: AgentState) -> AgentState:
    log_transition(state, "Drafting final response to customer.")
    prompt = (
        f"Draft a response. Intent: {state.extracted_intent}. "
        f"Refund status: {state.verification_status}. Amount: {state.refund_amount}. "
        f"Original query: {state.original_query}"
    )
    response = completion(
        model=REASONING_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    log_transition(state, f"Drafted response: {response.choices[0].message.content[:60]}...")
    state.next_step = "complete"
    return state

# Orchestrator. Code, not a model, decides who runs next.
def execute_workflow(initial_state: AgentState) -> AgentState:
    state = initial_state
    max_steps = 5  # hard iteration cap, a little above the longest expected path
    step_count = 0
    while state.next_step != "complete" and step_count < max_steps:
        step_count += 1
        if state.next_step == "classifier":
            state = run_classifier(state)
        elif state.next_step == "billing_agent":
            state = run_billing_agent(state)
        elif state.next_step == "support_agent":
            state = run_support_agent(state)
        else:
            raise ValueError(f"Unknown state: {state.next_step}")
    if step_count >= max_steps and state.next_step != "complete":
        log_transition(state, "Workflow halted: max iterations reached. Escalating to a person.")
    return state

if __name__ == "__main__":
    test_state = AgentState(
        customer_id="cust_9921",
        original_query=(
            "I was charged twice for my subscription yesterday. "
            "Please refund the duplicate charge."
        ),
    )
    final_state = execute_workflow(test_state)

Three things to notice. First, every handover is a Pydantic model, so a malformed response fails loudly and early instead of corrupting downstream state. Second, the billing node is the only one that holds billing credentials. The support agent cannot touch them, even if a prompt injection tries. Third, the orchestrator owns the control flow. The models make decisions about data; they do not decide who runs next.

Swap the model slot per node. A fast, cheap model classifies; a stronger model runs only where the node has to reason. Semantic routing can take this further, with semantic routing to the cheapest capable model chosen from the request content, once per session.

Observability and loop guards

Once you split into nodes, debugging becomes a distributed-systems problem. You cannot grep print statements across five processes. Every node must emit structured traces tied to a single trace_id, so a failed run can be replayed end to end.

[Trace: request_4821]
├── Node: Classifier    (time to first token, generation, cost per call)
├── Node: Billing       (time to first token, generation, cost per call)
└── Node: Support       (time to first token, generation, cost per call)

Tracing tools such as Langfuse are built for this. The traces are also the raw material for step-level evaluation, which grades each handover rather than the final answer.

The second non-negotiable is a hard iteration cap at the orchestrator level. If two nodes can call each other, there is a path where they ping-pong on an edge case and burn API credits unattended. A max_iterations guard, set a little above your longest expected path, plus a graceful halt that pages a person is the cheapest insurance available. A hard iteration cap and a clean handoff are two of the seven controls in the agents hub post.

Frequently asked questions

How do we decide between a router pattern and a full stateful graph? Draw the workflow. If the path is a tree (classify, then one specialist runs, then done), a router is enough, and it will be faster and cheaper. If the path has loops, retries that depend on tool outputs, or approval nodes that can send work back upstream, you need a stateful graph. Do not pay for graph complexity you will not use.

How do we prevent infinite loops between agents? Two guards. A hard iteration counter in the orchestrator, not in any single agent, that halts after a fixed number of transitions. And strict schema validation on every handover. An agent that receives malformed state routes to a person or an error node, never back to the sender. Clarification loops are where runaway bills come from.

Can we mix open-source and proprietary models across nodes? Yes, and for many production workloads you should. Classification and routing suit smaller open-weight models on your own GPUs or a cheap API tier. Reserve the stronger proprietary models for the one or two nodes that need multi-step reasoning. A gateway such as LiteLLM makes the swap a configuration change rather than a rewrite. For the local inference side, see the on-prem LLM speed guide.

The practical rule

Start with a single agent. This week, run your real workflow through it and write down where it breaks: tool-selection errors, schema drift, a credential boundary you cannot defend. Split exactly those boundaries, and only those. Keep the control flow in your orchestrator, not in model-to-model conversation. Put tracing and a loop cap in on day one, not after the first runaway bill.

The default we start from is one router plus two or three specialists. Not seven. Not twelve. Enough to isolate the credentials and the personas that conflict, and nothing more.