Multi-model AI architectures: routing work across LLMs

By Marco Lopes · Published 2025-01-10 · AI & Machine Learning

At Kordspace, we work across software, AI, Web3, hardware and design as a remote collaborative studio. That breadth gives us a practical reason to question the idea of choosing one large language model for everything. Extracting fields from a document, interpreting a circuit-board photograph and reviewing application code are different jobs. We should not assume they need the same engine.

As of January 2025, we have credible options across hosted model families, managed cloud platforms and open-weight deployments. The opportunity is not simply to connect more providers. It is to assign work deliberately while keeping quality, latency, privacy and operating cost understandable.

We call this a multi-model architecture. Here is how we would build one without creating a maze of prompts and fragile dependencies.

Start with the work, not the model catalogue

We begin by separating a product’s AI features into tasks with observable acceptance criteria. “Help our users” is too broad to route. “Extract invoice fields into a validated schema” is specific enough to test.

For each task, we document:

• Accepted input types and maximum input size.
• Whether the answer requires private or current information.
• The consequences of an incorrect result.
• Required response format and acceptable latency.
• Whether the system may recommend actions or actually execute them.

We also ask whether an LLM is necessary. Deterministic parsing, database queries and ordinary search can be cheaper and more reliable for well-defined operations.

Our first architectural win is often removing an unnecessary generation step. Multi-model design should mean appropriate computation, not maximum model involvement.

Separate routing, orchestration and execution

We keep three responsibilities distinct.

Routing selects an eligible model or processing path. Orchestration controls the sequence of steps, including retrieval and validation. Execution calls models and tools through well-defined adapters.

A useful request path is:

Application → policy checks → task router → model adapter → validation → response

Retrieval and tool execution sit alongside that path rather than disappearing inside an unrestricted prompt. Our application owns authorization, state transitions and side effects.

We maintain a small model registry containing supported modalities, approved tasks, region constraints, context limits and operational status. The registry also records the exact deployed model identifier and prompt version.

This separation lets us change a provider without rewriting business logic. More importantly, it prevents a model’s suggestion from silently becoming permission to run a tool, access another customer’s records or submit a transaction.

Give models clear jobs

We prefer a small initial portfolio with meaningful differences rather than several nearly interchangeable options.

A fast model can handle classification, straightforward extraction and short transformations. A more capable model can address difficult synthesis, ambiguous instructions or substantial code changes. A multimodal path can handle images or documents whose layout matters.

By this date, families such as Gemini 1.5, GPT-4o and Claude 3.5 give us candidates to evaluate for these roles. We do not infer suitability from branding alone. We test the exact model versions, endpoints and configurations available to the project.

Open-weight models can also make sense when deployment control or a narrow, repeated workload justifies the operational burden. We include serving capacity, updates, monitoring and security maintenance in that decision.

We avoid treating a larger context window as a substitute for retrieval design. More input can mean more noise, not more understanding.

Build a policy router before a learned router

We start with explicit rules because we can inspect, test and explain them.

A basic policy might route a short, low-risk classification request to our fast model; a photograph-based inspection request to an approved vision-capable endpoint; and a complex technical review to our stronger model. Requests involving restricted data stay within approved deployment boundaries regardless of another model’s apparent capability.

The order matters. We first eliminate ineligible destinations based on security, modality and context requirements. Only then do we optimize among eligible choices for quality, latency and cost.

We reserve escalation for concrete signals: invalid structured output, missing required evidence, failed checks or a task explicitly marked high consequence. A model declaring itself “90% confident” is not, by itself, a reliable routing signal.

If we later train a routing classifier, we evaluate it against these rules. We also include the classifier’s latency, expense and errors in our measurements. A sophisticated router that makes simple requests slower is not automatically an improvement.

Choose the right composition pattern

We distinguish between three patterns that are often grouped together.

Direct routing sends each request to one suitable model. We use this as the default because it minimizes coordination and duplicated work.

Cascading starts with a cheaper path and escalates when a measurable condition fails. We use it when easy cases are common and validation is strong enough to identify unsuccessful attempts.

Parallel review sends selected work to multiple models and compares or adjudicates the outputs. We reserve it for tasks where the additional cost and latency are justified.

Agreement is not proof. Two models may repeat the same misconception, especially when they share the same weak evidence. For factual questions, we prefer checking against authoritative sources. For generated code, we prefer tests, static analysis and review. For consequential actions, we preserve an appropriate human approval boundary.

Standardize contracts without hiding capabilities

We define a common request envelope with task type, tenant identity, policy labels, deadline and references to authorized context. We define a common response envelope with status, result, model identifier, usage information and validation outcomes.

For extraction, our result might include document_type, fields and evidence_spans. We validate the schema in application code and separately check business constraints. Valid JSON can still contain the wrong invoice total.

We also normalize operational outcomes: refusal, timeout, truncated output and unsupported input are different conditions, not generic empty responses.

However, we avoid forcing every provider into a lowest-common-denominator interface. Adapters can expose supported capabilities explicitly. A route requiring image input or tool calling should fail eligibility checks when an adapter cannot satisfy it, rather than silently changing the task.

Keep context portable and permissions narrow

We store durable workflow state outside provider-specific conversation objects. That makes routing decisions and provider changes less dependent on one service’s session format.

When we transfer work between models, we pass a bounded handoff: the task, relevant evidence, completed steps and unresolved questions. We do not blindly forward an entire transcript. Summaries can lose qualifications, so important claims retain links or identifiers pointing to their source material.

We treat retrieved pages, uploaded files and tool results as untrusted content. Instructions embedded in a document cannot grant access or override routing policy.

For our Web3 and hardware work, this distinction is particularly important. Generating a transaction proposal is not signing it; suggesting a device command is not authorizing execution. Our backend enforces permissions and validates parameters independently of whichever model produced them.

Make fallbacks policy-aware

A fallback is useful only if it preserves the request’s requirements.

We distinguish transient failures, quota exhaustion, malformed output and safety refusals. We can retry some transient errors with bounded backoff. We can sometimes repair or regenerate malformed output. We do not respond to a safety refusal by automatically shopping the same request across providers until one complies.

Each request gets a total deadline and retry budget. Otherwise, a nominally fast route can become a long chain of expensive attempts.

We use idempotency controls around external actions so retries cannot duplicate payments, messages or provisioning operations. Model generation and tool execution have separate retry policies.

When no compliant route remains, we return a clear limitation or queue work where the product permits it. We would rather expose a temporary failure than silently move sensitive data to an unapproved endpoint.

Deploy a thin control plane on Google Cloud

Google Cloud is our preferred starting point, while we also work with AWS and Azure when project requirements favor them.

On Google Cloud, we can run the routing and adapter service on Cloud Run, use Vertex AI for supported managed models, and keep provider credentials in Secret Manager. Cloud Tasks or Pub/Sub can support asynchronous processing according to the workflow’s delivery and scheduling needs.

We apply least-privilege service identities and verify model availability, regional processing behavior and contractual data handling for each selected endpoint. A shared platform name does not guarantee identical terms across every model.

We instrument traces with route decisions, model versions, latency, usage and validation results. We redact sensitive content and set retention deliberately rather than logging every prompt by default.

On AWS or Azure, we preserve the same logical boundaries even when the managed services change.

Evaluate the router as part of the product

We build an evaluation set from representative, appropriately sanitized tasks. It includes ordinary requests, ambiguous cases, adversarial documents, oversized inputs and provider failures.

We measure the complete path, not just the final generation. Our scorecard includes task success, policy compliance, latency distribution and total cost per successful task. For cascades, that total includes failed first attempts and validation calls.

We compare the routed system with a simple single-model baseline. If routing adds complexity without a meaningful benefit on our workload, we simplify it.

Model-based judging can help triage results, but we calibrate it against human review and deterministic checks. We pay particular attention to false acceptance: outputs that look polished but should have failed.

Before changing a route, prompt or model version, we rerun the relevant tests. We then use a limited rollout with a clear rollback path.

Keep our launch checklist short

Before enabling production traffic, we confirm:

• Every task has an acceptance test and an accountable owner.
• Routing rules enforce privacy and capability constraints first.
• Model identifiers, prompts and policies are versioned.
• Outputs receive schema and task-specific validation.
• Retries, deadlines and spending limits are bounded.
• Tool permissions and side effects remain application-controlled.
• Logs support debugging without unnecessary sensitive content.
• Fallbacks, human review and rollback paths have been exercised.

We launch with a small number of routes and expand only when evaluation results justify another branch. Every added model creates another dependency to maintain.

Build for deliberate choice

The strongest multi-model architecture is not the one that calls the most models. It is the one that makes each choice understandable: why this task went to this model, what evidence supported the result, and what happened when the path failed. We aim for systems our distributed team can operate together, not demonstrations that depend on invisible prompt behavior.

At Kordspace, we bring software engineering, AI, design and infrastructure decisions into the same conversation. We can help define a focused first workflow and the evaluation gates around it. To explore an implementation with us, start a project, or read our related services pages to see how we approach the work.