Agentic Inference and Mixture-of-Experts: Two Levels of Conditional Execution

Abstract

Large-language-model inference increasingly incorporates conditional execution at multiple levels of the system stack. Agentic inference introduces an inter-call control loop in which a model selects tools, observes external results, updates persistent state, and determines whether another model invocation is required. Mixture-of-Experts (MoE) introduces an intra-model routing decision in which each token activates only a subset of feed-forward experts. These mechanisms are architecturally independent, but they create related systems problems: dynamic execution paths, irregular batching, load imbalance, variable latency, and a larger failure surface. This article formalizes both mechanisms, identifies their distinct costs, and examines their interaction in serving systems.

1. Introduction

A conventional account of LLM inference follows a single model invocation: tokenize a prompt, execute Transformer layers, sample a token, update the key-value (KV) cache, and repeat. This account is necessary but no longer sufficient for two important classes of systems.

First, an agentic application places the model inside a stateful control loop. The model may emit a tool invocation instead of a user-visible token sequence, and the tool result may trigger another model call. The unit of execution therefore expands from a token-generation step to a sequence of model and external-system transitions.

Second, an MoE model replaces selected dense feed-forward networks (FFNs) with a bank of experts and a learned router. Each token activates only a small subset of those experts. The model exposes more parameter capacity without activating every parameter for every token, but sparse routing introduces communication and load-balancing costs.

The two mechanisms operate at different levels:

• Agentic orchestration determines which model or external operation executes next.
• MoE routing determines which FFN parameters execute for each token inside a model call.

Their common systems property is conditional execution. Static execution graphs become data dependent, and average computational cost no longer fully predicts tail latency or resource demand.

2. Background and Scope

In a dense Transformer block, attention mixes information across tokens, while the FFN transforms each token independently. During autoregressive decoding, the KV cache avoids recomputing keys and values for earlier tokens. Each newly generated token nevertheless traverses every layer and every dense FFN.

Agentic inference does not alter this internal Transformer computation. It changes the application-level sequence of model invocations. MoE does the opposite: it changes the internal FFN computation but does not define the application control loop. Treating them as orthogonal avoids two common misconceptions: tool use is not a new attention mechanism, and sparse experts do not remove the need for repeated model calls in an agent.

3. Agentic Inference as a Stateful Control Loop

3.1 Execution model

Let $s_t$ denote the complete agent state at step $t$. The state may include the conversation, system instructions, tool schemas, prior tool results, retrieved documents, and persistent memory. The model implements a stochastic policy

where $a_t$ may be a tool call, a request for another model operation, a final response, or a stop decision. Executing the action produces an observation $o_t$, after which the orchestrator applies a transition function

The application terminates when the selected action satisfies an explicit stopping condition or when a resource policy rejects further execution.

request -> state s_t -> model policy -> action a_t
              ^                         |
              |                         v
              +---- state update <- observation <- tool or environment

The model remains autoregressive within each invocation. Agentic behavior emerges from the orchestrator that connects multiple invocations to external state transitions.

3.2 Cost model

For an execution containing $T$ agent steps, request latency can be approximated as

This expression exposes a departure from single-completion serving. Token count remains important, but latency and monetary cost also depend on the number of loop iterations, tool round trips, and prompt growth between iterations. A short model completion followed by a slow external query may dominate an otherwise efficient decode.

Agent state also changes the prompt distribution. Tool outputs, structured arguments, execution logs, plans, and retrieved documents consume context capacity and extend prefill time. Reusing a KV cache within one model invocation does not automatically eliminate repeated prefill across separately constructed agent steps.

3.3 Reliability boundary

The relevant failure is no longer limited to an incorrect next token. The system may choose an incorrect tool, construct invalid arguments, misinterpret an observation, terminate prematurely, or continue indefinitely. A production orchestrator therefore requires mechanisms outside the model:

• schema validation for structured actions;
• explicit tool and permission allowlists;
• bounded iteration, time, and cost budgets;
• idempotency and retry policies for external effects;
• verification before irreversible operations;
• durable state and provenance for recovery and audit.

These mechanisms constrain the transition system rather than attempting to encode every safety property in the model prompt.

4. Mixture-of-Experts as Sparse Intra-Layer Execution

4.1 Router and expert computation

An MoE layer replaces a dense FFN with $E$ independently parameterized expert FFNs. For a token representation $x\in\mathbb{R}^{d}$, a router computes expert logits

After converting these logits to routing scores, the layer selects a set $\mathcal{T}(x)$ containing the top $k$ experts. The output is

where $g_e(x)$ is the gate assigned to expert $e$. Since $k\ll E$, each token activates only a small fraction of the layer’s parameters.

For example, consider four experts with routing probabilities

expert:       e1     e2     e3     e4
probability: 0.05   0.60   0.10   0.25

Top-2 routing selects $e_2$ and $e_4$. If their gates are renormalized over the selected set, the output is

The layer increases total parameter capacity while activating only two experts for this token.

4.2 Serving dataflow

Sparse arithmetic does not imply a sparse systems cost. For a batch $X\in\mathbb{R}^{n\times d}$, an MoE layer performs four stages:

route tokens -> dispatch by expert -> execute expert FFNs -> combine outputs

When experts are distributed across accelerators, dispatch and combine commonly require inter-device communication. Tokens must be grouped by destination expert, transferred to the device holding that expert, processed in expert-specific batches, and returned to their original token order. The resulting communication can offset part of the arithmetic saved by sparse activation.

4.3 Load imbalance and batching

Let $n_e$ denote the number of token assignments to expert $e$ in one layer. Ideal routing produces similar $n_e$ values. In practice, semantically related tokens may concentrate on a small number of experts. Layer latency is then influenced by

because other experts and devices may wait for the most heavily loaded expert before the layer can complete.

Routing skew also fragments matrix operations. Dense FFNs apply one large GEMM to the entire token batch. An MoE layer partitions that batch into smaller expert-specific GEMMs whose shapes vary from step to step. Training-time balancing objectives can reduce persistent skew, but inference still encounters input-dependent imbalance and variable utilization.

Capacity limits bound the number of assignments accepted by an expert. They protect execution from unbounded hotspots, but overflow handling introduces a semantic and performance tradeoff: assignments may be dropped, rerouted, or delayed.

4.4 Interaction with the KV cache

MoE routing and KV caching optimize different sublayers. The KV cache avoids recomputing attention keys and values for prior tokens. It does not cache the output of the current token’s expert FFN. Every generated token must therefore be routed, dispatched, and processed by experts at each MoE layer, even during cached decoding.

At small decode batch sizes, this distinction is consequential. Attention benefits from cached history, while each expert may receive too few current tokens to form an efficient GEMM. Serving systems often aggregate tokens from multiple requests to improve expert utilization, trading additional scheduling complexity against throughput.

5. Cross-Layer Composition

An agentic application can invoke either a dense or an MoE model. When it invokes an MoE model, conditional execution appears at two nested levels:

agent state
  -> select model action or external tool
     -> for each model-generated token
        -> for each MoE layer
           -> select and execute top-k experts

The outer loop determines the number and shape of model requests. The inner router determines token placement within each request. These decisions can compound variability. An agent step with a long retrieved context increases prefill work; its tokens may then route unevenly across experts; a subsequent tool call may add an unpredictable external delay.

This composition has three practical consequences.

First, admission control should account for complete workflows rather than isolated model calls. A request that appears small at admission may initiate many later calls.

Second, observability must preserve both levels of causality. Model-call traces should identify the responsible agent step, while MoE telemetry should expose per-expert token counts, communication time, and overflow behavior.

Third, batching policies may have conflicting objectives. Agent orchestration favors prompt response and low tool-to-model handoff latency, whereas MoE execution benefits from accumulating enough tokens to create balanced expert batches.

6. Design Implications

A serving system that supports both mechanisms should separate concerns explicitly:

• The orchestration layer owns state transitions, permissions, retries, stopping conditions, and workflow budgets.
• The model scheduler batches prefill and decode work across requests.
• The MoE runtime performs token dispatch, expert placement, load balancing, and output combination.
• The telemetry layer correlates workflow steps with model, communication, and expert-level costs.

Useful evaluation metrics consequently span multiple timescales. Agentic systems require task completion rate, number of model calls, tool error rate, and end-to-end tail latency. MoE systems require tokens per second, dispatch volume, per-expert load distribution, communication overhead, and layer tail latency. Reporting only model FLOPs or average token latency hides important conditional costs.

7. Limitations

• The state-machine model abstracts away application-specific planning, memory, and recovery policies.
• Router equations describe common top-$k$ MoE layers but do not cover every routing or expert architecture.
• Sparse parameter activation does not guarantee lower latency; communication, small expert batches, and imbalance may dominate.
• Agent quality and MoE model quality are separate empirical questions from the systems costs analyzed here.
• Hardware topology and expert placement substantially affect MoE behavior and require workload-specific evaluation.

8. Conclusion

Agentic inference and Mixture-of-Experts introduce conditional execution at different boundaries. The agent loop selects operations across model calls and external systems; the MoE router selects parameters within a Transformer layer. Their mechanisms are independent, but both replace a static execution path with data-dependent decisions. Efficient deployment therefore requires more than fast token generation: it requires bounded orchestration, structured failure handling, topology-aware expert dispatch, load-sensitive scheduling, and metrics that expose variability at both levels.