MentorAgent

← All articles

Fewer tokens, lower cost: how MentorAgent trims every request

Where the tokens of an AI agent go, what MentorAgent does about each part, and the settings to start from for a back office, long chats, many users and a local model.

MentorAgent team · 13 min read· Leggi in italiano

MentorAgent cover: the title Fewer tokens, lower cost; a request to the model is shown as a stack of blocks (instructions, tools, history, context, question); MentorAgent keeps only the blocks the turn needs, and sends the turn to a small or a strong model.
In this article

A chatbot sends a question and gets an answer. An agent that acts sends much more. Every call carries its instructions, the definition of every tool, the conversation so far and the context of the page. Then it calls a tool, and the whole request goes out again.

MentorAgent trims each of these parts, with options you can read, set and measure. This article shows where the tokens go, what each lever does, which settings to start from, and how to check the result.

In an agent built with MentorAgent, a request to the model is made of:

  • Instructions: the system prompt, with the rules, the pages and the user.
  • Tool definitions: the name, the description and the parameters of every action the model may call.
  • History: the conversation so far, with the tool calls and their results.
  • Context: the page that is open, the date, the user's memory, the documents found by RAG.
  • The question and the answer.

A turn can also make more calls: one per tool round trip, the small calls of MentorAgent's own decisions, and those of the specialists and teams you add.

Two parts grow on their own: tool definitions grow with your application, history with the conversation. Most levers below work on these two, on the choice of model, or on the extra calls.

Some of the work needs no setting.

  • The instructions list no actions. The coordinator's prompt carries the number of direct actions, not the actions, and the pages and specialists with a short description. The text that never changes comes first.
  • An action the user may not run is not sent. The model never sees the definition of an action the user's roles do not allow. It is only told that some actions are not available.
  • One embedding per message. Tool filtering, semantic routing and memory share it. The tool descriptions are embedded once per process.
  • Skills load on demand. With skills on (EnableSkills), the prompt carries a name and a description for each skill, and the model loads the full text with load_skill only when it needs it.

Everything else is an option, and the levers that save tokens are off by default. You choose the ones that fit your application.

Tool definitions: send only the actions that match

Section titled “Tool definitions: send only the actions that match”

The problem. A back office has dozens of actions, and each definition goes to the model in every call.

What MentorAgent does. With EnableToolFiltering, only the actions that match the message are sent. MentorAgent compares the message with the name and description of each action by embedding similarity, so an Italian message can find an English action. At most ToolFilterMaxTools actions travel (12 by default), above the ToolFilterMinScore threshold (0.35). The core tools, such as navigation and the handoff to specialists, always travel. The selection is made once per message and holds for every round trip of the turn. The filter needs an EmbeddingGenerator: without one, every tool is sent and a warning is logged.

An embedding score ranks well but is a poor threshold, so the filter has safety nets:

  • When no action clears the threshold, the three best-ranked actions that need no confirmation and no role are sent anyway. The docs put this at about 80 input tokens per call.
  • When every matching action needs confirmation, up to three best-ranked actions that do not need it travel with them.
  • When a message matches nothing, such as "Yes, go ahead", the actions of the previous message travel again.

With filtering on, the same filter also decides on each turn whether the provider's hosted tools (web search, file search) are declared (FilterHostedTools, on by default).

A measure. The test app of this article has thirty actions. For the question "What is the status of order 1042?", the request carries 31 tools (the 30 actions and the navigation tool) and 4,818 characters of definitions without the filter. With the filter, it carries 7 tools and 1,296 characters. The test runs MentorAgent's real pipeline with stand-in models and calls no provider, so the sizes are characters, not tokens, and the actions picked depend on the embedding model. What carries over is the proportion.

With a handful of actions there is little to save, while the embedding call and the risk of a wrong pick remain. Try it on your own questions.

The problem. The history goes out with every call, tool results included, so each call of a long chat is larger than the one before.

What MentorAgent does. Two settings work together.

  • MaxSessionMessages (50 by default) bounds the stored history. MentorAgent removes whole turns, oldest first, and keeps each tool call with its result, so the model still sees the data a tool read.
  • EnableCompaction shortens what is sent, without calling a model. It keeps the last CompactionMaxTurns turns (8) and leaves the older ones out. It also compacts older tool calls and, above CompactionTokenThreshold (4000 tokens), drops the oldest messages, but it always keeps the most recent ones. The stored conversation stays whole.

A measure. In the test app, fifteen turns each read a 2,800-character result. In the last request, the messages sent are 37,415 characters without compaction and 23,033 with it. From the ninth turn on, the size stays flat: the window of eight turns does the work, and a long result inside those turns is still sent whole. Capping each result at 2,000 characters with OnToolResult, as in the settings below, brings it to 16,633. Again, these are characters measured with stand-in models.

What it costs. Cutting history saves tokens and can cost accuracy. The MentorAgent README compares a history that drops tool calls and results with the default one, which keeps them: "in a lab run of 84 turns the turns answered correctly went from 85% to 95%, and tokens grew by about 9%." Compact as little as you need.

Compaction is not installed with UseServiceManagedHistory (for example the Responses API with a stored conversation): the service keeps the history, so there is nothing to trim locally.

Context: keep the start of the prompt stable

Section titled “Context: keep the start of the prompt stable”

The problem. The page, its data, its address and the time change with every page and every minute. By default they sit at the end of the instructions, which come before the whole conversation. A model that caches the prompt by its beginning then reads the whole conversation again.

What MentorAgent does. With SessionContextPlacement = LastUserMessage, those lines move to the start of the user's last message, in the request sent to the model only. The instructions stay the same through the session, and the stored conversation keeps the user's own words. In the test app, the instructions have the same size on the first and the second turn.

It matters most for a local model, where reading the conversation again can take many seconds on a CPU. Hosted providers that cache by prefix follow the same rule: the Azure OpenAI documentation asks for stable content at the beginning and dynamic content at the end, and reports cache hits as cached_tokens in the usage of each response (prompt caching (external site)). MentorAgent has no option that turns provider caching on or off. It keeps the start stable and leaves the rest to the provider.

Good to know. The RAG documents, the user's name and roles and the memory stay in the instructions. The option has no effect with UseServiceManagedHistory or a prebuilt Agent, and a warning says so. The tool filter changes the tool list from message to message, which can work against a provider that caches tool definitions with the prompt: compare the provider's cached tokens before and after.

The problem. A greeting and a multi-step analysis do not need the same model.

What MentorAgent does. Set a small ChatClient, a StrongChatClient, and a RoutingStrategy that picks the model for each turn:

StrategyHow it decidesExtra cost
Custom (default)your UseStrongModelAsync or UseStrongModelwhatever your code does
Semanticsimilarity of the message to example turnsnone beyond the embedding, shared with tool filtering
Classifiera short call asks a model whether the turn is hardone small call per turn, on ChatClient or RoutingClassifierClient
Cascadethe small model answers, a judge reads the answer, the strong model answers again if it fell shorta judge call, plus a second answer when it escalates; only without streaming: in the chat, which streams, it decides like Classifier

The decision holds for the whole tool loop of the turn. If a strategy lacks what it needs, such as Semantic without an embedding model, MentorAgent logs a warning and every turn stays on ChatClient.

ClassifierChatClient serves MentorAgent's own small decisions: the safety checks, the off-topic check, the check on a reply that called no tool, the facts to remember, the hosted-tool check and the checks behind the two "require" options. It defaults to ChatClient. Point it at a small, fast model.

Specialists, team members and the handoff router run on ChatClient, without routing or tool filter. Their tools go to them only, which keeps the coordinator's list short, but a handoff adds at least two model calls.

Some options add a call before or after the answer.

  • Memory. With UseMemoryContext, MemoryAutoCapture (on by default) makes one small call after each message to extract facts. Turned off, it saves the call, but the model writes memories with a remember tool, less reliably, and the tool and its prompt block go out with every call. MemoryContextCount (10) bounds the memories in the prompt; MemoryRelevanceFiltering sends only the related ones.
  • Safety checks. EnableSafetyCheck and EnableOutputSafetyCheck add one call each. RefuseOutOfScope adds one small call per turn, or shares the input check's call when that is on. It then ends an off-topic turn without calling the coordinator.
  • The two "require" options. RequireLookupForRecordQuestions and RequireActionForChangeRequests add one classifier call per turn. They buy reliability, not savings.
  • The check on replies without a tool. Since 1.3.0, CheckAnnouncedActions (on by default) adds one call to the classifier model after every reply that called no tool: it looks for a reply that announces or reports an action that did not run, and when it finds one the coordinator gets one more request in the same turn. It costs a few hundred tokens per such reply and runs after the reply is shown, so the user does not wait for it. CheckAnnouncedActions = false saves the call, but a reply that announces an action and runs nothing then stays as it is. More in Cost and latency and in the options reference.
  • Hosted tools. HostedToolDomainCheck asks the classifier model whether a message belongs to your application before a paid tool is declared. MaxHostedToolCallsPerSession caps a session; FileSearchMaxResults bounds what file search injects.
  • RAG. RagResultCount (5) and RagMinScore decide how many documents go into the prompt.
  • Teams. Each member's turn is a call, up to MaxIterations (10 by default in [MentorTeam]). Keep it low.

MaxMessageLength (4000 characters) refuses a long message before the model is called. RateLimitPerUser counts messages per user in a window: 0 by default, which means no limit. It holds per person only for signed-in users. Anonymous users are counted per session by default, so a reload starts again.

The library has no per-user or per-turn token budget, and no option for the maximum length of the answer. If you need one, ConfigureChatClientPipeline adds your own middleware around the coordinator's model calls. Only those calls pass through it.

Four scenarios, a few lines each. They combine: a long chat on a local model uses both.

Program.cs
using Microsoft.Extensions.AI;
using Novior.MentorAgent.Extensions;
using OpenAI;

var openAI = new OpenAIClient(builder.Configuration["OpenAI:Key"]!);

builder.Services.AddScoped<BackOfficeActions>();
builder.Services.AddMentorAgent(options =>
{
    options.AppName = "Back office";
    options.ScanAssemblies = [typeof(BackOfficeActions).Assembly];
    options.ChatClient = openAI.GetChatClient("gpt-4.1").AsIChatClient();

    // Only the actions that match the message travel with it
    options.EmbeddingGenerator = openAI.GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();
    options.EnableToolFiltering = true;
    options.ToolFilterMaxTools = 12;

    // MentorAgent's own short decisions go to a small model
    options.ClassifierChatClient = openAI.GetChatClient("gpt-4.1-mini").AsIChatClient();
});

BackOfficeActions is the class with your [MentorAction] methods. Keep the descriptions short and specific: the filter matches the message against the name and description of each action.

C#
using System.Text.Json;

options.EnableCompaction = true;
options.CompactionTokenThreshold = 4000;
options.CompactionMaxTurns = 8;

// A text result is capped before the model reads it; the hook receives it as a JsonElement
options.OnToolResult = (tool, result) =>
    result is JsonElement { ValueKind: JsonValueKind.String } json && json.GetString() is { Length: > 2000 } text
        ? text[..2000]
        : result;

The cap cuts the text. Use it where the start of a result is enough, and let structured results pass untouched, as here.

C#
options.ChatClient = openAI.GetChatClient("gpt-4o-mini").AsIChatClient();
options.StrongChatClient = openAI.GetChatClient("gpt-4o").AsIChatClient();

// Needs the EmbeddingGenerator: the message is embedded once per turn and shared with the tool filter
options.RoutingStrategy = MentorRoutingStrategy.Semantic;

options.MaxMessageLength = 2000;
options.RateLimitPerUser = 20;
options.RateLimitWindowSecs = 60;

With an embedding model already in place for the tool filter, Semantic routing adds no model call. Without one, Classifier needs nothing more, at the price of one small call per turn.

C#
using System.ClientModel;

options.ChatClient = new OpenAIClient(
        new ApiKeyCredential("ollama"),
        new OpenAIClientOptions
        {
            Endpoint = new Uri("http://localhost:11434/v1"),
            NetworkTimeout = TimeSpan.FromMinutes(20),
        })
    .GetChatClient("llama3.2-16k")
    .AsIChatClient();

// The start of the prompt stays the same through the session; the history sent stays short
options.SessionContextPlacement = MentorSessionContextPlacement.LastUserMessage;
options.EnableCompaction = true;

// The safety checks run on the same slow model: give them time
options.SafetyCheckTimeout = TimeSpan.FromMinutes(2);

MentorAgent's prompt can be longer than Ollama's default context, and Ollama then cuts the start of the prompt without an error. Create a variant with a larger context (PARAMETER num_ctx 16384 in a Modelfile) and use its name, here llama3.2-16k. More in Ollama and other local models.

Do not change five options at once. Change one, ask the same questions, read the numbers.

  • The log. Each model call writes an Information line: model, input and output tokens, time, session totals.
  • The meter. IMentorMetrics.GetSnapshot() returns tokens, requests and average latency for each model id, so the small model, the strong one and the embedding model are told apart. The MentorDashboard component and GET /mentor/admin/metrics show the same figures. The counts live in memory unless you register an IMentorMetricsStore.
  • The cost. MentorAgent ships no prices. You enter them in ModelPricing, per million tokens, under the model id the dashboard shows. The meter does not tell cached input tokens apart, so with provider caching its cost is an estimate, not the bill.
Program.cs
// Prices per million input and output tokens, under the model id the dashboard shows
options.ModelPricing = new Dictionary<string, ModelPrice>
{
    ["gpt-4o-mini"] = new(
        configuration.GetValue<decimal>("Prices:Small:InputPer1M"),
        configuration.GetValue<decimal>("Prices:Small:OutputPer1M")),
    ["gpt-4o"] = new(
        configuration.GetValue<decimal>("Prices:Strong:InputPer1M"),
        configuration.GetValue<decimal>("Prices:Strong:OutputPer1M")),
};

app.MapGet("/ops/tokens", (IMentorMetrics metrics) =>
    metrics.GetSnapshot().Models.Select(m => new
    {
        m.ModelId,
        m.InputTokens,
        m.OutputTokens,
        m.RequestCount,
        m.AvgLatencyMs,
        m.TotalCost,
    })).RequireAuthorization();

The traces of EnableObservability cover only the coordinator's model calls: wrap the other clients with UseOpenTelemetry() to trace them too. For a gate in your build pipeline, MentorEvaluator runs test prompts on your ChatClient and fails when token use or quality gets worse. It calls the model, so run it when you choose to.

LeverOption and defaultUse it when
Only the relevant toolsEnableToolFiltering off, ToolFilterMaxTools 12, ToolFilterMinScore 0.35; needs EmbeddingGeneratorthe application has dozens of actions
Shorter history sentEnableCompaction off, CompactionTokenThreshold 4000, CompactionMaxTurns 8chats are long
Capped tool resultsOnToolResult nonea tool returns long text and the start is enough
Stable start of the promptSessionContextPlacement Instructionslocal model, or a provider that caches by prefix
Small model by defaultStrongChatClient none, RoutingStrategy Customeasy and hard turns are mixed
Small model for own decisionsClassifierChatClient none (uses ChatClient)any safety, scope, memory or "require" check is on
Fewer memories in the promptMemoryContextCount 10, MemoryRelevanceFiltering offusers have many memories
Fewer documents in the promptRagResultCount 5, RagMinScore 2 or the source's ownRAG is on and answers need fewer documents
Hosted tools under controlFilterHostedTools on, HostedToolDomainCheck off, MaxHostedToolCallsPerSession 0web search or file search is on
LimitsMaxMessageLength 4000, RateLimitPerUser 0the application is open to many users

New to MentorAgent? Start with What is MentorAgent.

Written with the help of AI; code and facts checked by the MentorAgent team.