Remember me
When multiple AI agents are strung together to cooperate and address complex workflows, the sheer volume of tokens — text elements or units, so to speak — may easily escalate. Everything adds up: from memory logs to detailed tool specifications, system instructions, and so on. Eventually, this leads to dragged down speed of executions and computing budget exhaustion.
Consequently, managing token usage is vital for today's AI developers and practitioners as a whole. There's good news, though: scaling up and streamlining a multi-agent architecture doesn't necessarily entail equal scaling of costs if you know how to properly implement some strategies for saving token usage. This article introduces and shows four of them in action.
# Four Key Strategies for Saving Token Usage
Below are four commonly adopted best practices to streamline multi-agent AI solutions while optimizing the use of tokens.
// 1. Using Static Instruction Caching (Prefix-Match Caching)
Consider this one as a "don't repeat yourself" rule. Large language models (LLMs), an indispensable part of modern AI agents, invest much processing energy re-reading the same system prompts in successive turns. Prefix caching, which consists of storing key-value pairs, helps address this issue by storing such static, long instructions as a reference guide prepared ahead of time. Rather than having the model re-read the whole "how-to-act-as-this-agent" instruction manual every time, the model bookmarks a summarized state upon receiving a query. In subsequent turns, the model only needs to open that bookmark and go straight to processing the new prompt. As a result, latency due to preparation is significantly cut down, and so are the associated token costs.
// 2. Using Semantic Caching: Intent-Based Recall
If an AI agent has already solved a specific problem before, why ask it to generate a brand new response from scratch? This strategy leverages embeddings — numerical, vector-based representations of text that "retain" semantic properties — and uses them to quickly identify similar past intents. For instance, two distinct users' prompts like "How can I reset my router?" and "What are the steps to restart my wifi box?" would be recognized as the same intent based on semantic caching. In certain cases, this opens up the opportunity of entirely bypassing the LLM while still providing the right answer.
// 3. Using Just-in-Time Tooling
Also known as lazy loading, this technique is designed to tackle a frequent pitfall in AI agent building: front-loading context windows with huge "reference manuals" of every single API, tool, and database schema within their reach. Of course, that would be the perfect recipe for bloated, noisy prompts and excessive token consumption. Instead, why not give the agent a high-level, lean directory of its capabilities? Only when the agent identifies a specific task required at a given moment does it trigger the fetching of the detailed, fine-grained instructions and parameters needed for that specific tool.
// 4. Using Task Escalation: Cost-Efficient Model Routing
Not all user prompts require a massive, heavy-hitting model to be properly addressed. Effective multi-agent AI architectures are designed to act as triage centers, endowed with a routing layer that analyzes every incoming task based on its nature and complexity. Accordingly, simpler tasks like formatting data, summarizing text, or classifying intent are routed to lightweight, often free models capable of successfully running these tasks locally. Meanwhile, heavy, compute-intensive models where token consumption matters are "reserved" exclusively for complex tasks like those requiring deep reasoning or orchestration across multiple steps.
# Implementation Example in a Nutshell
Now that we've covered four practical strategies that can help optimize token usage in multi-agent AI applications, how about illustrating how some of them work through a high-level example?
This code snippet illustrates how to combine two of them: semantic caching and model routing. The code uses an actual model — a sentence transformer — to convert text into the embeddings needed for semantic caching. The calls to LLMs are mocked, but you can easily replace the code (especially for the lightweight model part) with an actual, free-weights model like those available at Groq, as shown in this article, for instance.
import numpy as np from sentence_transformers import SentenceTransformer # Loading a free, local model to convert text into embeddings embedder = SentenceTransformer('all-MiniLM-L6-v2') # In-memory semantic cache and similarity threshold (0.90 = 90% similar) semantic_cache = SIMILARITY_THRESHOLD = 0.90 def cosine_similarity(vec1, vec2): """Calculates how closely related two queries are.""" return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2)) def route_and_respond(user_query): # 1. Converting the current query into an embedding vector query_vector = embedder.encode(user_query) # 2. Semantic Caching: Check if a similar problem was solved recently for cached_vector, past_response in semantic_cache.values(): if cosine_similarity(query_vector, cached_vector) >= SIMILARITY_THRESHOLD: return f"[Served from Cache] " # 3. Model Routing: Triage the task based on complexity # Simple tasks get routed to a free, locally hosted model (e.g. Llama 3 via Ollama) # This routing logic is illustrative-only; not be used in production if "summarize" in user_query.lower() or len(user_query)
The complexity of the task requested in the prompt passed to route_and_respond() will determine which model type is used.
The output of executing this code will be either one of the two return messages in these functions:
def call_free_local_agent(prompt): return "Action completed by local, zero-cost model." def call_heavy_reasoning_agent(prompt): return "Action completed by complex orchestration agent."# Wrapping Up
This article described four key strategies to be aware of when implementing multi-agent AI applications and architectures, with emphasis on optimizing token usage and reducing costs and latency. Through a practical, mock-based example, we reinforced our understanding of applying two of them in combination: semantic caching and model routing.
Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.