Agents with Retrieval and Function Calling

Introduction: The Evolution of Intelligent Systems

In the landscape of modern software development, we have moved beyond static applications that simply execute a hard-coded set of instructions. Today, we are building systems that can reason, gather context from external data, and perform actions on behalf of the user. We call these systems "Agents." An agent is essentially an autonomous entity powered by a Large Language Model (LLM) that can interact with its environment to achieve a goal.

Why does this matter? If you only use an LLM for text generation, you are limited by the model’s training data—a snapshot in time that lacks knowledge of your specific business context. By integrating retrieval systems (which provide the agent with current, relevant data) and function calling (which allows the agent to execute code or API calls), you transform a simple chatbot into a functional digital assistant. This lesson focuses on how to build these agents using Foundry, a framework designed to handle the orchestration of these complex tasks.

Callout: The Agentic Paradigm Shift Traditional software follows a deterministic path: Input A always leads to Process B and Output C. Agentic systems are non-deterministic. You provide the agent with a goal and a set of tools, and the agent decides which tools to use and in what order to reach that goal. This shift allows for the automation of complex workflows that were previously impossible to hard-code.

The Architecture of an Agent

To understand how to build an agent in Foundry, we must first break down its core components. An agent is not a single piece of code; it is an orchestration layer that sits between the user's intent and the system's capabilities.

1. The Large Language Model (LLM)

The LLM serves as the "brain." It acts as the reasoning engine that parses user requests, decides if it needs more information, determines which function to call, and interprets the results of those functions to generate a final answer.

2. Retrieval Augmented Generation (RAG)

Retrieval is the process of fetching relevant information from your data repositories (like document stores, databases, or wikis) and injecting that context into the LLM's prompt. Without retrieval, the agent is "blind" to your private data.

3. Function Calling (Tool Use)

Function calling is the mechanism that gives the agent "hands." When the model identifies a task that requires an external action—such as checking a stock price, querying a SQL database, or sending an email—it outputs a structured request to execute a specific function defined in your code.

4. The Orchestration Layer (Foundry)

Foundry manages the state, history, and communication between the LLM and the tools. It ensures that the agent follows a logical loop: Thinking, Acting, Observing, and Repeating until the task is complete.

Setting Up Your Environment

Before building an agent, you need a stable environment. Foundry provides a modular interface to define these components. You will need your API keys for your chosen LLM provider and access to your data source.

Step-by-Step Initialization

1. Install the SDK: Use your package manager to install the Foundry core libraries.
2. Configure the Provider: Initialize the LLM client by pointing it to your model endpoint.
3. Define the Context: Identify the document repository or database that will serve as the source of truth for your agent.
4. Register Tools: Map your internal functions (e.g., get_weather, query_crm) to the agent’s toolset.

Note: Always ensure that your functions are well-documented with docstrings. The LLM uses these docstrings to understand when and why it should call a specific function. A poorly described function will lead to the agent failing to use it effectively.

Implementing Retrieval (RAG)

Retrieval is the primary way your agent stays relevant. In Foundry, we implement this through a "Retriever" object that sits between the agent and your vector database.

How Retrieval Works

When a user asks a question, the agent sends the query to the Retriever. The Retriever performs a semantic search—finding documents that are conceptually similar to the user's query—and returns snippets of text. These snippets are then appended to the system prompt as "context."

Code Example: Building a Basic Retriever

from foundry import Retriever, VectorStore

# Initialize your vector store (e.g., Pinecone, Weaviate, or local FAISS)
vector_db = VectorStore(index_name="company-knowledge-base")

# Create the retriever
knowledge_retriever = Retriever(
    source=vector_db,
    top_k=3,  # How many chunks to retrieve
    similarity_threshold=0.7
)

# The agent now uses this retriever during its execution loop
agent.add_tool(knowledge_retriever)

Best Practices for Retrieval

• Chunking Strategy: Do not pass entire documents to the LLM. Break them into smaller, meaningful chunks (e.g., 500-1000 characters) to avoid exceeding token limits and to improve accuracy.
• Metadata Filtering: Use metadata (like dates or department tags) to narrow down the search space before doing a vector search. This significantly improves precision.
• Hybrid Search: Combine keyword search (BM25) with vector search for the best results, as vector search sometimes struggles with specific technical acronyms or part numbers.

Implementing Function Calling

Function calling is where the agent becomes truly autonomous. By defining tools, you allow the model to interact with the real world.

The Anatomy of a Tool

A tool in Foundry consists of three parts:

1. Name: The identifier the model uses to call the tool.
2. Description: A detailed explanation of what the tool does.
3. Schema: The expected input parameters (usually in JSON format).

Code Example: Defining a Custom Tool

from foundry import Tool

def get_order_status(order_id: str):
    """
    Retrieves the current shipping status of an order from the ERP system.
    
    Args:
        order_id (str): The unique identifier for the customer order.
    """
    # Logic to query your ERP database
    status = database.query(f"SELECT status FROM orders WHERE id = '{order_id}'")
    return status

# Registering the tool with the agent
status_tool = Tool.from_function(get_order_status)
agent.register_tool(status_tool)

The Execution Loop

When the agent receives a request like "What is the status of order 12345?", the following happens:

1. The agent analyzes the prompt and realizes it needs information not in its memory.
2. The agent outputs a JSON block indicating it wants to call get_order_status with order_id="12345".
3. Foundry intercepts this, executes the actual Python function.
4. Foundry feeds the result back to the LLM.
5. The LLM summarizes the result for the user.

Comparison of Agent Strategies

When building agents, you have different architectural choices depending on the complexity of the task.

Callout: ReAct (Reasoning + Acting) ReAct is a design pattern where the agent is prompted to "think" before it "acts." By forcing the agent to write down its reasoning process (e.g., "I need to check the database for X, then compare it to Y"), you drastically reduce the hallucination rate and improve task completion accuracy.

Common Pitfalls and How to Avoid Them

Even with a robust framework like Foundry, building agents is challenging. Here are the most common mistakes developers make:

1. Excessive Tool Bloat

If you provide an agent with 50 different tools, the model will struggle to select the right one, leading to "decision paralysis" or incorrect tool calls.

• The Fix: Keep the toolset lean. Use "tool routing" where a controller agent decides which subset of tools to make available to the worker agent.

2. The Infinite Loop

Sometimes, an agent will get stuck in a loop, calling the same tool repeatedly with the same parameters.

• The Fix: Implement a maximum iteration count in your agent configuration. If the agent doesn't reach a conclusion in, say, 5 steps, force it to return an error or ask the user for clarification.

3. Ignoring Error Handling

When a function call fails (e.g., a database timeout), the LLM might try to "guess" why it failed, which leads to unpredictable behavior.

• The Fix: Wrap all tool executions in try-except blocks. Return a clear, descriptive error message to the agent so it knows exactly what went wrong and can decide whether to retry or pivot.

4. Poor Prompt Engineering for Tools

If your function description is vague, the agent won't know when to use it.

• The Fix: Write your tool descriptions from the perspective of the agent. Instead of saying "Used for database access," say "Use this tool to fetch customer billing information when the user asks about invoices or payment status."

Advanced Workflow: Integrating Multiple Agents

As your application grows, you might find that one agent cannot handle all tasks. You can implement a "Multi-Agent" pattern where a "Manager Agent" assigns sub-tasks to "Specialist Agents."

For example, in an e-commerce scenario:

• Manager Agent: Receives the user request and decides if it is a "Billing" issue or an "Order Tracking" issue.
• Billing Agent: Has access to financial tools and sensitive payment APIs.
• Tracking Agent: Has access to logistics and inventory databases.

This separation of concerns keeps each agent focused, making them easier to test, debug, and maintain.

Step-by-Step Implementation Guide

Let’s walk through building a "Support Assistant" that can look up policy documents and check order statuses.

Step 1: Define the Tools

# Tool 1: Order Lookup
def get_order_details(order_id):
    # Logic...
    return {"status": "shipped", "delivery_date": "2023-10-25"}

# Tool 2: Policy Search
def search_return_policy(query):
    # Logic to search vector store...
    return "Returns are accepted within 30 days of purchase."

Step 2: Initialize the Agent

from foundry import Agent

support_agent = Agent(
    name="SupportBot",
    llm="gpt-4-turbo",
    system_prompt="You are a helpful support assistant. Use tools to find info."
)

support_agent.add_tool(Tool.from_function(get_order_details))
support_agent.add_tool(Tool.from_function(search_return_policy))

Step 3: Execute the Interaction

response = support_agent.run("My order 999 hasn't arrived, what is the return policy?")
print(response)

What happens under the hood:

1. The agent realizes it needs two pieces of information.
2. It calls get_order_details(order_id="999") and search_return_policy(query="return policy") in parallel.
3. It aggregates the results and generates a response: "Your order 999 has shipped and is expected on 2023-10-25. If it does not arrive, please note that our return policy allows returns within 30 days of purchase."

Best Practices for Production

Building an agent that works in a notebook is one thing; deploying it to production is another. Follow these industry standards to ensure reliability.

Monitoring and Observability

You cannot fix what you cannot see. Use tracing tools to log every step of the agent's reasoning process. If an agent gives a bad answer, you need to be able to see exactly which tool provided the wrong data or which reasoning step went off the rails.

Human-in-the-Loop (HITL)

For high-stakes actions (like processing a refund or deleting data), never allow the agent to execute the action automatically. Implement a confirmation step where the agent presents the proposed action to a human, and the human must click "Approve" before the function executes.

Security and Sandboxing

Never run agent-generated code or API calls with unrestricted permissions. The agent should always use a service account with the principle of least privilege. If the agent only needs to read from a database, the API key provided to the tool should be read-only.

Warning: Never pass raw user input directly into a database query (SQL injection) or a shell command. Always sanitize inputs within your tool functions. Even though the LLM is "smart," it is essentially just text generation and can be tricked by malicious user inputs into executing unintended commands.

FAQ: Common Questions

Q: How do I handle large amounts of data in RAG? A: Use a hierarchical indexing strategy. Store summaries of documents in one index and detailed chunks in another. The agent can first search the summaries to find the right document, then search the chunks within that document.

Q: Can I use local LLMs with Foundry? A: Yes, Foundry is framework-agnostic. As long as your local LLM (e.g., Llama 3 running via Ollama) supports tool calling (JSON mode), you can integrate it just like a cloud-based model.

Q: How do I measure agent performance? A: Use an evaluation framework. Create a test suite of 50-100 common user questions with "golden answers." Run your agent against these questions and measure how often the agent calls the correct tool and how often the final answer matches the golden answer.

Q: Does the agent need to be "stateless"? A: Generally, yes. Store the conversation history in a database (like Redis) and pass the relevant history to the agent at the start of each turn. This allows your agent to handle long-running conversations across multiple sessions.

Summary and Key Takeaways

Building agents with retrieval and function calling is the most effective way to move from "chatting with a model" to "solving business problems." By leveraging Foundry to orchestrate these interactions, you gain control over the agent's reasoning process and its ability to interact with your internal systems.

Key Takeaways

1. Agents are Orchestrators: They use an LLM to reason and tools to act. The quality of your agent depends equally on the model's intelligence and the quality of your tool definitions.
2. Retrieval is Essential: Without RAG, your agent is limited to its training data. Use vector databases to provide the agent with the specific, up-to-date information it needs.
3. Tool Design Matters: Write clear, descriptive docstrings for every tool. A tool is only as useful as the model's ability to understand when to use it.
4. Adopt a ReAct Pattern: Encourage your agents to "think" before they "act." This structure leads to more reliable, debuggable, and accurate results.
5. Prioritize Safety: Implement human-in-the-loop controls for sensitive actions and ensure all tool-based interactions operate under the principle of least privilege.
6. Monitor Everything: Use tracing to log the agent's thought process. This is the only way to effectively debug non-deterministic systems.
7. Iterate and Evaluate: Building agents is an iterative process. Use a test suite of queries to measure performance and refine your prompts and tools based on data, not intuition.

By mastering these concepts, you are no longer just building software; you are building intelligent systems that can scale your operations and provide deeper value to your users. Start small with a single tool, prove the concept, and gradually expand your agent's capabilities as you learn more about its behavior and limitations.