Introduction to RAG: Basics to Mastery. 3-Agentic RAG-Giving Your Retrieval Pipeline a Brain

Author(s): Taha Azizi

Originally published on Towards AI.

Part 3 of the mini-series introduction to RAG

Introduction

In the last two articles, we built:

1. A basic RAG Pipeline with semantic search.
2. A Hybrid RAG that combined keyword + semantic search.

Now we’re going one step further — making RAG agentic.

An Agentic RAG doesn’t just retrieve once and answer — it can:

• Plan multi-step retrievals
• Call external tools (search engines, APIs, calculators)
• Reason about what’s missing and fetch additional info before answering

This approach is especially useful for complex, multi-hop questions.

An Agentic RAG system doesn’t just retrieve once and answer — it can plan multi-step retrievals, reason, and use tools.

Agentic System Overview

Our agent now operates in two main modes:

1. RAG Retriever:
   Retrieves relevant information from a document collection using semantic, keyword, or hybrid search techniques and generates an answer using a language model.
2. Calculator:
   Evaluates mathematical expressions safely when the query involves arithmetic operations. This prevents the LLM from generating potentially wrong calculations. Example implementation:

def tool_calculator(expression: str) -> str:
 try:
 import math
 return str(eval(expression, {"__builtins__": {}}, {"math": math}))
 except Exception as e:
 return f"CALC_ERROR: {e}"

This approach is especially useful for complex, multi-step questions like:

“What is the total renewable energy growth in Germany if last year it grew by 8% and the previous year by 6%?”

A basic RAG might only retrieve growth rates.
An agentic RAG with a calculator tool will:

1. Retrieve growth rate data from the documents.
2. Calculate combined growth using the calculator tool.
3. Combine both into one accurate answer.

Theory

Agentic RAG is powered by LLM agents — models that don’t just answer but decide what steps to take next.

Key agentic capabilities:

• Break down tasks into logical steps.
• Pick tools dynamically: database retriever, keyword search, API, calculator, web search.
• Reason iteratively: if an answer looks incomplete, the agent can try again with a refined query.
• Integrate knowledge sources: mix local documents, external APIs, and reasoning steps.

Common frameworks for Agentic RAG include:

• LangChain Agents
• LlamaIndex Agents

For this tutorial, we’ll use LangChain with Ollama, since it works fully locally and is easy to extend with extra tools.

Setup

Install dependencies:

pip install langchain langchain-community sentence-transformers chromadb

(We already have Ollama, ChromaDB, and embeddings from previous articles.)

Step-by-Step Code

Here is the step by step guide:

1. Create Tools

We’ll wrap our RAG retriever from Article 1 into a LangChain Tool.

RAG Retriever Tool:

from langchain.tools import Tool
def rag_search(query):
 query_embedding = embedder.encode([query], convert_to_numpy=True)[0]
 results = collection.query(query_embeddings=[query_embedding], n_results=3)
 return "\n".join(results["documents"][0])
retrieval_tool = Tool(
 name="RAG Retriever",
 func=rag_search,
 description="Use this to search local documents for relevant context."
)

Calculator Tool:

calculator_tool = Tool(
 name="Calculator",
 func=tool_calculator,
 description="Use this to perform safe arithmetic calculations."
)

2. Set Up an Ollama-Powered Agent

from langchain.llms import Ollama
from langchain.agents import initialize_agent, AgentType
llm = Ollama(model="mistral")
agent = initialize_agent(
 tools=[retrieval_tool],
 llm=llm,
 agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
 verbose=True
)

3. Ask a Multi-Hop Question

query = "Summarize the top three renewable energy policies in Germany from our dataset."
agent.run(query)

4. What Happens Here

The agent workflow looks like this:

• The agent reads the question.
• It decides it needs the RAG Retriever tool.
• It retrieves chunks from your local Chroma database.
• It may call the retriever multiple times for different sub-queries.
• It generates a final synthesized answer.

Expected Output

You’ll see LangChain printing the reasoning steps:

> Entering new AgentExecutor chain...
Thought: I need to find Germany’s renewable energy growth for last two years.
Action: RAG Retriever
Action Input: renewable energy growth Germany last two years
Thought: I need to combine the growth rates.
Action: Calculator
Action Input: "0.08 + 0.06"
Final Answer: The combined renewable energy growth in Germany for the last two years is 14%.

Why This Is Powerful

Agentic RAG shines when:

• The question requires multi-step reasoning.
• Ensures accuracy for calculations using the calculator tool.
• Information needs to be combined from multiple sources.
• You want to extend capabilities with new tools (web search, APIs, calculators).
• You want the system to be adaptive instead of static — avoiding “one-shot” retrieval failures.

Next Steps

In Article 4, we’ll explore RAG with MCP (Model Context Protocol) — letting the model fetch context dynamically during the conversation in a more standardized way.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI