Tracing a LLM Agent with LangSmith

Author(s): Pere Martra

Originally published on Towards AI.

This article is based on content from my book “Large Language Models Projects: Apply and Implement Strategies for Large Language Models” (Apress) and a free GitHub course about LLMs.

For the creation of this article, I will be using an agent that was built in a previous article: How to Create a Medical Agent / RAG System.

But don’t worry; both the article and the accompanying notebook contain all the necessary code. The explanations on how to create the agent won’t be as detailed as in the mentioned article, as in this case, I will focus more on LangSmith and how the internal traffic of the agent is traced.

As always, this article will be based on the code from a notebook, which you can find here:

My recommendation is to have the notebook open while reading the article and create your own version when you’re finished.

First, let’s load the dataset. As I mentioned earlier, I’ll include the code but without detailed explanations.

!pip install -q langchain==0.3.0
!pip install -q langchain-openai==0.2.0
!pip install -q langchainhub==0.1.21
!pip install -q datasets==3.0.0
!pip install -q chromadb==0.5.5
!pip install -q langchain-community==0.3.0

from datasets import load_dataset
data = load_dataset("keivalya/MedQuad-MedicalQnADataset", split='train')
data = data.to_pandas()
# Uncoment this line if you want to limit the size of the data.
# data = data[0:100]

Now you have the dataset loaded into the variable Data. Remember, if you're running a test, it's better to use only a portion of the data to save on execution time.

The tool our agent uses is a retriever, which will search for information stored in a vector database, essentially functioning as a simple RAG system. The vector database I’ll be using is ChromaDB, and since the agent will be built using LangChain, is necessary to load some libraries, and load the data.

Before continuing, I’d like to emphasize that it’s not necessary to use LangChain to create the agent. You can use any other framework, such as LlamaIndex, and it will work just as well with LangSmith.

Load and store the dataset in the vector database.

from langchain.document_loaders import DataFrameLoader
from langchain.vectorstores import Chroma

df_loader = DataFrameLoader(data, page_content_column="Answer")

The relevant data for our agent is stored in the df_loader variable, and it needs to be transferred to the vector database. However, since these are texts of considerable length, they need to be split into chunks first.

To split the document into different chunks, we’ll use the CharacterTextSplitter class from Langchain.

from langchain.text_splitter import CharacterTextSplitter

text_splitter = CharacterTextSplitter(chunk_size=1250,
 separator="\n",
 chunk_overlap=100)
texts = text_splitter.split_documents(df_document)

The overlap is the amount of text that is repeated between chunks. For example, if you specify an overlap of 100, the first 100 characters of a chunk will match the last 100 characters of the previous one.

The chunk won’t always be split at character 1250; in fact, this will happen very rarely. To split the text, the function waits until it finds the specified separator character, by default it is ‘\n\n’ but I changed it by ‘\n’, so there will be times when it splits before and other times later.

Time to transform the text into embeddings and store them into ChromaDB.

But first, let’s configure the environment with the necessary keys to proceed with the development.

os.environ["LANGCHAIN_API_KEY"] = getpass("LangChain API Key: ")
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_ENDPOINT"]="https://api.smith.langchain.com"
os.environ["LANGCHAIN_PROJECT"]="langsmith_test_agent"

You can obtain Your LangChain API Key from your Personal->Settings Area in LangSmith panel.

Everything is now ready to create the database and load the properly formatted information into it.

# We load the text-embedding-ada-002 model from OpenAI.
from langchain_openai import OpenAIEmbeddings

model_name = 'text-embedding-ada-002'

embed = OpenAIEmbeddings(
 model=model_name
)

To transform the text into embeddings, we will use OpenAI’s model ‘text-embedding-ada-002’.

You create the vector database by passing it the text and the embeddings model.

model="gpt-4o"
#model="gpt-3.5-turbo"

from langchain.chat_models import ChatOpenAI
from langchain_openai import OpenAI
from langchain.chains.conversation.memory import ConversationBufferWindowMemory
from langchain.chains import RetrievalQA

llm=OpenAI(temperature=0.0)

conversational_memory = ConversationBufferWindowMemory(
 memory_key='chat_history',
 k=4, #Number of messages stored in memory
 return_messages=True #Must return the messages in the response.
)

qa = RetrievalQA.from_chain_type(
 llm=llm,
 #chain_type="stuff",
 retriever=chroma_db.as_retriever()
)

It may seem that we’ve only created the retriever that the agent will use to access the information from the vector database. And that’s true, but since you’ve already configured the necessary environment variables for LangSmith to function, as soon as the retriever is used, information will start being logged in LangSmith.

First logs in LangSmith.

For example, something as simple as this call to the retriever:

qa.run("What is the main symptom of LCM?")

Generates an entry in LangSmith with:

• the query.
• the documents found.
• the returned text.
• and the time taken for each action.

But there’s more! You’ll also find the prompt that was created alongside the model’s response.

Create & trace the Agent.

I suppose you’re as surprised as I was the first time I used LangSmith. “But I haven’t done anything! I just had to set a few environment variables, and that’s it!” Well, yes, it’s really that simple — and there’s more.

To continue, the first step will be to create the agent, starting with the tool it will use to access the information.

from langchain.agents import Tool

#Defining the list of tool objects to be used by LangChain.
tools = [
 Tool(
 name='Medical KB',
 func=qa.run,
 description=(
 """use this tool when answering medical knowledge queries to get
 more information about the topic"""
 )
 )
]

You create the agent and the agent executor:

from langchain.agents import create_react_agent
from langchain import hub

prompt = hub.pull("hwchase17/react-chat")
agent = create_react_agent(
 tools=tools,
 llm=llm,
 prompt=prompt,
)

# Create an agent executor by passing in the agent and tools
from langchain.agents import AgentExecutor
agent_executor2 = AgentExecutor(agent=agent,
 tools=tools,
 verbose=True,
 memory=conversational_memory,
 max_iterations=30,
 max_execution_time=600,
 handle_parsing_errors=True
 )

Now, we can make the call to the agent and see how the information is stored in LangChain.

agent_executor2.invoke({"input": """I have a patient that can have Botulism,
how can I confirm the diagnosis?"""})

You can see that the amount of traced information is quite high. The call to the Retriever is grouped under RetrievalQA, and the final response is under OpenAI.

As you can understand, I can’t include all the LangSmith screens with all the generated information. I don’t think it would add much value, and it would be a waste of time and space. If you’re interested in exploring the stored information, the best thing to do is to run the notebook and see it for yourself!

Conclusion.

If you weren’t already familiar with LangSmith, I assume you’re now surprised by its power and ease of use.

I recommend you also read the article: Evaluating LLM Summaries using Embedding Distance with LangSmith. It uses LangSmith’s evaluators to gather metrics on the performance of a Large Language Model.

LangSmith is part of a new wave of tools connected to the world of Large Language Models, developed to make building applications with them easier.

The full course about Large Language Models is available at Github. To stay updated on new articles, please consider following the repository or starring it. This way, you’ll receive notifications whenever new content is added.

I write about Generative AI, Deep Learning and TensorFlow regularly. Consider following me on Medium to get updates about new articles. And, of course, You are welcome to connect with me on LinkedIn.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI