Streaming with Pydantic AI

Author(s): Barrett Studdard

Originally published on Towards AI.

Building a pattern around LLM streamingCredit Volodymyr Hryshchenko on Unsplash

In my prior article on building a DAX LLM, I utilized base API calls to stream data from Anthropic. In this article, we’ll look to utilize Pydantic AI to further productionize and expand on this pattern to also incorporate tool calls.

Pydantic AI is a newer LLM agent framework. It’s built by the team behind Pydantic, which I’m a fan of through past use in FastAPI projects. It’s smaller in scope compared to something like LangChain and provides a bridge for creating LLM building blocks.

async def anthropic_stream_api_call(chat_input_list: list) -> AsyncGenerator[str, None]: """Streams anthropic response. Args: chat_input_list (list): List of chat inputs to send to the API. Yields: AsyncGenerator[str, None]: Stream of anthropic response. """ # Build message list message_input = build_anthropic_message_input(chat_input_list=chat_input_list) # Setup and make api call. client = AsyncAnthropic(api_key=ANTHROPIC_API_KEY) stream = await client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=4096, temperature=0.2, system=get_system_prompt(chat_input_list), messages=message_input, stream=True ) async for event in stream: if event.type in ['message_start', 'message_delta', 'message_stop', 'content_block_start', 'content_block_stop']: pass elif event.type == 'content_block_delta': yield event.delta.text else: yield event.type

The most relevant example with the base API call approach is the following function:

The API call is made in an async function and then events handled for streaming and displaying where needed via a… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI