Author(s): Vatsal Saglani

Originally published on Towards AI.

Part 4: Abstracting the LLM Layer: Connecting Models to MCPBlog Header Generated by the Author

In our previous post, we built a minimal MCP Client hub, a lightweight FastAPI service that discovers tools from our specialized servers and routes execution requests. This hub provides a clean interface, but it’s only half the story. Now we need the intelligence: the LLM Host that decides which tool to use and when.

The challenge isn’t just connecting to the LLM. It’s building an abstraction that lets us swap models without touching our tool ecosystem. Today’s choice might be o3, GPT-4.1, Llama-4-Maverick, DeepSeek R1/V3, or Claude Sonnet 3.7, but we’ve been seeing many LLMs coming out every month. We might even want to use some local LLMs using Ollama or LMStudio. Our architecture needs to handle this flexibility while maintaining consistent tool interactions.

We’ll solve this with two key components. First, a provider system that abstracts away model-specific details. Whether we’re calling OpenAI’s API, using Groq’s OpenAI-compatible endpoint, or connecting to Anthropic, the rest of our code shouldn’t need to know the difference. Second, we’ll build an LLM MCP Client that handles the discovery-execution loop, parsing tool calls from LLM output and feeding results back to the next decision.

This separation creates a powerful pattern:… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI