Semantic Caching in Generative AI Chatbots

Author(s): Marie Stephen Leo

Originally published on Towards AI.

Reduce LLM latency and cost by over 95% using OpenAI, LiteLLM, Qdrant, and Sentence Transformers!
Image generated by Author using Dall E 3

Latency and costs are significant challenges with LLM-based chatbots today. The problem is even more pronounced in Retrieval Augmented Generation (RAG) agents, where we must make multiple calls to the LLM before returning an answer to the user. Often, LLM RAG agent chatbots can have latencies of over 5 seconds! Semantic Caching is an easy way to drastically reduce your chatbot’s latency to <0.1s when many users ask “similar” questions.

In the context of web applications, a cache is a fast, low-latency database (DB) that temporarily stores commonly accessed data. When the app requires some information, it will first check if the cache has it, and if so, directly use the data from the cache. If the cache doesn’t have the requested data, it fetches it from the underlying transactional database (OLTP DB). This type of cache is called a Read Through cache. You can read more about different caching strategies on ByteByteGo here.

A typical web application might use a NoSQL database like Redis as a cache. In contrast, it would use a traditional SQL database like PostgreSQL as the actual transactional database, which is the final source of truth. There are two primary… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI