Accelerating Drug Approvals Using Advanced RAG

Author(s): Arunabh Bora

Originally published on Towards AI.

Using RAG with multi-representation indexing to get full context data from technical documents

This member-only story is on us. Upgrade to access all of Medium.

This article is inspired by a project I recently did, which was centered around fetching a lot of technical data from PDF documents (mostly tables, but they also had some images and chemical names). I initially tried to do it using a basic RAG (Retrieval Augmented Generation) approach but I found that it was not able to fetch the full context of information from the documents. It was either fetching incomplete tables or mixing up the information with text from another part of the documents.

Since I was dealing with a lot of regulatory data, I needed something that would capture the complete context from the raw documents without adding any interpretations.

Large language models are trained on a lot of generic data. We often want to augment that data with our own private and confidential data. RAG bridges this gap by integrating an our own datasets with the pre-trained models. RAG is widely used throughout industries for building tools, where users obtain information from a large corpus of data by ‘conversing’ with it.

Filings or drug dossiers are collections of documents submitted by pharmaceutical companies to regulatory… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI