Exploring GraphRAG: Smarter AI Knowledge Retrieval with Neo4j & LLMs

Last Updated on March 6, 2025 by Editorial Team

Author(s): sridhar sampath

Originally published on Towards AI.

Exploring GraphRAG: Smarter AI Knowledge Retrieval with Neo4j & LLMs

Traditional AI models struggle with retrieving structured knowledge efficiently. Large Language Models (LLMs) rely on text-based data, often leading to hallucinations, fragmented context, and limited reasoning.

GraphRAG (Graph + Retrieval-Augmented Generation)-a technique that enhances AI capabilities by integrating Neo4j Knowledge Graphs with LLMs like OpenAI.

In this guide, I explore GraphRAG’s potential using a Football Knowledge Graph chatbot, but this technique applies to finance, healthcare, legal AI, and enterprise knowledge management.

📌 Why GraphRAG? (Graph + RAG)

GraphRAG (Graph + Retrieval Augmented Generation) is a technique pioneered by Microsoft, designed to enhance the accuracy and reasoning of LLM (Large Language Model) responses using knowledge graphs. Traditional LLMs often struggle with hallucinations, fragmented context, and limited reasoning-GraphRAG fixes these gaps by introducing structured graph-based retrieval before generating AI responses.

🔹 Key Advantages of GraphRAG

✅ Improved Contextual Understanding:
In this demo , LLMs retrieve knowledge from Neo4j GraphDB, allowing them to understand the relationships between players, clubs, leagues, and historical data, resulting in more contextually accurate answers.

✅ Higher Accuracy & Reduced Hallucinations:
By grounding responses in a structured knowledge graph, GraphRAG ensures fact-based retrieval from trusted sources rather than relying on the LLM’s pretrained memory.

✅ Multi-Hop Reasoning & Deep Insights:
Unlike simple retrieval, GraphRAG supports multi-hop queries-enabling complex questions like:
“Which player has the highest goal-scoring in La Liga?” This requires chaining multiple relationships:
(Players → Clubs → Leagues → Goals)

LLM Output Finally for above Cypher

✅ Increased Transparency & Source Traceability:
Every response is backed by structured Cypher queries to Neo4j, ensuring that answers can be traced back to the graph for verification.

🔹 The GraphRAG Process: From Query to Answer

GraphRAG combines Knowledge Graphs, Graph Retrieval, and LLM Summarization in a single AI pipeline:

1️⃣ Neo4j Knowledge Graph Construction → Extract structured Nodes and relationships (Players, Clubs, Leagues).
2️⃣ Graph-Based Retrieval → Convert user queries into Cypher queries for structured retrieval.
3️⃣ LLM Response Generation → Use GPT to format the retrieved knowledge into human-readable responses. Model which I used is gpt-3.5-turbo.

🚀 Why This Matters?
Traditional LLMs rely solely on embeddings, but GraphRAG + Neo4j enables AI-driven reasoning, making AI more explainable, accurate, and scalable.

Below diagram shows the Graph-Based Retrieval and the LLM Response Generation .

GraphRAG in Action: Building a Football Knowledge Graph Chatbot

To illustrate GraphRAG’s capabilities, lets’s built a Football Knowledge Graph Chatbot using Neo4j, OpenAI, and Streamlit.

⚽ Why Football Data?

Football is just an example use case to showcase how GraphRAG enhances AI-powered retrieval. The same approach applies to medical,finance, legal, or enterprise AI applications.

Used the Kaggle dataset: Top Football Leagues Scorers

This dataset includes:

✅ Top goal scorers from major leagues

✅ Players, clubs, and league affiliations

✅ Performance stats (goals, xG, shots, matches played, etc.)

This is the Excel data Screenshot

Final Demo Screenshot

Neo4j Auro Console Output for the above Query in our Demo

📢 Note:

This dataset is not exhaustive and I used it for structured football data sample for demo purpose. The results are based only on the CSV data loaded into Neo4j and do not reflect real-world live stats.

In this guide, lets walk through how to built a Football Knowledge Graph Chatbot that combines Neo4j, OpenAI, and Streamlit to answer complex football queries.

📂 Step 1: Creating a Neo4j Knowledge Graph

We set up a Neo4j AuraDB instance and connect it using the Neo4j Python driver. The Football Knowledge Graph structures relationships between Players, Clubs, and Leagues, enabling efficient retrieval of football insights.

Create your free Neo4j AuraDB instance here: Neo4j AuraDB Setup Guide

Graph Structure:

Players → (:Player)-[:PLAYS_FOR]->(:Club)
Clubs → (:Club)-[:PART_OF]->(:League)
Leagues → (:League)-[:IN_COUNTRY]->(:Country)

Once the data is loaded, we can visualize our graph structure in the Neo4j Browser. Below is a sample screenshot of the Football Knowledge Graph displaying the connected entities:

Screenshots from Neo4j Aura

✅ This enables powerful knowledge retrieval, such as:

“What are the stats for Erling Haaland?”
“Who has played the most matches in the Bundesliga?”
“Which players have similar goal-scoring stats to Mohamed Salah?”

Loading Data into Neo4j (File: `football_kg_loader.py`)

1️⃣ Prepare a CSV file with player, club, and league data.

2️⃣ Use Python & Neo4j Driver to insert data into the graph.

3️⃣ Run Cypher queries to define relationships like .

MERGE (p:Player {name: "Lionel Messi", year: 2023, goals: 30, matches: 38}) 
RETURN p

🔍 Step 2: Enhancing Retrieval with OpenAI Embeddings

Why Use AI Embeddings?

🔹 Find players with similar playing styles
🔹 Compare performance stats across leagues
🔹 Improve AI-powered player recommendations

How It Works?

Enhance our GraphRAG pipeline by:
1️⃣ Extracting player statistics (goals, xG, matches, shots, etc.).
2️⃣ Generating OpenAI embeddings for each player.
3️⃣ Storing embeddings in Neo4j for efficient retrieval.
4️⃣ Performing similarity search using vector queries.

Creating a Vector Index for Player Embeddings (File: `football_kg_embeddings.py`)

Before the storing and Quering embeddings, We need to create a vector index in Neo4j.

Why Create a Vector Index?

🔹 Fast retrieval: Helps in performing quick similarity searches over thousands of players.
🔹 Optimized search: Uses cosine similarity to efficiently compare player embeddings.
🔹 Structured AI-powered queries: Allows Neo4j to store & query OpenAI-generated embeddings directly.

Cypher Query to Create Vector Index:

CREATE VECTOR INDEX football_players_embeddings IF NOT EXISTS 
FOR (p:Player) ON (p.embedding) 
OPTIONS { 
 indexConfig: { 
 `vector.dimensions`: 1536, 
 `vector.similarity_function`: 'cosine' 
 } 
}

👉 This ensures that we can efficiently retrieve similar players based on statistical embeddings stored in the graph.

Storing Embeddings in Neo4j

Once the index is created, we generate vector embeddings using OpenAI’s Embedding API and store them in Neo4j.

CALL db.create.setNodeVectorProperty(p, "embedding", vector)

✅ This process converts numerical player stats into AI-readable embeddings, allowing similarity-based retrieval.

Performing Similarity Search

With embeddings stored in the knowledge graph, we can now retrieve players similar to a given player using a vector search query:

CALL db.index.vector.queryNodes( 
 'football_players_embeddings', 5, 
 genai.vector.encode("Find players similar to Lionel Messi") 
) YIELD node AS player, score 
RETURN player.name, score

✅ This query returns the top 5 players whose stats closely match Messi’s.

🤖 Step 3: AI Chatbot with Streamlit + OpenAI

Building the Chatbot (File: `football_kg_chatbot.py`)

How It Works:

1️⃣ User asks a football question in Streamlit.

2️⃣ LLM converts the query into Cypher.

3️⃣ Neo4j fetches structured football data.

4️⃣ LLM (OpenAPI) formats a natural language response and enhances the response.