Author(s): Michalzarnecki

Originally published on Towards AI.

Hi! In this chapter I’ll explain what is the purpose of using vector databases in LLM-based applications and why embeddings are so important in natural language processing.

There are multiple database engines that support data storage for vectorized texts, among others Pinecone, Chroma, Weaviate, Qdrant, Faiss. Also popular projects like elasticsearch and PostgreSQL support vector storage, even SQLite can be used with extension that enables features related to vector type and operations.

Why vector store is so important?

Vector storage is very important in natural language processing as it stores the meaning of the text.
Consider popular example below where we represent word “king” as 2-dimensional vector [2, 4] and “queen” as vector [5, 3] and similar representations for “man” and “woman”. With such represenatation we can actually do calculations between words and even whole texts keeping their dependencies which are stored as float number in multidimensional vector.

For example “king” minus “man” + “woman” is close to “queen”.
“I walked” minus “I walk” is close to “I ran” minus “I run”,
and so on…

Below we have 100-dimensional representation of word “go”.

Additionally let’s take a look at the same representation for word “away”.

Now we can take average from “go” and “away” which gives us representation for phrase “go away”. What is important here is that result of this computation gives as a result vector of the same 100-dimensional length.

Each float numbers stores some abstraction about meaning of the text. It’s hard to interpret each single number separately but together they stand for complex meaning of each text.

To compare vectors we use different methods. One popular method is cosine similarity which measures cosine of angle between 2 vectors representing compared texts.

When texts are similar there is small angle between their vector representations and cosine is close to 1.
When text are very different the angle between them is close to 90 degrees and cosine is equal to 0.
Finally when texts have opposite meaning the angle is 180 degrees and cosine is close to -1.
Considering above we have a very useful metric which can compare two texts by their meaning. For instance we can encode one text to embeddings representation (vector) and compare it with other vectorized texts in database by cosine similarity, then pick results close to 1. By this operation we perform semantic search.

Let’s jump now to the code examples showing how to get text embeddings and vector storage with semantic search.

Install libraries

!pip install faiss-cpu sentence_transformers

Generate embeddings

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

model.max_seq_length = 256

sentences = [
 "dinosaurs live in africa but in different time dimension", 
 "this is sentence about little cat that liked this eat fast food",
 "this is the another sample sentence which is here just this not be matched while other one is"
]

embeddings = model.encode(sentences, normalize_embeddings=True)

embeddings

output:

array([[-0.04987683, 0.03634831, 0.01747592, ..., -0.05154557,
 0.01327896, -0.05160031],
 [ 0.07967837, 0.06043367, 0.0579344 , ..., 0.1495044 ,
 0.06014546, 0.02409914],
 [-0.00895106, 0.05474589, 0.08349688, ..., 0.0095391 ,
 0.09756648, 0.00701756]], shape=(3, 384), dtype=float32)

print(len(embeddings[1]))

output:

384

Create vector database and load documents

import faiss

d = 384 # number of dimensions

index = faiss.IndexFlatL2(d) # build the index
index.add(embeddings) # add vectors to the index

Search the vector database

queryText = "french fries"
embeddingSearch = model.encode([queryText], normalize_embeddings=True)
embeddingFound, idx = index.search(embeddingSearch, 1) # actual search
print(queryText + " matches:\\n" + sentences[idx[0][0]])

queryText = "not similar text"
embeddingSearch = model.encode([queryText], normalize_embeddings=True)
embeddingFound, idx = index.search(embeddingSearch, 1) # actual search
print(queryText + " matches:\\n" + sentences[idx[0][0]])

output:

french fries matches:\nthis is sentence about little cat that liked this eat fast food
not similar text matches:\nthis is the another sample sentence which is here just this not be matched while other one is

Vector database stored on disk

Last example show usage of SQLite with sqlite_vss extension. This configuration allows to store vectorized texts data on disk.

!pip install sentence-transformers sqlite-vss

import sqlite3
import json
import numpy as np
from sentence_transformers import SentenceTransformer
import sqlite_vss

DB_PATH = "vectors.db"
DIM = 384
model = SentenceTransformer("all-MiniLM-L6-v2")

def embed_norm(texts):
 v = model.encode(texts).astype("float32")
 v /= (np.linalg.norm(v, axis=1, keepdims=True) + 1e-12) # kosinus → IP
 return v

con = sqlite3.connect(DB_PATH)
con.enable_load_extension(True)
sqlite_vss.load(con)
cur = con.cursor()

# 1) Tabel
cur.executescript(f"""
CREATE TABLE IF NOT EXISTS docs(
 id INTEGER PRIMARY KEY,
 text TEXT NOT NULL
);
CREATE VIRTUAL TABLE IF NOT EXISTS doc_index USING vss0(emb({DIM}));
""")
con.commit()

# 2) Data + index
docs = [
 (1, "The Fibonacci sequence starts with 0 and 1."),
 (2, "Dijkstra's algorithm finds the shortest paths in a graph."),
 (3, "Recursion is a function calling itself."),
]
cur.executemany("INSERT OR IGNORE INTO docs(id, text) VALUES (?,?)", docs)

embs = embed_norm([t for _, t in docs]).tolist()

# (a) if exist — remove old vectors by rowid
cur.executemany("DELETE FROM doc_index WHERE rowid = ?", [(d[0],) for d in docs])
# (b) insert as raw JSON string
cur.executemany(
 "INSERT INTO doc_index(rowid, emb) VALUES (?, ?)",
 [(docs[i][0], json.dumps(embs[i])) for i in range(len(docs))]
)
con.commit()

query = "How does the Fibonacci sequence begin?"
q = embed_norm([query])[0].tolist()

rows = cur.execute("""
WITH knn AS (
 SELECT rowid, distance
 FROM doc_index
 WHERE vss_search(emb, ?)
 ORDER BY distance DESC
 LIMIT 5
)
SELECT d.id, d.text, knn.distance
FROM knn
JOIN docs AS d ON d.id = knn.rowid
ORDER BY knn.distance DESC;
""", (json.dumps(q),)).fetchall()

for rid, text, score in rows:
 print(f"id={rid} score={score:.4f} text={text}")

id=2 score=1.6242 text=Dijkstra's algorithm finds the shortest paths in a graph.
id=3 score=1.1552 text=Recursion is a function calling itself.
id=1 score=0.3085 text=The Fibonacci sequence starts with 0 and 1.

That’s all in this part dedicated to vector storage, embeddings and semantic search. In the next article we will build RAG chatbot application using streamlit framework.

see next chapter

see previous chapter

see the full code from this article in the GitHub repository

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI