Designing Multimodal AI Search Engines for Smarter Online Retail

Author(s): Ashish Abraham

Originally published on Towards AI.

“Wow, that shirt looks amazing. I want one just like it!” No brand name, no fashion jargon, not even the fabric. Just a guess and a quick search on Amazon. Instantly: thousands of options, all sizes, every color. One click, and it’s at your door the next day. That’s the magic of AI-powered shopping and the new face of customer satisfaction.

AI has shifted the landscape for many industries lately, and e-commerce is one of the biggest beneficiaries. In the past, I searched by colors and brands and often abandoned the search when I couldn’t find the right match. Now, I can describe exactly what I want and get it instantly — that’s the change AI has brought. An efficient, personalized search engine is one of the most critical features for a successful online business. AI goes beyond simple keyword matching by analyzing user intent, context, and behavior, thereby reducing irrelevant results and abandoned searches. According to recent studies, this advancement can boost revenue by up to 40% and enhance customer satisfaction.

Table of Contents

· Search Engines Beyond Text
· Vectors in Search Engines
∘ Vector Embeddings
∘ How Images Fit in
∘ Sparse Vectors
· Payload and Metadata Filtering
· Reranking for Quality
· Getting Started
∘ Set Up Qdrant
· Data Ingestion
∘ Attribute Selection & Embedding Strategy
∘ Creating Embeddings
∘ Vector Quantization
∘ Payload Indexing
· Querying the Database
∘ Query with Filters
· Bonus: Dynamic Query Filters with a Fine-tuned NER Model
· Wrapping Up
· References

Search Engines Beyond Text

Modern search engines have taken this a step further by letting you search with similar images alongside your text query. Love Google’s Circle to Search? You’re not alone. Multimodal search, which uses different types of data, such as text and images, helps you find exactly what you need faster and with greater accuracy. However, building these systems is no trivial task. Let’s take a closer look using a dataset from the e-commerce website Shein.

1. Heterogeneous Product Data

Each product in the Shein dataset includes a mix of textual descriptions, images, structured attributes (like color, category, and brand), and even price. For example, one row might have:

• Name: “Solid Form Fitted Tee”
• Category: “Tops”
• Price: $6.99
• Image URL: “https://img.ltweb.com/images/2024/08/09/bd_square.png"

This variety demands a multimodal search system that can understand and index all types of content – text, visuals, and structured data. Users can search for anything, from a phrase to a photo.

2. Query Ambiguity

A user might type something like “black dress” or “comfy top.” But what does comfy mean? Cotton? Oversized? Sleeveless? The dataset doesn’t always label products using the same terms as users. This semantic gap makes it difficult to return relevant results unless the system understands intent, not just keywords.

3. Scalability

Even a filtered version of the Shein dataset contains tens of thousands of SKUs, and production systems may have millions. Your search infrastructure must support real-time retrieval at scale. Vector indexes, hybrid search models, and efficient filtering pipelines are critical to ensuring results are delivered in milliseconds.

4. Personalization

Two users typing “floral dress” might want very different things: one prefers boho maxi dresses, another likes short skater styles. To solve this, search engines need to integrate user behavior and preferences in real time, matching results not just to queries but to individuals.

5. Metadata Filtering

E-commerce users often want to refine their search using filters: price range, color, size, material, sleeve length, and more. In the Shein dataset, these are available as structured fields, but integrating them into a seamless faceted search experience without slowing down retrieval is no small feat.

6. Real-Time Requirements

Search engines can’t afford to be slow. Every query must hit a large-scale index, match across multiple data types, apply filters, and optionally re-rank results, all under 200ms, even during peak load.

Solving these challenges requires combining natural language processing, computer vision, vector search, index tuning, and real-time personalization. The Shein dataset, while just a sample, offers a snapshot of the messy, multimodal world a modern e-commerce search engine must navigate, and why doing it well is a serious engineering feat.

Vectors in Search Engines

To deliver smarter, more relevant results, modern e-commerce platforms use vector search or semantic search, a technique that helps search systems understand meaning rather than simply match keywords. This is achieved through vector embeddings..

What Are Vector Embeddings?

At its core, vector search converts product data like titles, descriptions, and images into numerical representations called vector embeddings (or dense vectors). These embeddings capture semantic meaning so that similar items are positioned closer together in a multi-dimensional space.

 Text: "White Floral Dress"
 ↓
 [0.12, 0.75, -0.33, ...]
 ← Dense vector →

For example, imagine embedding several fashion products like dresses, bags, and shoes into a 2D space for visualization:

• The “Red satin evening gown” and “Black sleeveless cocktail dress” cluster together because they’re both formal wear, even though their keywords differ.
• The “White canvas tote bag” and “Beige leather shoulder bag” are placed near each other as they belong to the same category and serve similar functions.
• Footwear like “Chunky white sneakers”, “Black suede ankle boots”, and “Brown leather loafers” form their own cluster, grouped by style and material.

Here’s what that might look like:

In reality, embeddings are not 2-D representations but a high-dimensional array of numbers.

How Images Fit in

Text isn’t the only data type that can be converted into embeddings; images can, too. The key is to use vision–language models like CLIP (Contrastive Language–Image Pretraining), which are trained to map both images and their corresponding descriptions into a shared embedding space.

This means an image of a “red high-heeled shoe” and the phrase “red high-heeled shoe” will be embedded into similar vector representations. When you upload a photo, the system can find its closest matches among product descriptions, enabling reverse image search, similar to Google Lens or Circle to Search.

A common way to compare vectors is cosine similarity, calculated as the dot product of the vectors divided by the product of their magnitudes. The closer the result is to 1, the more similar the vectors are:

C(A,B) = cos(θ) = A.B / ||A|| ||B||

What Are Sparse Vectors?

While dense vectors excel at capturing semantic meaning, they aren’t always ideal for exact keyword matching, especially for short queries or specific terms. This is where sparse vectors come into play.

Sparse vectors are high-dimensional representations where most values are zero, but a few key dimensions are active, corresponding directly to important keywords or tokens. You can think of them as an advanced evolution of traditional keyword-based search methods (like BM25 or TF-IDF), adapted for modern retrieval engines.

Text: "White Floral Dress"
 ↓
[0, 0, 0, 0, 1.2, 0, 0, 0, 0, 0.9, 0, 0, 0, 1.5, 0, 0, 0, 0, 0, 0, ...]
 ↑ ↑ ↑
 "white" "floral" "dress"
 (1.2) (0.9) (1.5)

Product titles like “Nike Air Jordans” and “Adidas Messi F50” are often short and packed with searchable terms. Sparse vector models (such as BM25 or SPLADE) preserve this term-level precision, ensuring that queries return exactly those products, even if they lack rich descriptions.

Sparse vector search is commonly implemented through algorithms like BM25 (Best Match 25). BM25 determines the most relevant documents for a given query by considering two factors:

• Term Frequency (TF): How often do the query words appear in each document? (The more, the better.)
• Inverse Document Frequency (IDF): How rare are the query words across the entire document set? (The rarer, the better.)

The BM25 score for a document D with respect to a query Q is the sum of scores for the individual query terms:

BM25(D, Q) = ∑(IDF(q) * ((TF(q, D) * (k1 + 1)) / (TF(q, D) + k1 * (1 — b + b * (|D| / avgdl)))))

where:

• IDF (q) = inverse document frequency
• TF (q,D) = term frequency
• |D| = document length
• avgdl = average document length
• k1 and b = tunable constants

Payload and Metadata Filtering

In e-commerce, users often search with an intent that goes beyond keywords. For example:

• “Red sneakers under ₹2000”
• “Cotton tops from H&M in size M”

To support such attribute-driven searches, you need more than embeddings; you need metadata filtering. Metadata can include any structured attribute associated with a product, such as:

• Brand: H&M, Nike, Shein
• Color: Black, White, Red
• Category: Dresses, Bags, Shoes
• Price: ₹499, ₹1299
• Size: S, M, L, XL
• Material: Cotton, Polyester, Leather

This structured information is often stored as part of a payload, a key-value dictionary attached to each product in the vector database. While embeddings handle semantic similarity, metadata filtering enforces hard constraints, ensuring users don’t see winter boots when they ask for summer sandals.

For example, in a vector search system, you could retrieve items semantically similar to “white summer dress” (via dense vectors) and then filter by:

brand = SHEIN; price < 1500; size = M

This combination allows for some rule-based precision, thereby reducing the chances of irrelevant results.

Here’s a sample payload in Qdrant format:

{
 "id": "SKU123",
 "vector": [0.12, 0.75, -0.33, ...],
 "payload": {
 "brand": "SHEIN",
 "color": "White",
 "category": "Dress",
 "price": 1299,
 "size": ["M", "L"],
 "material": "Cotton"
 }
}

When searching, you can use filters like these along with the vector search.

filter = {
 "must": [
 {"key": "category", "match": {"value": "Dress"}},
 {"key": "price", "range": {"lt": 1500}},
 {"key": "size", "match": {"value": "M"}}
 ]
}

You’ll see this in action later in the tutorial.

Reranking for Quality

Reranking is an optional but powerful post-processing step in modern search systems. It refines the initial set of results returned by a vector search. While vector similarity can efficiently retrieve the top-k most relevant candidates based on embedding distance, it may not fully capture nuances such as user intent, context, or domain-specific relevance.

Reranking works by taking the top-k results and passing them, along with the original query, into a more powerful model, often a cross-encoder or transformer-based reranker. This model examines the query and each result together (rather than independently) and assigns a relevance score based on deeper semantic understanding.

The candidates are then reordered according to these scores, which usually pushes the most precise and contextually relevant results to the top.

Now that you understand the concepts, let’s move on to building a multimodal search engine. You will use Qdrant as the vector database.

Getting Started

Set Up Qdrant

If you’re working locally, make sure you have Docker installed and the Docker engine running. Qdrant can be installed by pulling its Docker image:

! docker pull qdrant/qdrant

Then run the Qdrant Docker container:

! docker run -p 6333:6333 \
 -v $(pwd)/qdrant_storage:/qdrant/storage \
 qdrant/qdrant

Alternatively, a more convenient option is to use Qdrant Cloud. Log in to the cloud platform, create a cluster, and retrieve your API key.

Install the required Python libraries.

! pip install qdrant-client datasets fastembed transformers qdrant-client[fastembed] openai

Now you are ready to start a client.

from qdrant_client import models, QdrantClient
from google.colab import userdata


client = QdrantClient(
 url="YOUR_QDRANT_CLOUD_INSTANCE_URL",
 api_key=userdata.get('qdrant_api_key'),
)

Data Ingestion

Start by preparing the database. We’ll use the same Shein dataset we discussed earlier.

import pandas as pd
path = "https://raw.githubusercontent.com/luminati-io/eCommerce-dataset-samples/main/shein-products.csv"
df = pd.read_csv(path)

Index(['product_name', 'description', 'initial_price', 'final_price',
 'currency', 'in_stock', 'color', 'size', 'reviews_count', 'main_image',
 'category_url', 'url', 'category_tree', 'country_code', 'domain',
 'image_count', 'image_urls', 'model_number', 'offers',
 'other_attributes', 'product_id', 'rating', 'related_products',
 'root_category', 'top_reviews', 'category', 'brand',
 'all_available_sizes'],
 dtype='object')

You can drop rows that contain null values for certain important fields.

df = df.dropna(subset=['color'])

Another issue seen here is that product descriptions contain the phrase ”Free Returns ✓ Free Shipping✓”. Since this information is irrelevant for search purposes, remove it from all rows. Embeddings should be based on clean, relevant text to ensure accurate retrieval.

df['description'] = df['description'].str.replace('Free Returns ✓ Free Shipping✓.', '', regex=False).str.strip()

For simplicity, only minimal data cleaning is performed here.

Attribute Selection & Embedding Strategy

Once the data is ready, decide which fields to include or exclude, and determine how you’ll generate embeddings and filters to achieve optimal accuracy and efficiency.

The fields containing the most text content and context are best for generating dense vectors. In this dataset, the highest-quality embeddings can be obtained by combining:

product_name + description + category

documents = []
for _, row in df.iterrows():
 doc_text = ""
 if pd.notna(row.get('product_name')):
 doc_text += str(row['product_name'])
 if pd.notna(row.get('description')):
 if doc_text:
 doc_text += " " + str(row['description'])
 else:
 doc_text = str(row['description'])
 if pd.notna(row.get('category')):
 doc_text += " " + str(row['category'])


 documents.append(doc_text)

For images, the dataset includes a main_image column and an image_urls column with additional product images. Download all images locally so they can be used to compute image embeddings later.

import os
import urllib
import pandas as pd
import numpy as np
import json
from typing import Optional


def download_images_for_row(row, base_folder="data/images") -> Optional[str]:
 folder_name = str(row.name)
 folder_path = os.path.join(base_folder, folder_name)
 os.makedirs(folder_path, exist_ok=True)


 urls = []
 # Handle main_image
 main_image = row.get("main_image")
 if isinstance(main_image, str) and main_image.startswith("http"):
 urls.append(main_image)


 # Handle image_urls (should be a stringified list)
 image_urls_raw = row.get("image_urls")
 try:
 image_urls = json.loads(image_urls_raw) if isinstance(image_urls_raw, str) else []
 if isinstance(image_urls, list):
 urls.extend(image_urls)
 except Exception as e:
 print(f"Failed to parse image_urls: {e}")


 # Download images
 for i, url in enumerate(urls):
 try:
 ext = os.path.splitext(url)[-1].split("?")[0]
 filename = f"image_{i}{ext or '.jpg'}"
 filepath = os.path.join(folder_path, filename)
 if not os.path.exists(filepath):
 urllib.request.urlretrieve(url, filepath)
 except Exception as e:
 print(f"Failed to download {url}: {e}")


 if os.listdir(folder_path): # At least one image downloaded
 return folder_path
 return None


# Apply to each row
df["image_folder_path"] = df.apply(download_images_for_row, axis=1)


# Drop rows with failed downloads
df = df.dropna(subset=["image_folder_path"])


# Preview sample
display(df[["main_image", "image_folder_path"]].sample(5).T)

Fields like price, color, size, rating, and brand make excellent metadata attributes. These can often be extracted from a natural language query and applied as filters during search.

Creating Embeddings

To compute embeddings, use fastembed, a Qdrant-developed library that efficiently runs quantized HuggingFace models.

from fastembed import TextEmbedding, LateInteractionTextEmbedding, SparseTextEmbedding, ImageEmbedding

Dense Embeddings

For dense embeddings, you can use common SOTA models like all-MiniLM-L6-v2.

dense_embedding_model = TextEmbedding("sentence-transformers/all-MiniLM-L6-v2")
dense_embeddings = list(dense_embedding_model.embed(doc for doc in documents))

Sparse Embeddings

You can use SPLADE or MiniCOIL for the sparse embeddings. MiniCOIL (mini-Contextualized Inverted Lists) is a sparse neural embedding model for textual retrieval. It generates four-dimensional embeddings for each word stem, capturing the word’s meaning. These meaning embeddings are then combined into a bag-of-words (BoW) representation of the input text. If a word is absent from the vocabulary, its weight is determined solely by its BM25 score. The final sparse representation uses BM25 for term weighting.

minicoil_embedding_model = SparseTextEmbedding("Qdrant/minicoil-v1")
minicoil_embeddings = list(minicoil_embedding_model.embed(doc for doc in documents))

Image Embeddings

In an e-commerce website, product images are typically taken from various angles and positions. Since all these images contain relevant information, you can take an averaging approach to get the best overall representation. However, this approach assumes that all images are equally informative and that none are excessively noisy. For this step, we’ll use the CLIP Vision Transformer model discussed earlier.

clip_embedding_model = ImageEmbedding(model_name="Qdrant/clip-ViT-B-32-vision")


def get_average_image_embedding(folder_path: str) -> Optional[np.ndarray]:
 """Get average embedding of all images in a folder"""
 if not os.path.exists(folder_path):
 return None


 image_files = [f for f in os.listdir(folder_path)
 if f.lower().endswith(('.jpg', '.jpeg', '.png', '.bmp', '.gif', '.webp'))]


 if not image_files:
 return None


 image_paths = [os.path.join(folder_path, f) for f in image_files]


 try:
 # Get embeddings for all images in the folder
 embeddings = list(clip_embedding_model.embed(image_paths))


 if embeddings:
 # Convert to numpy arrays and compute average
 embedding_arrays = [np.array(emb) for emb in embeddings]
 average_embedding = np.mean(embedding_arrays, axis=0)
 return average_embedding


 except Exception as e:
 print(f"Error processing images in {folder_path}: {e}")
 return None


 return None




image_embeddings = []
for _, row in df.iterrows():
 folder_path = row['image_folder_path']
 avg_embedding = get_average_image_embedding(folder_path)
 image_embeddings.append(avg_embedding)


# Filter out None values and keep track of valid indices
valid_indices = [i for i, emb in enumerate(image_embeddings) if emb is not None]
valid_image_embeddings = [image_embeddings[i] for i in valid_indices]

If only the main_imageis relevant, you can use a weighted average where the main_image is given more weight, or simply use the main_imagealone:

weighted_avg = (0.6 * main_emb + 0.2 * side_emb + 0.2 * detail_emb)

Another alternative is to concatenate all embeddings and then reduce the dimensionality using PCA or a projection head.

from sklearn.decomposition import PCA


def get_concatenated_image_embedding(folder_path: str) -> Optional[np.ndarray]:
 """Concatenate embeddings of all images in a folder"""
 if not os.path.exists(folder_path):
 return None


 image_files = [f for f in os.listdir(folder_path)
 if f.lower().endswith(('.jpg', '.jpeg', '.png', '.bmp', '.gif', '.webp'))]


 if not image_files:
 return None


 image_paths = [os.path.join(folder_path, f) for f in image_files]


 try:
 embeddings = list(clip_embedding_model.embed(image_paths))


 if embeddings:
 # Concatenate all embeddings into one long vector
 embedding_arrays = [np.array(emb) for emb in embeddings]
 concatenated_embedding = np.concatenate(embedding_arrays, axis=0)
 return concatenated_embedding


 except Exception as e:
 print(f"Error processing images in {folder_path}: {e}")
 return None


 return None




concatenated_embeddings = []
for _, row in df.iterrows():
 folder_path = row['image_folder_path']
 concat_emb = get_concatenated_image_embedding(folder_path)
 concatenated_embeddings.append(concat_emb)


valid_indices = [i for i, emb in enumerate(concatenated_embeddings) if emb is not None]
valid_concat_embeddings = [concatenated_embeddings[i] for i in valid_indices]


pca = PCA(n_components=512) # Choose your target dimension
reduced_embeddings = pca.fit_transform(valid_concat_embeddings)

Embeddings for Reranking

Finally, compute the embeddings for reranking. Here, we’ll use the popular BERT-based colbertv2.0 model.

late_interaction_embedding_model = LateInteractionTextEmbedding("colbert-ir/colbertv2.0")
late_interaction_embeddings = list(late_interaction_embedding_model.embed(doc for doc in documents))

df["dense_embedding"] = dense_embeddings
df["image_embedding"] = image_embeddings
df["sparse_embedding"] = minicoil_embeddings
df["late_interaction_embedding"] = late_interaction_embeddings

Here’s what the embedding shapes would look like.

Dense embeddings: (384,)
miniCOIL embeddings: () sparse dimensions
Image embeddings: (512,)
Late interaction embeddings: (180, 128)

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Once these are ready, you can proceed to push the data into the vector database. The first step is to create a collection. This should specify the data and dimensions we are pushing, sort of like a blueprint.

from qdrant_client.models import Distance, VectorParams, models
client.recreate_collection(
 "shein_products",
 vectors_config={
 "all-MiniLM-L6-v2": models.VectorParams(
 size=len(dense_embeddings[0]),
 distance=models.Distance.COSINE,
 ),
 "colbertv2.0": models.VectorParams(
 size=len(late_interaction_embeddings[0][0]),
 distance=models.Distance.COSINE,
 multivector_config=models.MultiVectorConfig(
 comparator=models.MultiVectorComparator.MAX_SIM,
 ),
 hnsw_config=models.HnswConfigDiff(m=0) # Disable HNSW for reranking
 ),
 "clip": VectorParams(size=512, distance=Distance.COSINE)
 },
 sparse_vectors_config={
 "minicoil": models.SparseVectorParams(
 modifier=models.Modifier.IDF
 )
 },
 quantization_config=models.ScalarQuantization(
 scalar=models.ScalarQuantizationConfig(
 type=models.ScalarType.INT8,
 quantile=0.99,
 always_ram=True,
 ),
 ),
)

Vector Quantization

As shown in the code, I have used vector quantization to store embeddings more efficiently. As the database grows, there can be constraints on storage space and search latency. Instead of storing each vector as a high-precision float32, quantization converts them into a lower-bit format, commonly int8or even 1-bit binary representations, dramatically reducing memory usage while preserving most of the semantic similarity.

Scalar quantization treats each dimension of the vector independently, mapping its high‑precision floating point value to the nearest bin in a small integer range. This is similar to converting from float32 to int8 for each element. Scalar quantization alone can reduce embedding size by up to 4×, with retrieval performance typically remaining above 99% accuracy.

Payload Indexing

Create a payload index on any metadata field that you expect to filter on regularly. For example, category, brand, price, color, size, or material. This makes filtering much faster and also helps to combine filters with vector search results efficiently.

from qdrant_client.models import PayloadSchemaType

client.create_payload_index(
 collection_name="shein_products",
 field_name="color",
 field_schema=PayloadSchemaType.KEYWORD # keyword (for string match)
)

client.create_payload_index(
 collection_name="shein_products",
 field_name="final_price",
 field_schema=PayloadSchemaType.FLOAT # float (for range queries)
)

client.create_payload_index(
 collection_name="shein_products",
 field_name="category",
 field_schema=models.TextIndexParams(
 type="text",
 tokenizer=models.TokenizerType.WORD,
 min_token_len=2,
 max_token_len=10,
 lowercase=True,
 ),
)


client.create_payload_index("shein_products", "rating", PayloadSchemaType.FLOAT)
client.create_payload_index("shein_products", "brand", PayloadSchemaType.KEYWORD)
client.create_payload_index("shein_products", "category", PayloadSchemaType.KEYWORD)
client.create_payload_index("shein_products", "product_name", PayloadSchemaType.KEYWORD)
client.create_payload_index("shein_products", "currency", PayloadSchemaType.KEYWORD)

Finally, upload the data in batches into the vector database.

from qdrant_client.models import PointStruct, SparseVector, Document

def upload_points_in_batches(df, documents, batch_size=20):
 """Upload points in small batches to avoid payload size limits"""
 # Calculate average length only for documents that correspond to rows in the filtered df
 # This requires mapping back to the original documents
 original_indices = df.index.tolist()
 relevant_documents = [documents[i] for i in original_indices if i < len(documents)]
 avg_documents_length = sum(len(document.split()) for document in relevant_documents) / len(relevant_documents) if relevant_documents else 0

 total_uploaded = 0
 batch_points = []

 # Use enumerate to get a continuous index for accessing documents list
 for enum_idx, (df_idx, row) in enumerate(df.iterrows()):
 if row['image_embedding'] is None:
 continue

 # Use the original dataframe index to access the correct document
 original_doc_idx = df_idx
 if original_doc_idx >= len(documents):
 print(f"Warning: Original index {original_doc_idx} out of bounds for documents list. Skipping.")
 continue
 dense_emb = row['dense_embedding'].tolist() if isinstance(row['dense_embedding'], np.ndarray) else row['dense_embedding']
 late_interaction_emb = row['late_interaction_embedding'].tolist() if isinstance(row['late_interaction_embedding'], np.ndarray) else row['late_interaction_embedding']
 image_emb = row['image_embedding'].tolist() if isinstance(row['image_embedding'], np.ndarray) else row['image_embedding']
 minicoil_doc = Document(
 text=documents[original_doc_idx], # Use original index for the correct document text
 model="Qdrant/minicoil-v1",
 options={"avg_len": avg_documents_length}
 )

 point = PointStruct(
 id=original_doc_idx, # Use the original df index as the point ID
 vector={
 "all-MiniLM-L6-v2": dense_emb,
 "minicoil": minicoil_doc,
 "colbertv2.0": late_interaction_emb,
 "clip": image_emb,
 },
 payload={
 "document": documents[original_doc_idx], # Use original index for payload document
 "product_name": str(row.get('product_name', '')),
 "final_price": float(row.get('final_price', 0)) if pd.notna(row.get('final_price')) else 0.0,
 "currency": str(row.get('currency', ''))[:10],
 "rating": float(row.get('rating', 0)) if pd.notna(row.get('rating')) else 0.0,
 "category": str(row.get('category', ''))[:100],
 "brand": str(row.get('brand', ''))[:100],
 "image_path": str(row.get('main_image', '')),
 "color": str(row.get('color', '')),
 "image_url": str(row.get('main_image', ''))
 }
 )
 batch_points.append(point)
 # Upload when batch is full
 if len(batch_points) >= batch_size:
 client.upsert(collection_name="shein_products", points=batch_points, wait=True) # Added wait=True for robustness
 total_uploaded += len(batch_points)
 print(f"Uploaded batch: {total_uploaded} points")
 batch_points = []

 # Upload remaining points
 if batch_points:
 client.upsert(collection_name="shein_products", points=batch_points, wait=True) 
 total_uploaded += len(batch_points)
 print(f"Final batch uploaded: {total_uploaded} total points")

upload_points_in_batches(df, documents, batch_size=20)

Querying the Database

In a multimodal search engine, users may search using text, images, or a combination of both. Let us see how it’s done.

For a text-only query, we can convert the query into a dense vector and compare it with the dense vectors in the database.

query="Women's running shoes"
dense_vectors = list(dense_embedding_model.query_embed([query]))[0]

 prefetch = [
 models.Prefetch(
 query=dense_vectors,
 using="all-MiniLM-L6-v2",
 limit=limit,
 ),
 models.Prefetch(
 query=models.Document(
 text=query,
 model="Qdrant/minicoil-v1"
 ),
 using="minicoil",
 limit=limit,
 ),
 ]

 results = client.query_points(
 collection_name="shein_products",
 query=dense_vectors,
 prefetch=prefetch,
 with_payload=True,
 limit=limit,
 using="all-MiniLM-L6-v2", 
 )

Score: 0.7146 | Product: Nike Women's Flex Experience Run 11 Next Nature Running Sneakers From Finish Line
Score: 0.6983 | Product: Asics Women's Gel Kayano 30 Running Shoes In Blue Denim
Score: 0.5131 | Product: Women's Glitter Strappy Wrapped Wedge Heel Platform Sandals

For reranking, you need to embed the query using the ColBERT reranking model and pass it to the query_points function.

dense_vectors = list(dense_embedding_model.query_embed([query]))[0]
 late_vectors = list(late_interaction_embedding_model.query_embed([query]))[0]
 prefetch = [
 models.Prefetch(
 query=dense_vectors,
 using="all-MiniLM-L6-v2",
 limit=limit * 2,
 ),
 models.Prefetch(
 query=models.Document(
 text=query,
 model="Qdrant/minicoil-v1"
 ),
 using="minicoil",
 limit=limit * 2,
 ),
 ]

 # Final reranking with late interaction
 results = client.query_points(
 "shein_products",
 prefetch=prefetch,
 query=late_vectors,
 using="colbertv2.0",
 with_payload=True,
 limit=limit,
 )

Score: 26.5242 | Product: Asics Women's Gel Kayano 30 Running Shoes In Blue Denim
Score: 24.9610 | Product: Nike Women's Flex Experience Run 11 Next Nature Running Sneakers From Finish Line
Score: 19.6362 | Product: Nike Jordan 13 Retro Low Bred GS 310811 027

As you can see, performing reranking has eliminated irrelevant results: for example, removing “Platform Sandals” from a query for “Running Shoes.”

Similarly, for an image-based query, you can use:

query_image_path="/content/data/images/2/image_5.jpg"
image_vectors = list(clip_embedding_model.embed([query_image_path]))[0]
 # Direct image similarity search (no prefetch needed)
 results = client.query_points(
 "shein_products",
 query=image_vectors.tolist(),
 using="clip",
 with_payload=True,
 limit=limit,
 )

Score: 0.8669 | Product: 1PC Plus Size Sexy Lingerie Body Stocking Hollow Out See Through Cover Bodystocking Without Underwear Valentine's Day Women's Swimwear & Clothing Swimsuit
Score: 0.8095 | Product: 1PC Punk Women's Sexy Underwear Accessories Adjustable Belt Gothic Body Restraint Device Suitable For Halloween Party Costume Matching
Score: 0.7985 | Product: 1pc Tulle Bow Headpiece Minimalist Wedding Veil Hair Accessory Witch

If both text and image are involved, you can handle the query in the same way, but include the image embeddings in the prefetch step.

query="blue shoes",
query_image_path="/content/data/images/45/image_2.jpg"
dense_vectors = list(dense_embedding_model.query_embed([query]))[0]
image_vectors = list(clip_embedding_model.embed([query_image_path]))[0]
prefetch = [
 models.Prefetch(
 query=dense_vectors,
 using="all-MiniLM-L6-v2",
 limit=limit * 2,
 ),
 models.Prefetch(
 query=models.Document(
 text=query,
 model="Qdrant/minicoil-v1"
 ),
 using="minicoil",
 limit=limit * 2,
 ),
 models.Prefetch(
 query=image_vectors.tolist(),
 using="clip",
 limit=limit * 2,
 ),
]

# Use late interaction embeddings for final reranking
late_vectors = list(late_interaction_embedding_model.query_embed([query]))[0]

results = client.query_points(
 "shein_products",
 prefetch=prefetch,
 query=late_vectors,
 using="colbertv2.0",
 with_payload=True,
 limit=limit,
)

Score: 23.4887 | Product: Asics Women's Gel Kayano 30 Running Shoes In Blue Denim
Score: 16.3749 | Product: Unbeatablesale Jacks 2128F-WH-L Ribbed Bell Boots With Fleece - White, Large
Score: 14.3357 | Product: Nike Jordan 13 Retro Low Bred GS 310811 027

Query with Filters

As discussed earlier, we can use the query to create filters for certain attributes that are not well represented by embeddings. In Qdrant, this is done by applying filters, where additional conditions are set on the payload and the ID of the point using specific clauses.

For example, if we need to filter out a dress with red color under $500, we can set the filter like this:

filter = Filter(
 must=[
 FieldCondition(
 key="category",
 match=MatchValue(value="dress")
 ),
 FieldCondition(
 key="color",
 match=MatchValue(value="red")
 ),
 FieldCondition(
 key="price",
 range=Range(lt=500)
 )
 ]
)

As you can see, we have used clauses such as:

• must: Equivalent to the AND operator, where all conditions inside it must be true.
• match: Matches values exactly with those in the query.
• lt: Stands for “less than.”

You can read more about filter clauses and their combinations here.

A practical issue you might encounter is that static filters don’t make sense for all queries. Instead, we need a way to create dynamic filters based on the query. To achieve this, we can use an LLM to generate the appropriate filter for a given query, which we then pass to the query function.

from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
from openai import OpenAI

openai_client = OpenAI(api_key=userdata.get('OPENAI_APIKEY'))

user_query = "Show me SHEIN women's top handle bags in white under 15 USD"

def get_llm_filters(natural_language_query):
 system_prompt = "You are an assistant that extracts product filters from search queries. Output as JSON."
 user_prompt = f"""
Query: "{natural_language_query}"

Extract structured filters from this product search query. Return the filters in a JSON format.
Use only these allowed fields (all lowercase and exact spelling):
- product_name (string)
- final_price (numeric: supports lt, lte, gt, gte)
- currency (string)
- rating (numeric: supports lt, lte, gt, gte)
- category (string)
- brand (string)

Return a JSON object where:
- string fields are matched exactly (e.g., "category": "tops")
- numeric fields are expressed using comparison operators (e.g., "final_price": {{"lt": 500}})
- only include filters explicitly mentioned or implied in the query, you dont need to include all fields, 2 or 3 is fine

Example:
Input: "show me products from zara under 100 dollars with rating above 4"
Output:
{{
 "brand": "zara",
 "final_price": {{"lt": 100}},
 "rating": {{"gt": 4}}
}}
"""
 response = openai_client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[
 {"role": "system", "content": system_prompt},
 {"role": "user", "content": user_prompt}
 ],
 temperature=0
 )

 import json
 try:
 parsed = json.loads(response.choices[0].message.content)
 return parsed
 except Exception as e:
 print("Failed to parse filter JSON:", e)
 return {}


def convert_llm_filters_to_qdrant(llm_filters: dict) -> Filter:
 conditions = []
 for key, value in llm_filters.items():
 if isinstance(value, str):
 conditions.append(FieldCondition(
 key=key,
 match=MatchValue(value=value)
 ))
 elif isinstance(value, dict):
 range_args = {}
 if 'lt' in value:
 range_args['lt'] = value['lt']
 if 'gt' in value:
 range_args['gt'] = value['gt']
 if 'lte' in value:
 range_args['lte'] = value['lte']
 if 'gte' in value:
 range_args['gte'] = value['gte']

 conditions.append(FieldCondition(
 key=key,
 range=Range(**range_args)
 ))
 return Filter(must=conditions)

llm_filters = get_llm_filters(user_query)
qdrant_filter = convert_llm_filters_to_qdrant(llm_filters)
dense_vectors = list(dense_embedding_model.query_embed([user_query]))[0]

results = client.query_points(
 collection_name="shein_products",
 query=dense_vectors,
 limit=5,
 query_filter=qdrant_filter,
 with_payload=True,
 using="all-MiniLM-L6-v2"
)

Score: 2.6087 | Product: Casual Solid Color Shoulder Bag New Style Handbag Fashion Simple Crossbody Square Bag- Women Top Handle Bags at SHEIN.

You can also have the LLM construct this filter code directly, if possible. Feel free to use whichever approach works best for your setup. Combine all of this into a polished UI, and you’ll have a production-ready search engine ready for deployment.

Bonus: Dynamic Query Filters with a Fine-tuned NER Model

Another approach to building dynamic query filters is to use a Named Entity Recognition (NER) model. By passing user queries through a fine-tuned NER model, you can extract relevant attributes — such as color, category, or price — and automatically generate filter conditions.

In this example, we’ll use NER models from the spaCy library. Start by creating a custom data loader class with sample training data.

import random
import json
from typing import List, Dict, Tuple
import pandas as pd

class SHEINNERDataGenerator:
 """Generate fine-tuning data for NER model based on SHEIN payload fields"""
 def __init__(self):
 # Sample data using only relevant payload fields
 self.sample_products = [
 {
 "product_name": "Women's Casual Floral Print Midi Dress",
 "final_price": 25.99,
 "currency": "USD",
 "rating": 4.3,
 "category": "Dresses",
 "brand": "SHEIN"
 },
 {
 "product_name": "Men's Basic Cotton T-Shirt Black",
 "final_price": 8.50,
 "currency": "USD",
 "rating": 4.1,
 "category": "Tops",
 "brand": "SHEIN"
 },
 {
 "product_name": "Plus Size High Waist Skinny Jeans Blue",
 "final_price": 32.00,
 "currency": "USD",
 "rating": 4.5,
 "category": "Bottoms",
 "brand": "SHEIN"
 },
 {
 "product_name": "Women's Chunky Knit Oversized Sweater Pink",
 "final_price": 29.99,
 "currency": "USD",
 "rating": 4.2,
 "category": "Sweaters",
 "brand": "SHEIN"
 },
 {
 "product_name": "Summer Beach Sandals White Platform",
 "final_price": 18.75,
 "currency": "USD",
 "rating": 3.9,
 "category": "Shoes",
 "brand": "SHEIN"
 }
 ]

 self.colors = [
 "black", "white", "red", "blue", "green", "yellow", "pink", "purple",
 "orange", "brown", "gray", "grey", "navy", "beige", "khaki", "maroon",
 "coral", "mint", "turquoise", "burgundy", "olive", "cream", "ivory",
 "gold", "silver", "rose gold", "neon", "pastel", "dark blue", "light blue",
 "dark green", "light green", "bright red", "deep red", "hot pink", "lime",
 "lavender", "peach", "teal", "magenta", "charcoal", "nude", "camel", "tan"
 ]

 self.categories = [
 "dress", "dresses", "midi dress", "maxi dress", "mini dress", "bodycon dress",
 "shirt", "top", "blouse", "t-shirt", "tank top", "crop top", "tube top", "camisole",
 "pants", "jeans", "trousers", "leggings", "joggers", "sweatpants", "cargo pants",
 "skirt", "mini skirt", "midi skirt", "maxi skirt", "pleated skirt", "pencil skirt",
 "jacket", "blazer", "cardigan", "coat", "hoodie", "sweater", "pullover", "vest",
 "shoes", "boots", "sneakers", "heels", "flats", "sandals", "pumps", "loafers",
 "bag", "purse", "backpack", "tote", "clutch", "crossbody", "handbag",
 "accessories", "jewelry", "necklace", "earrings", "bracelet", "ring", "watch",
 "swimwear", "bikini", "swimsuit", "beachwear", "lingerie", "bra", "underwear"
 ]

 self.price_phrases = [
 "under", "below", "less than", "maximum", "max", "up to", "within",
 "over", "above", "more than", "minimum", "min", "starting from", "at least",
 "between", "from", "to", "around", "approximately", "roughly"
 ]

 self.rating_phrases = [
 "highly rated", "top rated", "best rated", "good reviews", "well reviewed",
 "above", "over", "more than", "at least", "minimum", "4+ stars", "5 star"
 ]

 self.sizes = ["XS", "S", "M", "L", "XL", "XXL", "plus size", "petite", "tall"]

 self.styles = [
 "casual", "formal", "business", "party", "evening", "summer", "winter",
 "vintage", "boho", "gothic", "street", "preppy", "minimalist", "trendy",
 "oversized", "fitted", "loose", "tight", "flowy", "structured"
 ]


 def generate_training_queries(self, num_samples: int = 100) -> List[Tuple[str, Dict]]:
 training_data = []
 templates = [
 "{color} {category} under ${price}",
 "{brand} {category} with {rating_phrase}",
 "{style} {color} {category} from {brand}",
 "Looking for {style} {category} in {color} with rating above {rating}",
 "{brand} {category} between ${price_min} and ${price_max}",
 "Top rated {color} {category} from {brand}",
 "{category} with rating {rating}+ stars",
 "Cheap {color} {category} under ${price}"
 ]
 for _ in range(num_samples):
 vars_used = {
 "color": random.choice(self.colors),
 "category": random.choice(self.categories),
 "brand": "SHEIN",
 "style": random.choice(self.styles),
 "price": round(random.uniform(10, 100), 2),
 "price_min": round(random.uniform(10, 40), 2),
 "price_max": round(random.uniform(41, 100), 2),
 "rating": round(random.uniform(3.0, 5.0), 1),
 "rating_phrase": random.choice(self.rating_phrases),
 }
 template = random.choice(templates)
 try:
 query = template.format(**vars_used)
 entities = self._extract_entities_from_query(query, vars_used)
 training_data.append((query, {"entities": entities}))
 except KeyError:
 continue

 return training_data


 def generate_realistic_queries_from_products(self, products_sample: List[Dict] = None) -> List[Tuple[str, Dict]]:
 if products_sample is None:
 products_sample = self.sample_products
 training_data = []
 for row in products_sample:
 product = {
 "product_name": str(row.get('product_name', '')),
 "final_price": float(row.get('final_price', 0)) if pd.notna(row.get('final_price')) else 0.0,
 "currency": str(row.get('currency', ''))[:10],
 "rating": float(row.get('rating', 0)) if pd.notna(row.get('rating')) else 0.0,
 "category": str(row.get('category', ''))[:100].lower(),
 "brand": str(row.get('brand', ''))[:100]
 }
 queries = [
 f"Looking for {product['category']} under ${product['final_price'] + 10}",
 f"{product['category']} with rating above {product['rating'] - 0.5}",
 f"{product['brand']} {product['category']} with good reviews"
 ]
 for color in self.colors:
 if color in product["product_name"].lower():
 queries.append(f"{color} {product['category']} under ${product['final_price'] + 5}")
 for query in queries:
 entities = self._extract_entities_from_query(query, {})
 if entities:
 training_data.append((query, {"entities": entities}))

 return training_data


 def _extract_entities_from_query(self, query: str, vars_used: Dict) -> List[Tuple[int, int, str]]:
 query_lower = query.lower()
 entities = []
 for color in self.colors:
 if color in query_lower:
 idx = query_lower.find(color)
 entities.append((idx, idx + len(color), "COLOR"))
 for category in self.categories:
 if category in query_lower:
 idx = query_lower.find(category)
 entities.append((idx, idx + len(category), "PRODUCT_CATEGORY"))

 if "shein" in query_lower:
 idx = query_lower.find("shein")
 entities.append((idx, idx + 5, "BRAND"))

 for style in self.styles:
 if style in query_lower:
 idx = query_lower.find(style)
 entities.append((idx, idx + len(style), "STYLE"))

 return self._remove_overlapping_entities(entities)


 def _remove_overlapping_entities(self, entities: List[Tuple[int, int, str]]) -> List[Tuple[int, int, str]]:
 entities.sort(key=lambda x: x[0])
 non_overlap = [entities[0]] if entities else []
 for entity in entities[1:]:
 if entity[0] >= non_overlap[-1][1]:
 non_overlap.append(entity)
 return non_overlap


 def save_training_data(self, training_data: List[Tuple[str, Dict]], filename: str):
 with open(filename, 'w') as f:
 json.dump(training_data, f, indent=2)
 print(f"Saved to {filename}")


 def print_sample_data(self, training_data: List[Tuple[str, Dict]], count: int = 10):
 for i, (text, annotations) in enumerate(training_data[:count]):
 print(f"\nExample {i+1}: {text}")
 for start, end, label in annotations["entities"]:
 print(f" {label}: '{text[start:end]}'")

Here’s a peek into how the NER model identifies entities in a query.

(
 "Show me red dresses from SHEIN under $500",
 {
 "entities": [
 (8, 11, "COLOR"), # "red"
 (12, 19, "PRODUCT_CATEGORY"), # "dresses"
 (25, 30, "BRAND"), # "SHEIN"
 (37, 41, "PRICE") # "500"
 ]
 }
)

Each training sample is a tuple containing a query string and a dictionary with entity annotations in the spaCy NER format. Use this script to train the model with the provided data.

import spacy
from spacy.training import Example
from spacy.util import minibatch, compounding
import json
import random
from pathlib import Path
from typing import List, Dict, Tuple, Optional

class EnhancedNERTrainer:
 """NER trainer for SHEIN product queries using SHEINNERDataGenerator"""

 def __init__(self, model_name: str = "en_core_web_sm"):
 try:
 self.nlp = spacy.load(model_name)
 except OSError:
 print(f"Model {model_name} not found. Installing...")
 spacy.cli.download(model_name)
 self.nlp = spacy.load(model_name)

 self.training_nlp = spacy.blank("en")

 if "ner" not in self.training_nlp.pipe_names:
 self.ner = self.training_nlp.add_pipe("ner")
 else:
 self.ner = self.training_nlp.get_pipe("ner")

 self.entity_labels = [
 "COLOR",
 "PRODUCT_CATEGORY",
 "BRAND",
 "SIZE",
 "STYLE",
 "PRICE",
 "RATING",
 ]

 for label in self.entity_labels:
 self.ner.add_label(label)


 def load_training_data_from_generator(self, generator: 'SHEINNERDataGenerator', num_synthetic: int = 300) -> List[Tuple[str, Dict]]:
 synthetic_data = generator.generate_training_queries(num_synthetic)
 realistic_data = generator.generate_realistic_queries_from_products()
 all_data = list({q[0]: q for q in (synthetic_data + realistic_data)}.values())
 print(f"Generated {len(all_data)} unique training examples from generator")
 return all_data


 def train_model(self, training_data: List[Tuple[str, Dict]],
 model_output_path: str = "custom_shein_ner",
 iterations: int = 50,
 dropout: float = 0.35) -> spacy.Language:

 print(f"Training NER model with {len(training_data)} examples...")
 examples = [Example.from_dict(self.training_nlp.make_doc(text), annotations) for text, annotations in training_data]
 self.training_nlp.initialize(lambda: examples)

 for iteration in range(iterations):
 random.shuffle(examples)
 losses = {}
 batches = minibatch(examples, size=compounding(4.0, 32.0, 1.001))
 for batch in batches:
 self.training_nlp.update(batch, losses=losses, drop=dropout)
 if iteration % 10 == 0:
 print(f"Iteration {iteration:3d} - Loss: {losses.get('ner', 0.0):.4f}")

 Path(model_output_path).mkdir(exist_ok=True)
 self.training_nlp.to_disk(model_output_path)
 print(f"Model saved to {model_output_path}")
 return self.training_nlp


 def evaluate_model(self, test_data: List[Tuple[str, Dict]], model_path: str = None) -> Dict:
 nlp = spacy.load(model_path) if model_path else self.training_nlp
 correct, total = 0, 0
 precision_scores, recall_scores = [], []
 for text, annotations in test_data:
 doc = nlp(text)
 predicted = {(ent.start_char, ent.end_char, ent.label_) for ent in doc.ents}
 true = set(annotations["entities"])
 if predicted:
 precision_scores.append(len(predicted & true) / len(predicted))
 if true:
 recall_scores.append(len(predicted & true) / len(true))
 if predicted == true:
 correct += 1
 total += 1
 accuracy = correct / total if total else 0
 precision = sum(precision_scores) / len(precision_scores) if precision_scores else 0
 recall = sum(recall_scores) / len(recall_scores) if recall_scores else 0
 f1 = 2 * precision * recall / (precision + recall) if precision + recall > 0 else 0
 print("Evaluation Results:")
 print(f"Accuracy: {accuracy:.4f}\nPrecision: {precision:.4f}\nRecall: {recall:.4f}\nF1 Score: {f1:.4f}")
 return {
 "accuracy": accuracy,
 "precision": precision,
 "recall": recall,
 "f1_score": f1,
 "total_examples": total
 }


 def test_model_on_queries(self, model_path: str, test_queries: List[str]):
 nlp = spacy.load(model_path)
 print("\nTesting model on sample queries:\n" + "=" * 80)
 for i, query in enumerate(test_queries, 1):
 doc = nlp(query)
 print(f"\nQuery {i}: {query}")
 if doc.ents:
 for ent in doc.ents:
 print(f" Entity: '{ent.text}' -> {ent.label_}")
 else:
 print(" No entities found")


def create_comprehensive_training_pipeline():
 trainer = EnhancedNERTrainer()
 generator = SHEINNERDataGenerator()
 training_data = trainer.load_training_data_from_generator(generator, num_synthetic=400)
 random.shuffle(training_data)
 split = int(0.8 * len(training_data))
 train_data, test_data = training_data[:split], training_data[split:]
 print(f"Train: {len(train_data)} | Test: {len(test_data)}")
 trainer.train_model(train_data, iterations=50)
 metrics = trainer.evaluate_model(test_data)
 queries = [
 "red summer dress under $40",
 "SHEIN black skinny jeans with 4+ stars",
 "casual white cotton t-shirt from SHEIN",
 "looking for plus-size formal dress below $60"
 ]
 trainer.test_model_on_queries("custom_shein_ner", queries)
 return trainer, metrics

You can now deploy this trained model in your pipeline to extract entities, construct filters, and apply them during querying.

from qdrant_client import models
import spacy
from typing import Dict, Optional
ner_model = spacy.load("custom_shein_ner")

# Helper to convert extracted entity list into a dict
def extract_ner_payload(query: str) -> Dict[str, str]:
 doc = ner_model(query)
 print("OUTPUT: ", doc)
 payload = {
 "product_name": None,
 "final_price": None,
 "currency": None,
 "rating": None,
 "category": None,
 "brand": None,
 }

 for ent in doc.ents:
 if ent.label_ == "PRICE":
 payload["final_price"] = ent.text.replace("$", "").strip()
 elif ent.label_ == "RATING":
 payload["rating"] = ent.text.replace("stars", "").strip()
 elif ent.label_ == "PRODUCT_CATEGORY":
 payload["category"] = ent.text.strip().lower()
 elif ent.label_ == "BRAND":
 payload["brand"] = ent.text.strip()
 elif ent.label_ == "STYLE":
 payload["product_name"] = ent.text.strip()
 elif ent.label_ == "CURRENCY":
 payload["currency"] = ent.text.strip().upper()

 return {k: v for k, v in payload.items() if v is not None}


# Convert NER payload to Qdrant filter
def ner_payload_to_qdrant_filter(payload: Dict[str, str]) -> Optional[models.Filter]:
 conditions = []
 if "final_price" in payload:
 try:
 price_value = float(payload["final_price"])
 conditions.append(models.FieldCondition(
 key="final_price",
 range=models.Range(lte=price_value)
 ))
 except ValueError:
 pass
 if "rating" in payload:
 try:
 rating_value = float(payload["rating"])
 conditions.append(models.FieldCondition(
 key="rating",
 range=models.Range(gte=rating_value)
 ))
 except ValueError:
 pass
 if "category" in payload:
 conditions.append(models.FieldCondition(
 key="category",
 match=models.MatchValue(value=payload["category"])
 ))
 if "brand" in payload:
 conditions.append(models.FieldCondition(
 key="brand",
 match=models.MatchValue(value=payload["brand"])
 ))
 if "currency" in payload:
 conditions.append(models.FieldCondition(
 key="currency",
 match=models.MatchValue(value=payload["currency"])
 ))

 return models.Filter(must=conditions) if conditions else None

# Example Usage
query = "Red Zara dress under $500 with 4.5 stars"
limit = 10

# 1. Extract dynamic filters
ner_payload = extract_ner_payload(query)
qdrant_filter = ner_payload_to_qdrant_filter(ner_payload)
print("FILTER: ", qdrant_filter)

dense_vectors = list(dense_embedding_model.query_embed([query]))[0]
prefetch = [
 models.Prefetch(
 query=dense_vectors,
 using="all-MiniLM-L6-v2",
 limit=limit * 2,
 ),
 models.Prefetch(
 query=models.Document(
 text=query,
 model="Qdrant/minicoil-v1"
 ),
 using="minicoil",
 limit=limit * 2,
 ),
]

results = client.query_points(
 collection_name="shein_products",
 query=dense_vectors,
 with_payload=True,
 limit=limit,
 query_filter=qdrant_filter,
 using="all-MiniLM-L6-v2",
)

Wrapping Up

In this article, you learned how multimodal search engines work behind the scenes and how your favorite shopping apps subtly guide your choices through intelligent search and ranking systems. You also explored how to build one using modern vector databases like Qdrant. Find the complete code on Colab or GitHub.

Multimodal search bridges the gap between how users think and how databases respond, combining the strengths of text, image, and metadata-based search.

References

[1] SuperLinked, Ashish Abraham(2024).Optimizing Rag with Hybrid-search & Reranking

[2] AzureAI(2025).Compress Vectors using Scalar or Binary Quantization

[3] Qdrant Docs (2025)

[4] HuggingFace Docs (2025)

Images

If not otherwise stated, all images are created by the author.

Enjoyed This Article?

💖Hit follow and stay tuned for more deep dives! Let’s connect on LinkedIn — I’d love to chat!

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI