In the previous article, we learned about AI embeddings and how text can be converted into numerical vectors.
But this creates an important question:
Where do we store all these vectors? How do we search millions of vectors efficiently?
This is where Vector Databases come into the picture.
Vector databases are an important component of modern AI applications, especially RAG (Retrieval-Augmented Generation) systems.
In this beginner-friendly guide, we will learn what a vector database is, how vector search works, similarity search, Top-K retrieval, metadata filtering, indexing, HNSW, hybrid search, and how vector databases are used in RAG applications.
What Is a Vector Database?
A Vector Database is a database designed to store, index, and search vector embeddings.
Instead of searching only for exact words or values, a vector database can search for vectors that are mathematically similar to a query vector.
Text ↓ Embedding Model ↓ Vector ↓ Vector Database ↓ Similarity Search ↓ Relevant Information
In a RAG system, a vector database is commonly used to store document chunks and their embeddings so that relevant information can be retrieved when a user asks a question.
Why Do We Need a Vector Database?
Imagine your company has 100,000 documents.
After chunking, those documents might produce millions of individual text chunks.
100,000 Documents
↓
Document Chunking
↓
Millions of Chunks
↓
Millions of Embeddings
If a user asks:
"How many vacation days do employees get?"
The application needs to find the most relevant chunks quickly.
Searching every vector one by one would become expensive and slow as the dataset grows.
Vector databases use specialized indexing and search techniques to make similarity search much more efficient.
Traditional Database vs Vector Database
A traditional relational database is excellent for structured data.
SELECT * FROM Employees WHERE Department = 'IT';
This query looks for an exact condition.
A vector database is designed for a different type of question:
"Find documents that are semantically similar to this question."
| Traditional Database | Vector Database |
|---|---|
| Structured data | Vector embeddings |
| Exact conditions | Similarity search |
| SQL queries | Vector queries |
| Rows and columns | Vectors and metadata |
| Excellent for transactions | Excellent for semantic retrieval |
What Does a Vector Database Store?
A vector database usually stores more than just a vector.
A typical record might contain:
ID: DOC-001-CHUNK-10 Vector: [0.12, -0.43, 0.71, 0.09, ...] Text: Employees receive 20 days of annual leave. Metadata: Document = LeavePolicy.pdf Page = 12 Department = HR Year = 2026
The vector is used for similarity search, while the text and metadata help the application understand and use the search result.
How Does Vector Search Work?
The basic process is straightforward.
User Question
↓
Embedding Model
↓
Query Vector
↓
Vector Database
↓
Similarity Search
↓
Relevant Vectors
↓
Original Text
↓
LLM
Let's look at each step.
Step 1: Create Document Embeddings
First, documents are divided into chunks.
Document ↓ Chunk 1 Chunk 2 Chunk 3 Chunk 4
Each chunk is converted into an embedding.
Chunk 1 → Vector 1 Chunk 2 → Vector 2 Chunk 3 → Vector 3 Chunk 4 → Vector 4
These vectors are stored in the vector database.
Step 2: User Asks a Question
The user asks:
"How many vacation days can I take?"
The question is converted into an embedding.
Question ↓ Embedding Model ↓ Query Vector
Step 3: Search the Vector Database
The query vector is compared against the stored vectors.
Query Vector
↓
Vector Database
↓
Similarity Search
↓
Relevant Vectors
The database returns the vectors that are closest or most similar according to the selected similarity or distance metric.
Step 4: Retrieve the Original Text
The vector search result points back to the corresponding document chunks.
Vector ↓ Document ID ↓ Chunk ↓ Original Text
For example:
Employees receive 20 days of paid annual leave.
Step 5: Send Context to the LLM
The retrieved information is provided to the LLM along with the user's question.
Question + Retrieved Context ↓ LLM ↓ Answer
The LLM can then generate a response based on the retrieved information.
What Is Similarity Search?
Similarity Search is the process of finding vectors that are closest to a query vector according to a chosen similarity or distance measure.
For example:
Query Vector
↓
┌───────────────┐
│ Vector DB │
│ │
│ Vector A ✓ │
│ Vector B ✓ │
│ Vector C │
│ Vector D │
│ Vector E ✓ │
└───────────────┘
↓
Top Relevant Results
The search engine ranks the results based on their similarity to the query.
Common Similarity Measures
Vector databases can use different mathematical measures for comparing vectors.
- Cosine Similarity
- Dot Product
- Euclidean Distance
The appropriate metric depends on the embedding model and the vector database configuration.
Cosine Similarity
Cosine similarity compares the angle between two vectors.
Vector A
↘
↘
↘
↘ Vector B
Smaller angle
↓
Greater similarity
With the common cosine-similarity convention, a value closer to 1 generally indicates greater similarity, while values closer to 0 indicate weaker similarity.
What Is Top-K Search?
A vector search does not normally return every matching vector.
Instead, the application usually requests the Top-K results.
If K = 5:
Query ↓ Vector Search ↓ Top 5 Results ↓ LLM
For example:
Result 1 → Similarity: 0.94 Result 2 → Similarity: 0.91 Result 3 → Similarity: 0.88 Result 4 → Similarity: 0.84 Result 5 → Similarity: 0.81
Only these results may be passed to the next stage of the RAG pipeline.
What Is Vector Indexing?
If a vector database has millions of vectors, comparing a query against every single vector can be expensive.
Vector indexing creates a data structure that helps the system search the vector space more efficiently.
Millions of Vectors
↓
Vector Index
↓
Efficient Search
↓
Relevant Vectors
This is similar to how traditional databases use indexes to speed up certain types of queries.
What Is HNSW?
HNSW stands for Hierarchical Navigable Small World.
It is a popular graph-based indexing technique used for approximate nearest-neighbor search.
Instead of comparing a query with every vector, HNSW builds a graph that helps navigate toward nearby vectors.
Vector A
/ \
/ \
Vector B Vector C
| |
Vector D Vector E
\ /
\ /
Vector F
The search can navigate through this graph to find good nearest neighbors efficiently.
Exact Search vs Approximate Search
Exact Nearest-Neighbor Search
The system compares the query against every vector to determine the exact nearest neighbors.
Query ↓ Compare with ALL vectors ↓ Find exact nearest neighbors
This can become expensive with very large datasets.
Approximate Nearest-Neighbor Search
The system uses an index to find highly relevant neighbors without exhaustively comparing every vector.
Query ↓ Vector Index ↓ Search likely candidates ↓ Top results
This is often much faster for large-scale vector search.
What Is Metadata Filtering?
Vector similarity is not always enough.
Suppose a company has documents from several countries:
India USA UK Canada Australia
A user asks about an HR policy for India.
The application could apply a metadata filter before or during retrieval, depending on the vector database.
User Question
↓
Metadata Filter
Country = India
↓
Vector Search
↓
Relevant Documents
Metadata might include:
- Country
- Department
- Document type
- Year
- Product
- Customer
- Access level
Why Metadata Is Important
Metadata can improve both relevance and security.
For example:
Department = Finance Year = 2026 DocumentType = Policy
The search can be restricted to documents matching those conditions.
What Is Hybrid Search?
Hybrid Search combines multiple retrieval approaches.
A common approach combines:
- Keyword search
- Vector search
User Query
↓
┌──────────┴──────────┐
↓ ↓
Keyword Search Vector Search
↓ ↓
└──────────┬──────────┘
↓
Combined Results
↓
LLM
Hybrid search can be particularly useful when exact terms matter.
For example:
Error Code: DB-1024 Product ID: POS-7000 API: PaymentAuthorization
Keyword matching can help with exact identifiers, while vector search can help with natural-language meaning.
Vector Database in a RAG System
Now let's connect everything together.
DOCUMENTS
↓
Chunking
↓
Embedding Model
↓
Vectors
↓
┌───────────────┐
│ Vector │
│ Database │
└───────┬───────┘
│
│
↓
User Question → Embedding
↓
Vector Search
↓
Relevant Chunks
↓
LLM
↓
Answer
Example: Employee Knowledge Base
Imagine a company has the following documents:
Employee Handbook.pdf Leave Policy.pdf Travel Policy.pdf Insurance Policy.pdf
During indexing:
PDF ↓ Extract Text ↓ Chunk ↓ Generate Embeddings ↓ Store in Vector Database
Now an employee asks:
"Can I carry my unused leave into next year?"
The application creates a query embedding and searches the vector database.
Question ↓ Query Embedding ↓ Vector Search ↓ Leave Policy Chunk ↓ LLM ↓ Answer
Vector Database vs Search Engine
Modern search systems can support both keyword and vector retrieval.
The choice between a dedicated vector database, a search engine with vector capabilities, or a traditional database with vector support depends on the application's requirements.
| Requirement | Possible Approach |
|---|---|
| Structured transactional data | Relational database |
| Semantic vector search | Vector database |
| Keyword + semantic search | Hybrid search platform |
| Existing database infrastructure | Database with vector support |
There is no single database that is best for every RAG application.
Can SQL Databases Store Vectors?
Some modern database systems support storing and searching vectors.
This means you do not always need a separate vector database.
Application
↓
SQL Database
┌───────────────┐
│ Normal Data │
│ Vector Data │
│ Metadata │
└───────────────┘
This can simplify architecture when an organization already has a database platform capable of handling the required vector workloads.
Popular Vector Database Technologies
Several technologies can be used for vector storage and retrieval.
- Qdrant
- Milvus
- Weaviate
- Pinecone
- Chroma
- FAISS
- PostgreSQL with vector extensions
- Other databases and search platforms with vector capabilities
The right choice depends on factors such as scale, deployment model, filtering requirements, performance, ecosystem, and cost.
Vector Database vs FAISS
FAISS is a library for efficient similarity search and clustering of dense vectors.
A vector database generally provides a broader database-oriented feature set, such as:
- Persistent storage
- Metadata management
- Filtering
- APIs
- Index management
- Scalability features
- Database operations
FAISS can be very useful when you need a similarity-search library rather than a complete database system.
How Many Vectors Can a Vector Database Store?
There is no universal limit.
Capacity depends on:
- Vector dimensions
- Number of vectors
- Index type
- Available memory
- Storage
- Database implementation
- Hardware
- Distributed architecture
A small application may have thousands of vectors, while a large system may have millions or billions.
Why Vector Dimension Matters
Suppose an embedding contains 384 dimensions:
[x1, x2, x3, ... x384]
Another model might produce 1536 dimensions:
[x1, x2, x3, ... x1536]
Higher dimensional vectors generally require more memory and computational resources.
However, higher dimensionality does not automatically mean better retrieval quality.
Vector Search Performance
Several factors influence vector search performance:
- Number of vectors
- Vector dimensions
- Index type
- Hardware
- Number of concurrent queries
- Metadata filtering
- Top-K value
- Search configuration
For larger systems, indexing and infrastructure design become increasingly important.
What Happens When Documents Are Updated?
Suppose the company updates its leave policy.
The application can process the updated document and replace or update the corresponding chunks in the vector database.
Updated Document
↓
Extract Text
↓
Chunk
↓
Generate Embeddings
↓
Update Vector Database
↓
New Information Available
The LLM itself does not need to be retrained simply because the source document changed.
Vector Database Security
Security is an important consideration when building enterprise RAG applications.
Vector databases may contain sensitive information or references to sensitive documents.
Applications should consider:
- Authentication
- Authorization
- Tenant isolation
- Encryption
- Access-controlled retrieval
- Metadata filtering
- Audit logging
A user should only be able to retrieve information they are authorized to access.
Simple RAG Example
Let's summarize the entire process with a simple example.
Document:
"Employees receive 20 days of paid annual leave."
Step 1 – Create embedding:
Document ↓ Embedding Model ↓ Document Vector
Step 2 – Store it:
Vector + Text + Metadata ↓ Vector Database
Step 3 – User asks:
"How many vacation days do I get?"
Step 4 – Create query embedding:
Question ↓ Embedding Model ↓ Query Vector
Step 5 – Search:
Query Vector
↓
Vector Database
↓
Similarity Search
↓
Relevant Chunk
Step 6 – Generate answer:
Relevant Chunk
+
User Question
↓
LLM
↓
"Employees receive 20 days
of paid annual leave."
Key Takeaways
- Vector Database: Stores and searches vector embeddings.
- Vector: Numerical representation of information.
- Similarity Search: Finds vectors that are close to a query according to a chosen metric.
- Top-K: Retrieves the most relevant results.
- Vector Index: Helps make large-scale vector search more efficient.
- HNSW: A popular approximate nearest-neighbor indexing technique.
- Metadata Filtering: Restricts results using additional attributes.
- Hybrid Search: Combines keyword and semantic retrieval.
- RAG: Uses vector retrieval to provide relevant information to an LLM.
Conclusion
A Vector Database is an important building block for modern AI applications.
It allows applications to store embeddings and efficiently search for information based on semantic similarity.
The basic flow is:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
User Question
↓
Query Embedding
↓
Similarity Search
↓
Relevant Information
↓
LLM
↓
Answer
Once you understand embeddings and vector databases, the next major concept is semantic search — how AI can search for information based on meaning rather than exact keywords.
