Why Do We Need Vector Databases?#

Traditional databases excel at “exact matching”: WHERE id = 42 or WHERE name = '张三'. But the most common need in AI applications is precisely the fuzzy kind—“find the passages that are closest in meaning to this sentence.”

For example:

• A user asks “How do I get a refund?”, but the knowledge base stores “How to apply for a return”—the wording barely overlaps, yet semantically they mean the same thing.
• Given an image, you want to find other images that “look the most similar.”

This kind of “semantic similarity” retrieval cannot be solved directly with SQL’s LIKE or an inverted index. Vector databases were born for exactly this scenario: they convert text, images, audio, and other content into high-dimensional vectors, then specialize in “quickly finding the batch of vectors most similar to a query vector.”

In a RAG (Retrieval-Augmented Generation) architecture, the vector database is a key link that bridges what comes before and after—upstream it receives vectors produced by Embedding models, and downstream it provides retrieval results to the LLM. Understanding it means understanding half of RAG.

Vectors and Embedding#

A vector is essentially an array of floating-point numbers, such as [0.12, -0.83, 0.45, ...]. Embedding is the process of “turning content into a vector,” and its core idea is: content with similar semantics is also close together in vector space.

1
"cat"        → [0.92, 0.11, -0.03, ...]
2
"kitty"      → [0.89, 0.15, -0.01, ...]   ← very close to "cat"
3
"car engine"   → [-0.41, 0.78, 0.62, ...]   ← very far from "cat"

Real production environments typically use Transformer models for Embedding, such as OpenAI’s text-embedding-3-small, BAAI’s open-source model bge-*, Sentence-Transformers’ all-MiniLM-L6-v2, and so on. These models can capture deep semantic relationships, not just keyword overlap.

To let you see clearly what a vector database is doing internally, below we’ll first hand-write a “minimal viable” vector database using pure NumPy.

Similarity Metrics#

Before getting hands-on, let’s first understand how to measure whether two vectors “look alike.” The three most commonly used metrics:

1
import numpy as np
2

3
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
4
    """Cosine similarity = dot product / (product of magnitudes)"""
5
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

Implementing a MiniVectorDB from Scratch#

Below, in under 60 lines of code, we’ll implement a minimal vector database capable of “writing vectors + semantic retrieval.” To be fully self-contained (without relying on any external API), we use a Feature Hashing trick to generate simple vectors: hash the character n-grams of the text, map them to a fixed dimension, and let the sign (positive or negative) be determined by the parity of the hash value.

Note: This is only a teaching demonstration. Real semantic retrieval should use Transformer Embedding models, not hashing tricks.

1
import hashlib
2
import numpy as np
3

4

5
def simple_embed(text: str, dim: int = 128) -> np.ndarray:
6
    """
7
    Simple Embedding: generate vectors using character-level n-gram hashing, no external API needed.
8
    For teaching demonstration only; replace with a Transformer Embedding model in real projects.
9
    """
10
    vec = np.zeros(dim, dtype=np.float32)
11
    text = text.lower()
12
    #1-gram and 2-gram
13
    tokens = [text[i:i + 1] for i in range(len(text))]
14
    tokens += [text[i:i + 2] for i in range(len(text) - 1)]
15
    for token in tokens:
16
        h = int(hashlib.md5(token.encode("utf-8")).hexdigest(), 16)
17
        idx = h % dim
18
        sign = 1.0 if (h // dim) % 2 == 0 else -1.0
19
        vec[idx] += sign
20
    norm = np.linalg.norm(vec)
21
    return vec / norm if norm > 0 else vec  #L2 normalization, then the dot product is the cosine similarity
22

23

24
class MiniVectorDB:
25
    """Minimal vector database: brute-force retrieval of top-k similar vectors"""
26

27
    def __init__(self, dim: int = 128):
28
        self.dim = dim
29
        self.vectors: list[np.ndarray] = []
30
        self.docs: list[dict] = []
31

32
    def add(self, doc_id: str, text: str) -> None:
33
        self.vectors.append(simple_embed(text, self.dim))
34
        self.docs.append({"id": doc_id, "text": text})
35

36
    def search(self, query: str, top_k: int = 3) -> list[tuple]:
37
        """Returns a list of (doc_id, similarity score, text), sorted by similarity in descending order"""
38
        q = simple_embed(query, self.dim)
39
        #Vectors are already normalized, so the dot product is the cosine similarity
40
        scores = [float(np.dot(q, v)) for v in self.vectors]
41
        order = np.argsort(scores)[::-1][:top_k]  #Sort descending and take top-k
42
        return [
43
            (self.docs[i]["id"], scores[i], self.docs[i]["text"])
44
            for i in order
45
        ]
46

47

48
if __name__ == "__main__":
49
    db = MiniVectorDB()
50
    db.add("cat", "Cats are common pets with independent personalities")
51
    db.add("dog", "Dogs are loyal friends to humans and love going out for walks")
52
    db.add("car", "A car's engine needs regular maintenance and oil changes")
53

54
    for q in ["kitten", "walk the dog", "engine failure"]:
55
        print(f"\nQuery: {q}")
56
        for doc_id, score, text in db.search(q, top_k=2):
57
            print(f"  [{doc_id}] similarity {score:.3f} | {text}")

After running it, you’ll see that “kitten” best matches cat, “walking the dog” best matches dog, and “engine failure” best matches car. This is the basic closed loop of vector retrieval: content → vector → similarity ranking → top-k.

The Bottleneck of Brute-Force Search and ANN#

The implementation above uses Brute Force search: computing the similarity between the query vector and every single vector in the database. When the data volume reaches the millions or tens of millions, this is clearly infeasible—a single query would require computing millions of dot products.

Hence Approximate Nearest Neighbor (ANN) algorithms: sacrificing a little accuracy in exchange for an order-of-magnitude speedup. Mainstream ANN indexes include:

• HNSW (Hierarchical Navigable Small World): Builds a multi-layer “small world” graph, quickly descending from coarse-grained upper layers to fine-grained lower layers. Retrieval complexity is about O(log N), making it one of the most widely adopted indexes in engineering today.
• IVF (Inverted File Index): First clusters the vector space into several buckets, and at query time searches only the nearest batch of buckets.
• PQ (Product Quantization): Performs lossy compression on vectors, storing more vectors in less memory.

The value of real vector databases lies largely in the engineering implementation of these indexes—which is exactly why you don’t need to reinvent the wheel yourself.

Comparison of Mainstream Vector Databases#

When choosing, prioritize: data scale, query latency requirements, whether metadata filtering is needed, and operational cost. Using ChromaDB at the prototype stage and then considering Milvus or Pinecone at the production stage is a common evolution path.

Hands-On: Semantic Search with ChromaDB#

Finally, let’s look at how to use a real vector database. ChromaDB’s default embedding function is all-MiniLM-L6-v2, ready to use out of the box:

1
pip install chromadb

1
import chromadb
2

3
#Create a local persistent client, with data written to disk at ./chroma_data
4
client = chromadb.PersistentClient(path="./chroma_data")
5
collection = client.get_or_create_collection(
6
    name="faq",
7
    #When embedding_function is not passed, the default all-MiniLM-L6-v2 is used
8
)
9

10
#Write documents (ChromaDB automatically calls the embedding function to convert them to vectors)
11
collection.add(
12
    ids=["1", "2", "3", "4"],
13
    documents=[
14
        "How do I request a return? Please click request refund on the order details page.",
15
        "When will the package ship? It usually ships within 24 hours after payment.",
16
        "How do I change the shipping address? You can change it yourself before the order ships.",
17
        "Where do I get coupons? They are released daily at 10 AM in the homepage promotions section.",
18
    ],
19
)
20

21
#Semantic search: relevant documents can be matched even with no keyword overlap
22
results = collection.query(
23
    query_texts=["I want a refund, how do I do that?"],
24
    n_results=2,
25
)
26

27
for doc, distance in zip(results["documents"][0], results["distances"][0]):
28
    print(f"[distance {distance:.3f}] {doc}")

Note that ChromaDB’s query returns distance by default (smaller means more similar), not a similarity score—different databases may define “score” differently, so always check the documentation first.

Conclusion#

The underlying principles of vector databases aren’t mysterious: embeddings turn semantics into vectors, similarity metrics turn “how alike” into “numbers,” and ANN indexes turn “full comparison” into “approximate lookup.” Once you understand these three things, you’ve grasped the essence of vector databases.

For everyday development, you don’t need to implement HNSW by hand—getting a prototype working with ChromaDB first, then evolving to Milvus or Pinecone as your data scales, is the most pragmatic choice. Vector databases are the cornerstone of RAG; master them, and you’ve taken a key step toward building AI applications.

References:

1. Malkov, Y., & Yashunin, D. “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs.” arXiv 2016. https://arxiv.org/abs/1603.09320
2. ChromaDB Official Documentation. https://docs.trychroma.com/
3. Milvus Official Documentation. https://milvus.io/docs
4. pgvector - GitHub. https://github.com/pgvector/pgvector
5. OpenAI Embeddings Guide. https://platform.openai.com/docs/guides/embeddings