Vector embeddings: meaning as geometry 🧭

An embedding is a fixed-length list of numbers (a vector) that a model assigns to a piece of text so that semantic similarity shows up as geometric closeness. "car" might become [0.02, -0.41, 0.88, ...] and "automobile" a nearly identical list, while "banana" points somewhere else entirely. You never read the numbers directly. You only ever compare them.

Now the words: the numeric list is a dense vector (dense = most entries are non-zero, unlike a sparse bag-of-words). The space it lives in is the embedding space or latent space. The property that similar meaning gives nearby vectors is the model being semantically meaningful. Each position in the vector is a dimension.

Each word carries a small hand-built vector. Click a star (or use the dropdown). The demo computes cosine similarity from that word to all fourteen others, connects the top three, and prints the numbers. Watch bank pull toward both money and river, and java pull toward both python and coffee. That split is polysemy, one string with two meanings, and it is a real failure mode in retrieval.

Now the words: the flattened picture is a projection (real tools use PCA, t-SNE, or UMAP to squeeze 768 dims into 2). The lines connect nearest neighbors. A word with two meaning-clusters pulling on it is exhibiting polysemy, and a single averaged vector for it is a known weakness of static embeddings.

Move the sliders. The angle slider rotates the blue vector; the length slider stretches it. Notice: cosine only changes when the angle changes. Stretch the vector all you want and cosine holds steady, while Euclidean distance jumps. For text embeddings we care about direction (topic), not magnitude, so cosine is the default.

Now the words: cosine similarity is the dot product divided by the product of the two L2 norms (magnitudes). Normalizing means scaling a vector to unit length. The inner-product metric IP equals cosine when vectors are normalized. FAISS and pgvector expose cosine, L2, and inner product as choices.

Real models output 384 (all-MiniLM-L6-v2), 768 (many BERT-family models), 1536 (OpenAI text-embedding-3-small), or 3072 dims (text-embedding-3-large). More dimensions buys capacity to separate meanings, but it costs storage, memory, and query time. There is a real tradeoff called the curse of dimensionality: in very high dimensions, random points drift toward being equally far apart, so distances get less discriminating and you lean harder on a good metric and a good model.

Now the words: the number of entries is the dimensionality. The distance-flattening problem is the curse of dimensionality / concentration of distances. Nearly-perpendicular vectors are near-orthogonal. Truncating a vector while keeping quality is Matryoshka Representation Learning (MRL).

Two paths. Hosted APIs: OpenAI text-embedding-3-small (1536 dims, cheap) and text-embedding-3-large (3072 dims), or Cohere embed-english-v3.0 / embed-multilingual-v3.0. Open-source local: sentence-transformers models like all-MiniLM-L6-v2 (384 dims, fast, runs on a laptop) or BAAI/bge-* for higher quality. In your rag-docs-chatbot you can start local with MiniLM, then swap to a hosted model by changing one function.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dim, ~80MB

docs = ["The bank raised interest rates.",
        "A puppy chased its tail."]
vecs = model.encode(docs, normalize_embeddings=True)   # L2-normalized
print(vecs.shape)          # (2, 384)

from openai import OpenAI
client = OpenAI()

resp = client.embeddings.create(
    model="text-embedding-3-small",    # 1536 dims
    input=["The bank raised interest rates."],
    dimensions=512                        # optional Matryoshka truncation
)
vec = resp.data[0].embedding          # list[float], length 512

Now the words: these are sentence / text embedding models, usually a transformer encoder with mean pooling over token vectors and a contrastive fine-tune. The public scoreboard is MTEB (Massive Text Embedding Benchmark). Managed services like AWS Kendra (used in your advisor-portal) and Bedrock Knowledge Bases hide the model choice and do the embedding for you.

Split each document into chunks (commonly 200 to 800 tokens) and embed each chunk separately. Add overlap (say 10 to 20 percent) so ideas straddling a boundary appear whole in at least one chunk. Small chunks are precise but lose surrounding context; big chunks keep context but dilute the vector and retrieve noise. Drag the sliders and watch the chunk layout change.

Now the words: the split unit is a chunk or passage; the shared region is overlap / stride. Splitting by structure is recursive / semantic chunking. Metadata carried with each chunk (source, page, section) enables metadata filtering at query time. A parent-document retriever embeds small chunks but returns the larger surrounding passage.

Six sentences, each pre-embedded with a tiny real bag-of-concepts model. Type a query (try hot drink, money and loans, kitten by the window, or coding in java) and hit search. The demo embeds your query the same way, scores every sentence by cosine, and animates them into ranked order with a similarity bar. This is exactly the retrieve step of your rag-docs-chatbot before the chunks go into the prompt.

Now the words: this flow is dense retrieval / vector search. At scale you do not compare against every vector; an ANN (approximate nearest neighbor) index like HNSW or IVF-PQ in FAISS, pgvector, Pinecone, or Weaviate finds the top-k fast. Combining dense + sparse and re-ranking is hybrid search with a cross-encoder reranker.

Now the words: incompatible query/doc models is a model mismatch. Space geometry shifting on upgrade is embedding drift. Rebuilding vectors is re-indexing / backfill. Serving old and new indexes during migration is a shadow / blue-green index.

The eight questions a senior gets asked about embeddings and RAG retrieval. Tap to reveal a crisp answer.