Encoders and vector embeddings: turning meaning into numbers
Generative models get the attention, but encoder models quietly power search, RAG, and recommendations by turning text into vectors that capture meaning. Here's how embeddings work.
Most of the AI conversation is about generation — models that write. But an enormous amount of production value comes from a quieter cousin: encoder models that turn text into vectors. Embeddings are what make semantic search, RAG, and recommendations work, and they're worth understanding in their own right.
Encoder vs decoder
Decoder models (the GPT family) generate text one token at a time. Encoder models (the BERT lineage) read text into a representation instead of continuing it. Embedding models are encoders whose whole job is to output a single vector that captures the meaning of the input. (Encoder-decoder models do both, for tasks like translation.)
What an embedding actually is
An embedding is a dense vector — hundreds or thousands of numbers — that places a piece of text at a point in a high-dimensional space. The space is arranged so that meaning maps to geometry: texts that mean similar things sit close together, and you measure 'close' with cosine similarity. 'A small cat' and 'a kitten' end up as neighbours; 'a car' ends up far away.
Why they matter
- Semantic search — match on meaning, not just keywords.
- RAG retrieval — the retrieval half of RAG is entirely embeddings and similarity search.
- Clustering and classification — group or label text by where it lands in the space.
- Recommendations and dedup — find similar items, or near-duplicates, by proximity.
from openai import OpenAI
client = OpenAI()
def embed(text):
return client.embeddings.create(
model='text-embedding-3-small', input=text
).data[0].embedding
# similar meaning -> high cosine similarity
score = cosine(embed('a small cat'), embed('a kitten'))Generation gets the spotlight, but embeddings do the unglamorous work of turning meaning into maths — and that's what search and RAG run on.