A text embedding is a dense, fixed-length vector representation of tokens, sentences, or documents that encodes semantics, syntax, and context in a continuous space. Learned by neural encoders (typically Transformer bi-encoders) with contrastive or masked objectives, embeddings enable similarity search, clustering, classification, personalization, and RAG grounding via cosine or dot-product distance.
What is Text Embedding?
Text embeddings are produced by encoders that map variable-length text to vectors using pooling (CLS token, mean pooling, or attention pooling). Training signals include supervised contrastive pairs/triplets (query–document, entailment, paraphrase) and unsupervised objectives (masked language modeling, inverse cloze). Practical systems normalize vectors (L2) for cosine similarity, pick dimensionalities (e.g., 256–1024) that balance accuracy and memory, and version models to avoid mixed-index drift. Multilingual and domain-specific encoders improve cross-language and specialized recall. Embeddings compose with hybrid search (keyword + vector), rerankers, and graph/RAG pipelines, and can be stored in vector databases for low-latency kNN with metadata filtering.
Why it matters and where it’s used
Embeddings bridge the lexical gap—capturing paraphrases and context beyond keywords—boosting recall and precision for search and assistants. They ground LLMs with semantically relevant passages, power deduplication and clustering, and enable personalization and recommendation. Operationally, compact vectors cut memory and bandwidth, scale to billions of items, and support near–real-time updates.
Examples
- Hybrid enterprise search: keyword prefilter → vector retrieve → cross-encoder rerank.
- RAG context building: embed queries/passages to assemble grounded prompts with citations.
- Deduplication: cosine-threshold near-duplicate detection for documents or chunks.
- Classification/extraction: embed items and centroid-label or kNN-label in vector space.
FAQs
- How many dimensions should I use? 256–1024 are common; larger can improve recall but cost more memory/latency—evaluate on your corpus.
- Cosine or dot-product? Use cosine on L2-normalized embeddings; dot-product can reflect magnitude if not normalized.
- Do I need to re-embed when I change models? Yes—recompute and reindex to avoid mixing spaces; maintain index versioning.
- Are sentence and document embeddings different? They use the same encoder; longer documents often use chunking + pooling, then aggregate at query time.