embedding model

An embedding model is a neural network trained to project discrete or high-dimensional inputs—such as tokens, sentences, images, audio segments, or knowledge-graph nodes—into dense, fixed-dimensional vector representations in a continuous latent space that preserves semantic and structural relationships. The geometry of the resulting embedding space encodes similarity: semantically related inputs map to vectors with high cosine similarity, enabling downstream tasks such as semantic search, clustering, classification, and retrieval-augmented generation. Embedding models are trained using objectives such as contrastive learning, masked language modelling, or cross-modal alignment, and are evaluated on standardised benchmarks including MTEB and BEIR.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:hasPart ai:LatentSpace))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:hasPart ai:PoolingLayer))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:hasPart ai:EmbeddingLayer))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:hasPart ai:Encoder))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:hasPart ai:AttentionMechanism))

Dependency Relationships

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:requires ai:GPUComputing))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:requires ai:FineTuning))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:requires ai:TrainingData))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:dependsOn ai:TransformerArchitecture))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:dependsOn ai:ContrastiveLearning))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:dependsOn ai:Backpropagation))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:dependsOn ai:Tokenisation))

Capability Relationships

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:SemanticSearch))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:RetrievalAugmentedGeneration))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:RecommendationSystem))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:AnomalyDetection))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:ZeroShotClassification))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:KnowledgeGraphCompletion))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:enables ai:CrossLingualTransfer))

Implementation Relationships

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:implements ai:DenseRetrieval))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:implements ai:BiEncoder))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:implements ai:SentenceEncoder))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:uses ai:VectorDatabase))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:uses ai:ApproximateNearestNeighbour))

Reduction Relationships

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:reducesTo ai:DenseVector))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:reducesTo ai:ContinuousRepresentation))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:contrastsWith ai:SparseRepresentation))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:contrastsWith ai:CrossEncoder))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:contrastsWith ai:LargeLanguageModel))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:contrastsWith ai:TF-IDF))

Support Relationships

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:supports ai:NaturalLanguageProcessing))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:supports ai:InformationRetrieval))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:supports ai:MultimodalAI))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:supports ai:ComputerVision))

Relational Relationships

SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:relatedTo ai:KnowledgeGraph))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:relatedTo ai:TransferLearning))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:relatedTo ai:DimensionalityReduction))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:relatedTo ai:EmbeddingLayer))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:bridges ai:VectorDatabase))
SubClassOf(ai:EmbeddingModel
  ObjectSomeValuesFrom(ai:bridges ai:SpatialIndex))

About

Embedding models occupy a foundational position in the modern Artificial Intelligence stack. The conceptual lineage traces back to distributional semantics in computational linguistics — the distributional hypothesis (Harris, 1954) that words occurring in similar contexts tend to have similar meanings — which motivated early count-based co-occurrence methods such as TF-IDF and Latent Semantic Analysis (LSA). However, these sparse vector methods could not capture synonymy, polysemy, or cross-lingual equivalence, and their dimensionality grew with corpus vocabulary. The breakthrough arrived with neural word embedding models: Bengio et al. (2003) demonstrated joint learning of word representations and a language model; Mikolov et al. (2013) showed that shallow architectures trained on local context prediction (Word2Vec) produce dense representations with striking algebraic properties at massive scale.

The contextualised embedding era opened with ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018), which replaced static token embeddings with context-dependent representations produced by running full Transformer Architecture inference. Sentence-BERT (Reimers & Gurevych, 2019) adapted the BERT encoder into a practical sentence embedding model by adding mean pooling and siamese-network fine-tuning with natural language inference data, producing the first performant sentence-level bi-encoder — a model that encodes query and document independently, enabling precomputed offline indexes. This bi-encoder paradigm became central to dense passage retrieval (Karpukhin et al., 2020, DPR), which replaced BM25 as the first-stage retrieval component in open-domain question answering pipelines. The Retrieval-Augmented Generation paradigm (Lewis et al., 2020, RAG) then fused the bi-encoder retriever with a generative language model decoder, establishing the dominant architecture for knowledge-grounded generation that powers enterprise search, chatbots, and AI assistants as of 2026.

Before dense embedding models, information retrieval relied on sparse lexical methods including TF-IDF and BM25, which match documents to queries purely by keyword overlap. Embedding models overcome these limitations by representing semantics geometrically: two documents about the same topic but using different vocabulary nonetheless map to nearby points in the Latent Space, enabling synonym-aware, paraphrase-aware, and cross-lingual retrieval. The dense vector representations produced by embedding models are compact (typically 384–3072 floating-point values vs. vocabulary-sized sparse vectors), differentiable (enabling end-to-end gradient training), and hardware-friendly (matrix operations on floating-point arrays run efficiently on GPU Computing hardware). Paired with Approximate Nearest Neighbour indexing structures such as HNSW or FAISS, dense retrieval across billion-scale corpora runs in milliseconds.

Training Objectives and Pooling Strategies

• Masked Language Modelling (MLM): predicting randomly masked tokens forces the Transformer Architecture to build deep contextual representations. This is the standard BERT-family pretraining objective and produces general-purpose token representations used in the Embedding Layer.

• Contrastive Learning: pairs of semantically similar (positive) and dissimilar (negative) examples are pushed together or apart in embedding space. The NT-Xent loss (Chen et al., 2020, SimCLR for vision; applied to text in SimCSE), InfoNCE, and multiple negatives ranking loss (Hendrycks et al.) are the primary objectives. Hard negative mining — selecting difficult negatives that are semantically plausible but factually distinct — is critical for high-precision retrieval. The E5, GTE, and BGE model families achieve top MTEB rankings through large-scale contrastive pretraining on web-crawled text pairs followed by instruction-tuned fine-tuning.

• Knowledge Distillation: large cross-encoder teacher models jointly encode a query-document pair and produce high-accuracy relevance scores; these scores are used as soft targets to train smaller, faster bi-encoder student models, enabling knowledge transfer from expensive rerankers to efficient retrievers.

• Cross-modal Alignment: CLIP (Radford et al., 2021) aligns image and text representations into a shared vector space using a contrastive objective over image-caption pairs; variants including SigLIP, Jina CLIP, and Qwen3-VL-Embedding extend this to video, audio, and document modalities.

• Pooling Strategies: (1) CLS token pooling uses the first [CLS] special token’s final-layer representation as the sentence embedding; (2) mean pooling averages all non-padding token representations — empirically robust and the default in Sentence Transformers; (3) weighted mean and attention-based pooling apply learned or task-specific weights; (4) last-token pooling is used in decoder-only models (e.g., E5-Mistral, GTE-Qwen2) where the final generated token provides a document representation.

  Architectures and Families

• BERT and Encoder-Only Transformers: foundational bi-encoder backbone. BERT (110M), RoBERTa (125M), and DistilBERT (66M) represent the established core. Modern Sentence-BERT fine-tunes these with NLI and STS training data. At smaller scales, models such as all-MiniLM-L6-v2 (22M, 384d) and bge-small-en-v1.5 offer excellent quality-to-latency trade-offs for local CPU-based inference.

• E5 / GTE / BGE families: state-of-the-art open models as of 2025–2026. E5-large-v2 (335M) and multilingual-E5-large use contrastive pretraining on curated web text pairs with instruction prefixes (“query:” / “passage:”). GTE-Qwen2-7B uses a decoder-only Qwen2 base with last-token pooling, and BGE-M3 achieves multi-lingual, multi-granularity retrieval with sparse, dense, and ColBERT-style multi-vector outputs in a single model.

• Qwen3-Embedding: released June 2025, built on the Qwen3 foundation model family. Qwen3-Embedding-8B achieves 70.6 MTEB score, surpassing all proprietary API models including OpenAI text-embedding-3-large (64.6) and Google text-embedding-004 (68.3). Supports over 100 languages and user-defined output dimensions from 32 to 1024 via Matryoshka Representation Learning.

• OpenAI text-embedding-3: proprietary dense encoders with 256–3072 dimensional MRL outputs; text-embedding-3-large leads proprietary leaderboards as of mid-2025 but is surpassed by Qwen3-Embedding-8B by 2026.

• Google Gemini Embedding 2: released March 2026, the first all-modality embedding model supporting text, image, video, audio, and PDF documents in a single 3,072-dimensional shared vector space. Achieves 68.3 MTEB text score and 92.8% cross-modal retrieval recall on multimodal benchmarks.

• Qwen3-VL-Embedding: open-source multimodal embedding series (2B and 8B variants) built on Qwen3-VL vision-language foundation models. Achieves hard_avg_R@1 = 0.945, surpassing Gemini (0.928) and Voyage (0.900) on cross-modal retrieval benchmarks.

• Knowledge Graph Embedding Models: TransE (Bordes et al., 2013), RotatE (Sun et al., 2019), ComplEx (Trouillon et al., 2016), and TH-RotatE (2025) embed entity and relation IDs into a continuous space optimised for link prediction scoring functions. Graph Neural Network variants (R-GCN) aggregate neighbourhood structure for richer node embeddings.

• Scientific Domain Models: protein language models (ESM-2, ProtTrans, AlphaFold3’s pLM component) embed amino-acid sequences; material science models embed crystal structure descriptors. These domain-specific embedding models bridge to scientific discovery workflows.

  Matryoshka Representation Learning and Quantisation

  Matryoshka Representation Learning (MRL), introduced by Kusupati et al. (2022), trains embedding models to produce vectors where the first d’ dimensions for any d’ < d are themselves a valid, useful embedding, forming a nested hierarchy of sub-vectors. This enables variable-length inference from a single model pass: a 3072-dimensional Qwen3-Embedding model can produce useful 64-, 128-, 256-, 512-, or 1024-dimensional embeddings by truncating, enabling storage and latency trade-offs without retraining. By 2025–2026, MRL training has become nearly universal among new embedding model releases (Qwen3, text-embedding-3, Jina v5).

  Binary and integer quantisation of embedding vectors reduces storage and retrieval cost. Int8 quantisation of 3072-dimensional float32 vectors reduces memory by 4× with minimal accuracy loss. Binary quantisation (1-bit representations) paired with bitwise XOR/POPCOUNT operations enables extremely fast retrieval at the cost of higher accuracy loss, partially recovered by re-ranking with full-precision vectors. Quantization Aware Matryoshka Adaptation (QAMA, CIKM 2025) combines both techniques in a unified framework. Ultra-sparse embedding methods (SPLADE, CSRv2) produce representations that activate only a small fraction of dimensions, enabling inverted-index retrieval — combining the efficiency of BM25 with the semantic capture of dense models.

  Evaluation Benchmarks

• MTEB (Massive Text Embedding Benchmark): 56-task, 8-category benchmark (retrieval, clustering, STS, classification, reranking, pair classification, bitext mining, summarisation) covering over 1000 languages and 58 English datasets. The primary leaderboard maintained by Hugging Face. As of 2026, QZhou-Embedding leads at 75.97 MTEB score overall, with Qwen3-Embedding-8B leading open-source at 70.6.

• BEIR (Benchmarking IR): zero-shot heterogeneous retrieval benchmark spanning 18 datasets across biomedical (BioASQ, TREC-COVID), financial, legal, and news domains to test out-of-distribution generalisation. Scores on BEIR are consistently lower than in-domain retrieval, revealing the difficulty of domain shift.

• STS-Benchmark: semantic textual similarity tasks measuring Pearson/Spearman correlation between model cosine scores and human judgements, used as an early standardised evaluation for sentence encoders.

• ANN Recall@k: practical index recall measuring how often the true nearest neighbour in exact search is recovered by the HNSW or FAISS approximate index at retrieval time; critical for production system validation.

• MIRACL: multilingual information retrieval benchmark covering 18 languages; used to evaluate cross-lingual embedding models.

  Use Cases

• Semantic Search and RAG: dense passage retrieval using bi-encoders replaced BM25 as the standard first-stage retrieval step. Combined with generative models in Retrieval-Augmented Generation pipelines, embedding models enable knowledge-grounded generation without full fine-tuning. Enterprise search platforms (Elasticsearch with dense vector support, Weaviate, Pinecone, Milvus) use embedding models to index millions of documents for sub-second semantic retrieval.

• Recommendation Systems: user and item embeddings produced by Two-Tower architectures enable real-time recommendation at scale. At inference, the item tower precomputes embeddings indexed in ANN structures; the user tower encodes the query online. Used in production at YouTube, LinkedIn, Pinterest, and major e-commerce platforms.

• Code and Technical Search: models such as CodeBERT and UniXcoder embed source code and natural language documentation into a shared space, enabling natural-language-to-code search, duplicate detection, and commit message retrieval used in GitHub Copilot’s retrieval components.

• Anomaly Detection and Fraud: transaction sequences, log events, or user behaviour sequences embedded into latent space allow Anomaly Detection by identifying points that deviate significantly from cluster centroids or learned normality manifolds. Applied in financial fraud detection, network intrusion detection, and industrial predictive maintenance.

• Cross-Lingual and Multilingual Tasks: multilingual embedding models (mBERT, LaBSE, multilingual E5, Qwen3-Embedding) align representations across languages without requiring parallel corpora, enabling zero-shot cross-lingual retrieval, classification, and question answering.

• Scientific and Medical Discovery: protein embedding models (ESM-2) power drug discovery workflows by embedding amino-acid sequences for functional similarity search and mutation effect prediction. Clinical NLP systems embed patient notes and medical codes for ICD-10 code mapping, phenotype extraction from EHRs, and clinical trial matching.

• Knowledge Graph Population: entity and relation embedding models trained on triple facts enable link prediction, entity disambiguation, and knowledge graph completion, powering question answering over structured knowledge bases such as Wikidata and biomedical ontologies.

  Standards and Governance

  No single ISO or IEEE standard governs embedding models, but evaluation methodology is converging around community benchmarks (MTEB, BEIR, STS-B) as de-facto standards. The Hugging Face Hub hosts the dominant open ecosystem of embedding models, with model cards specifying context length, pooling strategy, benchmark scores, and training data provenance as informal metadata standards. The Sentence Transformers library (sbert.net) provides a standardised Python interface for embedding model inference, fine-tuning, and evaluation, with over 10,000 pre-trained models available through the Hub. EU AI Act and emerging AI Governance frameworks classify embedding models as general-purpose AI components; when used in high-risk downstream systems (healthcare diagnostics, law enforcement, credit scoring), their provenance, training data, and evaluation characteristics must be documented under the GPAI provisions. ONNX and GGUF export formats enable hardware-agnostic deployment on edge devices, CPUs, and inference servers, reducing dependency on GPU Computing hardware for production retrieval systems.

  Academic Context

  The field of dense embedding models has produced several landmark methodological contributions. Bengio et al. (2003) established the neural language model as a joint word representation and probability model. Mikolov et al. (2013) demonstrated scale-efficient embedding via shallow architectures. Devlin et al. (2018) showed that deep bidirectional contextualisation via masked language modelling produces universally useful sentence representations. Reimers and Gurevych (2019, Sentence-BERT) solved the practical sentence embedding problem with siamese fine-tuning. Karpukhin et al. (2020, DPR) brought dense retrieval to open-domain QA. Lewis et al. (2020, RAG) integrated dense retrieval with generative models. Radford et al. (2021, CLIP) demonstrated cross-modal contrastive alignment at scale. Kusupati et al. (2022, MRL) enabled flexible-length embeddings via nested training. The theoretical understanding of contrastive learning — why InfoNCE loss produces well-separated clusters — was advanced by Wang and Isola (2020) through the alignment-uniformity framework. Key research groups include University of Massachusetts Amherst (MS MARCO, DPR), Facebook AI Research (FastText, DPR), Google Brain/DeepMind (T5, DPR, Gemini Embeddings), Microsoft Research (E5, MS MARCO), and Alibaba DAMO Academy (GTE, Qwen-Embedding families).

  Current Landscape (2026)

  By 2026, the embedding model landscape has undergone a significant shift: open-source models now outperform proprietary API models on the primary MTEB leaderboard. Qwen3-Embedding-8B (70.6 MTEB, released June 2025) surpasses OpenAI text-embedding-3-large (64.6) and Google text-embedding-004 (68.3) across retrieval, clustering, and classification tasks. The gap between open-source and proprietary models, once stark, has effectively reversed for companies with GPU inference infrastructure. Google’s March 2026 Gemini Embedding 2 introduced unified all-modality embedding (text, image, video, audio, PDF into 3,072-dimensional shared space), temporarily reclaiming multimodal leadership before Qwen3-VL-Embedding (2B, 8B) surpassed it on cross-modal retrieval benchmarks. The MTEB 2026 leaderboard summit (QZhou-Embedding, 75.97) reflects rapid iteration. Practical deployment patterns have consolidated around: (1) small efficient models (all-MiniLM-L6-v2, bge-small-en) for CPU edge/low-latency paths; (2) mid-size models (Qwen3-Embedding-0.6B, bge-m3) for GPU inference with multilingual needs; (3) large models (Qwen3-Embedding-8B, E5-Mistral-7B) for maximum quality in offline indexing pipelines. The Sentence Transformers library has become the canonical production interface, with 10,000+ models available via Hugging Face Hub.

  UK Context

  UK academia and industry have made consistent contributions to embedding model research and deployment. The University of Edinburgh’s School of Informatics — one of Europe’s foremost computational linguistics centres — has an active Edinburgh Clinical NLP group that competed in SemEval-2024, demonstrating embedding-based approaches for clinical NLP tasks including medical evidence inference. The University of Edinburgh’s UKRI Centre for Doctoral Training in Biomedical AI trains researchers in embedding models applied to health data, including phenotype embedding approaches for electronic health record analysis. Imperial College London’s AI research groups have contributed to embedding techniques for scientific computing and physics-informed neural networks. University College London (UCL) researchers contributed to the theoretical analysis of contrastive learning objectives and embedding geometry. The Alan Turing Institute co-ordinates national capacity in representation learning and hosts Enrichment Scheme students working on embedding-based knowledge graph reasoning. In Northern England, University of Manchester’s Centre for AI Fundamentals includes groups working on entity and knowledge graph embeddings for biomedical ontologies, with NHS Greater Manchester partnerships enabling clinical concept embedding for ICD-10 mapping and clinical note retrieval. University of Leeds hosts CISTIB (Centre for Computational Imaging and Simulation Technologies in Biomedicine), which applies embedding models to multimodal medical data integration. University of Sheffield’s NLP group (USFD) has longstanding expertise in embedding-based spoken language understanding, and the Sheffield Robotics centre applies embedding models to robot perception pipelines. At the industry level, UK enterprises including Elsevier (scientific content retrieval), Springer Nature, and UK-headquartered fintech firms deploy embedding models at production scale for document retrieval and fraud detection. Hugging Face’s European headquarters in Paris co-ordinates closely with UK research partners, and the Sentence Transformers maintainers (UKP Lab, TU Darmstadt — with strong EU-UK collaborations) publish new architectures adopted by UK-based RAG deployments.

  Future Directions (2026–2030)

• Unified multimodal embedding architectures: single models capable of embedding any modality (text, image, video, audio, 3D, molecular structure) into a shared metric space will become standard, replacing separate per-modality embedding pipelines.

• Long-context document embedding: extending effective context windows to 64K–1M tokens for embedding entire books, codebases, or legal documents as single vectors, combining pooling strategies with sparse and hierarchical attention.

• Token-level and multi-vector representations: ColBERT-style late-interaction models that retain per-token embeddings for finer-grained matching will compete with bi-encoder single-vector models, especially for technical and highly structured documents.

• Tokeniser-free and byte-level models: research into byte-level architectures (BYTELATENT, MegaByte) that eliminate the discrete vocabulary / embedding layer interface, potentially enabling more efficient and robust multilingual and multimodal embedding.

• On-device and edge embedding inference: 1-bit and 2-bit quantised embedding models running on mobile CPUs and NPUs, enabling private, offline semantic search without cloud API dependency — particularly relevant for healthcare and legal applications requiring data residency.

• Embedding models for agentic systems: as AI agents rely on memory retrieval to recall past context and tool use patterns, embedding models will be integrated into agent memory architectures (episodic memory, working memory buffers) and multi-agent coordination frameworks.

• Causal and counterfactual embeddings: moving beyond purely geometric similarity to embeddings that encode causal relationships, enabling reasoning about interventions and counterfactuals — relevant for scientific discovery and planning applications.

  Research and Literature

  1. Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of Machine Learning Research, 3, 1137–1155.
  2. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781.
  3. Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. EMNLP, 1532–1543.
  4. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, C. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
  5. Peters, M., Neumann, M., Iyyer, M., et al. (2018). Deep contextualised word representations (ELMo). NAACL-HLT, 2227–2237.
  6. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. EMNLP 2019, 3982–3992.
  7. Karpukhin, V., Oguz, B., Min, S., et al. (2020). Dense passage retrieval for open-domain question answering (DPR). EMNLP 2020, 6769–6781.
  8. Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks (RAG). NeurIPS 33.
  9. Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning transferable visual models from natural language supervision (CLIP). ICML 2021.
  10. Wang, T., & Isola, P. (2020). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ICML 2020.
  11. Gao, T., Yao, X., & Chen, D. (2021). SimCSE: Simple contrastive learning of sentence embeddings. EMNLP 2021.
  12. Kusupati, A., Bhatt, G., Rege, A., et al. (2022). Matryoshka representation learning. NeurIPS 2022.
  13. Wang, L., Yang, N., Huang, X., et al. (2024). Text embeddings by weakly-supervised contrastive pre-training (E5). arXiv:2212.03533.
  14. Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2022). MTEB: Massive text embedding benchmark. EACL 2023; arXiv:2210.07316.
  15. Thakur, N., Reimers, N., Rücklé, A., et al. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. NeurIPS 2021 Datasets Track.
  16. Bordes, A., Usunier, N., Garcia-Duran, A., et al. (2013). Translating embeddings for modelling multi-relational data (TransE). NeurIPS.
  17. Sun, Z., Deng, Z.-H., Nie, J.-Y., & Tang, J. (2019). RotatE: Knowledge graph embedding by relational rotation in complex space. ICLR 2019.
  18. Neelakantan, A., Xu, T., Puri, R., et al. (2022). Text and code embeddings by contrastive pre-training. arXiv:2201.10005.
  19. Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations (SimCLR). ICML 2020.
  20. Hofstätter, S., Lin, S.-C., Yang, J.-H., et al. (2021). Efficiently teaching an effective dense retriever with balanced topic aware sampling. SIGIR 2021.
  21. Khattab, O., & Zaharia, M. (2020). ColBERT: Efficient and effective passage search via contextualised late interaction over BERT. SIGIR 2020.
  22. Qwen Team, Alibaba (2025). Qwen3 embedding: Text-oriented embedding and reranker model series. Technical Report, June 2025.
  23. Qwen Team, Alibaba (2026). Qwen3-VL-Embedding: Open-source multimodal embedding series. Technical Report, 2026.
  24. Google DeepMind (2026). Gemini Embedding 2: All-modality embedding model. Technical Report, March 2026.
  25. Edinburgh Clinical NLP (2024). Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4. arXiv:2404.00484.
  26. Zhang, K., et al. (2025). Quantization aware Matryoshka adaptation (QAMA). ACM CIKM 2025.
  27. Zhu, D., et al. (2025). CSRv2: Unlocking ultra-sparse embeddings for information retrieval. arXiv:2602.05735.
  28. Rücklé, A., Glavaš, G., Losada, M., et al. (2021). AdaptSum: Towards low-resource domain adaptation for abstractive summarisation. arXiv:2103.11332. [Embedding-based domain transfer context.]

Provenance