Attention Mechanisms

Attention mechanisms are a family of neural network components that compute context-dependent weighted combinations of value representations by scoring the relevance of query vectors against key vectors, enabling each output position to draw selectively from any position in an input sequence — overcoming the fixed-length bottleneck of recurrent encoders. Originally introduced as additive (Bahdanau) attention for neural machine translation in 2015, they were generalised to scaled dot-product attention and multi-head attention in the 2017 Transformer architecture, where they became the sole sequence-modelling primitive, replacing recurrence entirely. Attention mechanisms now underpin virtually all frontier deep learning systems across language, vision, speech, biology, and multimodal reasoning.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:SelfAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:CrossAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:CausalAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:MultiHeadAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:ScaledDotProductAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:AttentionWeight))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:AttentionMask))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:hasPart ai:QueryKeyValue))

Dependency Relationships

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:requires ai:NeuralNetwork))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:requires ai:SoftmaxFunction))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:requires ai:Backpropagation))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:requires ai:GradientDescent))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:dependsOn ai:PositionalEncoding))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:dependsOn ai:FeedForwardNetwork))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:dependsOn ai:DeepLearning))

Capability Relationships

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:TransformerArchitecture))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:LargeLanguageModels))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:VisionTransformer))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:MultimodalAIArchitecture))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:GraphAttentionNetwork))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:DiffusionTransformer))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:MachineTranslation))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:enables ai:MultimodalReasoning))

Implementation Relationships

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:implements ai:AIModelArchitecture))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:uses ai:FlashAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:uses ai:GroupedQueryAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:uses ai:PagedAttention))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:uses ai:MixtureOfExperts))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:supports ai:ExplainableAI))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:supports ai:KnowledgeDistillation))

Reduction Relationships

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:reducesTo ai:AttentionMechanism))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:reducesTo ai:NeuralNetworkComponent))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:reducesTo ai:DeepLearning))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:contrastsWith ai:LSTM))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:contrastsWith ai:RecurrentNeuralNetwork))

SubClassOf(ai:AttentionMechanisms
  ObjectSomeValuesFrom(ai:supports ai:ExplainableAI))

About

Attention mechanisms constitute the dominant computational primitive of modern Deep Learning, enabling Neural Network models to dynamically route information by computing relevance scores between query representations and key representations over arbitrarily long sequences without the sequential bottleneck inherent to Recurrent Neural Network and LSTM architectures. Unlike recurrent models, which must process sequences step-by-step and compress all prior context into a fixed-size hidden state vector, or convolutional models, which are limited to receptive fields of fixed spatial extent, attention allows each output position to directly access any input position in a single forward pass. This property — unrestricted long-range dependency modelling — is the defining capability that separates attention-based architectures from all prior sequence-modelling paradigms. The mechanism’s efficiency also differs structurally: recurrence is inherently sequential (O(n) serial steps), blocking GPU parallelism; attention’s pairwise computation over the full sequence (O(n²) operations, all independent) saturates GPU tensor-core parallelism during training, which is why transformer training throughput scales so favourably with hardware generation.

The historical trajectory begins with early neural attention models and culminates in the Transformer architecture that now dominates all of frontier AI. Graves et al.’s Neural Turing Machine (2014) introduced differentiable memory addressing via attention-weighted access to external memory — a content-based attention mechanism enabling the NTM to learn algorithmic tasks from examples by storing and retrieving entries from a memory matrix. Bahdanau, Cho, and Bengio (2015) independently developed what became known as additive attention for neural Machine Translation: at each decoder time step, a small feed-forward network (the alignment model) computed an energy score e_{ij} = tanh(W_a * s_{i-1} + U_a * h_j) between the decoder hidden state s_{i-1} and each encoder hidden state h_j, then normalised via Softmax Function to produce attention weights alpha_{ij} = softmax(e_{ij}), used to construct a context vector c_i = sum_j(alpha_{ij} * h_j). This broke the fixed-length encoder-output bottleneck that forced all sequence information through a single vector, achieving state-of-the-art BLEU scores on English-to-French (29.3 BLEU on WMT 2014) and English-to-German (16.5 BLEU) benchmarks. Luong et al. (2015) generalised to multiplicative and dot-product attention variants, introducing global attention (attend to all source positions) and local attention (attend to a window around a predicted alignment position), analysing the trade-off between computational cost and modelling power.

The pivotal theoretical generalisation came in the 2017 “Attention Is All You Need” paper by Vaswani et al. at Google Brain, proposing the Transformer — a model that dispenses with recurrence and convolution entirely. The Transformer constructs all sequence representations from stacks of multi-head self-attention sub-layers and position-wise Feed Forward Network sub-layers, connected by residual connections (He et al. 2016) and layer normalisation (Ba et al. 2016). The original architecture used 6 encoder layers and 6 decoder layers, 8 attention heads, d_model=512, achieving 28.4 BLEU on English-to-German (state-of-the-art at publication time, surpassing the best recurrent and convolutional models by more than 2 BLEU points) at significantly lower training cost (12 hours on 8 P100 GPUs versus weeks of recurrent training). The Transformer’s key advantages over recurrence are: (1) O(1) path length between any two positions (versus O(n) for RNNs), allowing gradients to flow without vanishing across long contexts; (2) full parallelism during training over the sequence dimension; (3) constant number of sequential operations; (4) position-wise weight sharing that generalises across sequence positions identically.

BERT (Devlin et al., 2019) demonstrated that bidirectional Self Attention over masked tokens — the Masked Language Model (MLM) objective, inspired by the Cloze task, predicting 15% randomly masked tokens from full bidirectional context — pre-trained on BooksCorpus + English Wikipedia (3.3B words) transfers to 11 downstream Natural Language Processing tasks with fine-tuning, achieving new state-of-the-art on GLUE, SQuAD 1.1, and SQuAD 2.0 simultaneously. This established the pre-train/fine-tune paradigm: a single large self-attention model trained on vast unlabelled corpora encodes transferable linguistic and world knowledge, which downstream task-specific heads can exploit with minimal labelled data. This paradigm now underpins GPT-4o, Claude 3/4, and Gemini Multimodal Language Model, scaled to hundreds of billions of parameters. The Vision Transformer (Dosovitskiy et al., 2020) extended self-attention from token sequences to image patch sequences: a 224×224 image is divided into 14×14 non-overlapping 16×16 patches, each flattened to a 768-dimensional vector and linearly projected, then processed by a standard Transformer encoder. At scale (ViT-G, 1.8B parameters, trained on JFT-3B, 3 billion internal Google images), ViT achieves 90.45% top-1 ImageNet accuracy. This spawned Swin Transformer (Liu et al., 2021, hierarchical windowed attention), DeiT (Touvron et al., 2021, data-efficient ViT training with distillation), and Diffusion Transformer (Peebles & Xie, 2023, replacing U-Net in diffusion models).

The mathematical complexity of vanilla attention is O(n²d) in computation and O(n²) in memory for sequence length n and dimension d, a quadratic scaling that becomes prohibitive for contexts exceeding ~2,048 tokens on 40GB A100 GPUs. This constraint drove an entire sub-field of efficient attention research over 2019-2024: Sparse Transformer (Child et al. 2019, fixed-stride and local patterns), Longformer (Beltagy et al. 2020, sliding window + global tokens, O(n)), BigBird (Zaheer et al. 2020, random + window + global, O(n)), Performer (Choromanski et al. 2021, random feature approximation, O(n)), Linformer (Wang et al. 2020, low-rank key/value projection), and Reformer (Kitaev et al. 2020, LSH hashing for approximate O(n log n)). The most impactful industrial solution proved to be Flash Attention (Dao et al., 2022, NeurIPS), which is not a mathematical approximation but an I/O-aware exact algorithm exploiting the GPU memory hierarchy: by tiling Q, K, V matrices to fit in fast on-chip SRAM (96KB per SM on A100) and recomputing attention scores during the backward pass rather than storing the full n×n attention matrix in slow HBM, FlashAttention achieves 2-4x wall-clock speedup on A100 with bitwise-identical output. FlashAttention-2 (Dao 2023, ICLR 2024) restructured parallelisation to better exploit warp-level scheduling and achieved 50-70% of A100 theoretical FLOP/s. FlashAttention-3 (NeurIPS 2024 Spotlight) introduces asynchronous pipelining exploiting H100 Hopper’s asynchronous memory-copy units (TMA), warp-specialisation, and FP8 quantisation, reaching 75% of H100 theoretical FLOP/s and 1.5-2x speedup over FA-2 on H100. Grouped Query Attention (Ainslie et al. 2023, EMNLP), adopted in Llama 2/3, Mistral 7B, Gemma, and Falcon, reduces KV-cache memory by sharing K and V heads across groups of query heads: with G groups each sharing one KV head, GQA reduces KV-cache size from O(n · h · d_v) to O(n · G · d_v), where G << h, achieving near-MQA inference speed with near-MHA quality. Paged Attention (Kwon et al. 2023, SOSP), deployed in the vLLM serving engine, virtualises the KV-cache using operating-system-style paged memory allocation, eliminating internal fragmentation in the KV-cache, enabling serving throughput of 23x baseline TensorFlow Serving for equivalent quality.

Components / Architecture

Scaled Dot-Product Attention (fundamental unit)

• Input: Q (queries), K (keys), V (values) — all linear projections of the same or different input sequences

• Computation: softmax((QK^T) / sqrt(d_k)) V

• Scaling by sqrt(d_k) prevents dot-product magnitude from driving softmax into saturation regions, preserving gradient flow through Backpropagation

• Output: weighted sum of V vectors; same dimensionality as V

• Computational cost: O(n² d) time, O(n²) space for naive implementation

  Multi-Head Attention (MHA) — Vaswani et al. 2017

• Runs h parallel scaled dot-product attention heads over reduced-dimension projections (d_k = d_v = d_model/h)

• W_Q^i, W_K^i, W_V^i are learned projection matrices per head i

• Outputs concatenated and linearly projected: MultiHead(Q,K,V) = Concat(head_1,…,head_h) W

• Standard in all Transformer encoder/decoder stacks; original paper uses h=8, d_model=512

  Attention Variant Taxonomy

• Self Attention: Q=K=V=X (intra-sequence); bidirectional in encoders, causal-masked in decoders

• Cross Attention: Q from one sequence (decoder state), K,V from another (encoder output); bridges modalities in Multimodal AI Architecture

• Causal / Masked Attention: upper-triangular Attention Mask zeroes future positions; mandatory for auto-regressive generation

• Multi-Query Attention (MQA): single shared KV head; maximum inference speed, some quality loss

• Grouped Query Attention (GQA): G groups each sharing one KV head; Llama-3, Mistral, Gemma adopt GQA

• Sliding Window Attention: tokens attend only within a local window of size w; used in Longformer, Mistral

• Flash Attention: I/O-optimal exact computation via tiling; not a variant of the mathematical definition, but an implementation

  Positional Encoding (required complement)

• Absolute sinusoidal PE: Vaswani et al. 2017; non-learnable, infinite extrapolation

• Learned absolute PE: BERT, GPT-2; fixed maximum length

• Relative PE: Shaw et al. (2018); Transformer-XL; attends to relative positions

• Rotary Position Embedding (RoPE): Su et al. (2021); used in Llama, PaLM, Gemma; encodes position via rotation of query/key vectors; excellent length generalisation

• ALiBi: Press et al. (2022); adds negative linear bias to attention scores; strong length extrapolation; used in MPT, Falcon

  Use Cases / Major Families

  Large Language Models (decoder-only Transformers) The dominant application of attention mechanisms is in decoder-only causal-attention language models trained at scale. GPT-3 (Brown et al. 2020, 175B parameters, 96 attention layers, 96 heads, d_model=12288) demonstrated that causal self-attention over large pretrained corpora produces emergent few-shot in-context learning without weight updates — a capability absent at smaller scales. GPT-3’s attention architecture directly inspired the RLHF-alignment paradigm used in InstructGPT (Ouyang et al. 2022) and all subsequent instruction-tuned LLMs. GPT-4o (OpenAI 2024): extended causal attention to native multimodal operation, processing text tokens and image patch tokens within a single unified attention stream. Unlike Flamingo-style architectures that freeze the LM and inject vision via cross-attention, GPT-4o integrates a vision encoder that projects image patches into the same embedding space as text tokens, processed jointly by the decoder’s causal attention. Claude 3/3.5/4 (Anthropic 2024-2026): use long-context multi-head attention with RoPE scaled to 200K tokens via NTK-aware scaling of rotary base; incorporate Constitutional AI fine-tuning post-RLHF. Claude 3.5 Sonnet achieves near-GPT-4 reasoning on MMLU (90.4%) and HumanEval coding (89%) benchmarks at lower inference cost. Llama 3.1 (Meta AI 2024, 8B/70B/405B): adopts GQA (8 KV heads) with RoPE theta=500,000 for 128K context; trained on 15.6T tokens; Llama 3.3 70B (December 2024) achieves Llama-3.1-405B-level performance on key benchmarks, demonstrating significant efficiency gains from improved attention-training recipes. All frontier LLMs use Flash Attention 2 or 3 for training throughput; GQA or MQA for inference KV-cache efficiency; and RoPE or ALiBi for position encoding.

  Vision Transformers and Computer Vision Vision Transformer (ViT, Dosovitskiy et al. 2021) standardised self-attention for image classification:

• ViT-B/16: 86M parameters, 12 layers, 12 heads, 85.9% ImageNet top-1 (with 21K pretraining)

• ViT-L/16: 307M parameters, 16 heads, 87.1% ImageNet top-1

• ViT-G/14: 1.8B parameters, 16 heads, 90.45% top-1 (JFT-3B), state-of-the-art pure transformer on ImageNet Swin Transformer (Liu et al. 2021): introduces shifted-window self-attention creating hierarchical feature maps (C3/C4/C5 analogous to ResNets), enabling attention-based object detection (Cascade Mask-RCNN) and semantic segmentation (UperNet) at state-of-the-art; Swin-L achieves 86.3% ImageNet, 57.7 box AP on COCO object detection. Diffusion Transformer (DiT, Peebles & Xie 2023): replaces U-Net backbone in DDPM latent diffusion with a transformer encoder processing image patches plus diffusion timestep and class-label conditioning via adaptive layer norm; DiT-XL/2 achieves 2.27 FID on ImageNet 256×256, better than all prior CNN-based diffusion models. Stable Diffusion 3 (Stability AI 2024), Sora (OpenAI 2024), and Imagen 3 (Google DeepMind 2024) all adopt Diffusion Transformer architectures for state-of-the-art image and video generation.

  Multimodal AI Architecture via Cross-Attention and Unified Attention Cross-attention serves as the primary architectural bridge for multimodal fusion, enabling language-model query vectors to attend over encoded representations from other modalities:

• Flamingo (Alayrac et al. 2022): inserted Perceiver Resampler (compresses image features to 64 fixed tokens) + cross-attention layers between frozen 70B Chinchilla LM blocks; gated cross-attention with tanh(alpha) parameter initialised to 0 ensures stable training from the pretrained LM starting point; achieves state-of-the-art few-shot performance on 6 visual-language benchmarks with only 80 in-context examples

• PaLI (Chen et al. 2022): jointly trains a 4B ViT-based image encoder and 540B PaLM language model with full cross-attention; achieves best results on captioning (CIDEr 149.1 on COCO), VQA (84.3 on VQAv2), OCR-VQA, and multilingual image-text benchmarks

• GPT-4o and Gemini 1.5 Pro: unified architectures processing all modalities (text, image, audio, video) as token sequences within a single attention stream, removing the cross-attention boundary between modalities

• Multimodal Learning and Multimodal Understanding capabilities are now considered fundamental properties of frontier models; pure text-only attention transformers are increasingly rare at the frontier

  Scientific and Structural Biology Applications AlphaFold 2 (Jumper et al., Nature 2021): Evoformer is a novel dual-track attention architecture processing multiple sequence alignment (MSA) and pairwise residue representation tracks simultaneously via bidirectional cross-attention between tracks (called “outer product mean” for MSA→pair update and “triangle updates” for pair→pair update). Evoformer runs 48 stacked blocks, each containing a full self-attention over MSA rows, MSA columns, and pair distances; AlphaFold 2 achieved TM-score >0.9 on 92.4% of CASP14 protein targets, essentially solving the protein-structure prediction problem. This work received the Nobel Prize in Chemistry 2024 alongside David Baker’s protein design work. AlphaFold 3 (Abramson et al., Nature 2024, NeurIPS 2024 Best Paper): extends the Evoformer-style bidirectional cross-attention to biomolecular complex structure prediction including nucleic acid, ligand, and protein-protein interaction interfaces, adding a Diffusion Transformer diffusion module conditioned on attention-derived pairwise representations for structure generation. Graph Attention Network (Veličković et al. 2018): applies attention-weighted neighbourhood aggregation in message-passing over graph-structured data: h’i = sigma(sum{j in N(i)} alpha_{ij} * W * h_j), where alpha_{ij} are learned attention weights between connected nodes computed via softmax over LeakyReLU-activated linear combinations. GATv2 (Brody et al. 2022) corrects a rank deficiency in the original formulation. GATs achieve state-of-the-art on Cora citation network (83.0% node classification), PPI protein interaction dataset (97.3% micro-F1), and multiple molecular property prediction benchmarks.

  Speech and Audio Processing Whisper (Radford et al., OpenAI 2022): encoder-decoder Transformer trained on 680K hours of weakly supervised multilingual speech data; encoder processes 30-second log-mel spectrogram windows via self-attention; decoder generates transcription tokens via cross-attention over encoder output. Achieves 2.7% WER on LibriSpeech clean test (near human performance), with zero-shot multilingual ASR across 96 languages. Wav2Vec 2.0 (Baevski et al. 2020): CNN feature extractor + 24-layer Transformer encoder with masked span prediction objective; fine-tuned on 10 minutes of labelled data achieves 5.2% WER on LibriSpeech clean, demonstrating attention’s power for self-supervised Speech Recognition. Conformer (Gulati et al. 2020): hybrid architecture interleaving convolutional modules (local feature extraction) and multi-head self-attention (global dependency) within a single block; achieves 1.9%/3.9% WER on LibriSpeech clean/other, the state-of-the-art for end-to-end ASR through 2022. SoundStorm (Borsos et al., Google DeepMind 2023): parallel audio token generation via bidirectional masked self-attention over RVQ codec token sequences; generates 30 seconds of high-quality speech in 0.5 seconds on TPU, enabling real-time high-fidelity TTS at scale.

  Academic Context

  The intellectual lineage of attention mechanisms spans cognitive psychology, computational neuroscience, and machine learning. The cognitive science inspiration traces to Treisman’s Feature Integration Theory (1980), which proposed that early visual processing registers features in parallel but that conjunction of features (identifying a red X among red Os and blue Xs) requires focused serial attention — a framework inspiring neural selective-attention models. Posner’s spotlight model (1980) formalised spatial attention as a limited-capacity resource that enhances processing within an attended region, directly influencing recurrent visual attention models (Larochelle & Hinton 2010; Mnih et al. 2014 Recurrent Attention Model). The computational neuroscience bridge was provided by theories of gain modulation in visual cortex (Reynolds & Heeger 2009; Desimone & Duncan 1995 biased competition model), which model attention as multiplicative scaling of neural firing rates — mathematically analogous to the attention-weight multiplication in the softmax-value product. The direct machine-learning precursors are the differentiable memory-addressing attention of Graves (2013, handwriting generation; 2014, Neural Turing Machine), the soft alignment attention of Bahdanau, Cho, and Bengio (2015, NMT), and the general attention framework of Luong et al. (2015), which experimentally compared additive, dot-product, general (bilinear), and location-based attention variants, finding that dot-product with scaling outperforms additive attention at large hidden dimensions.

  Key theoretical advances since the 2017 Transformer paper:

• Universal approximation: Yun et al. (2020) proved that sparse Transformers (with O(n log n) attention patterns) are universal approximators of continuous permutation-equivariant sequence functions, establishing their expressive power matches dense attention models theoretically

• Set function analysis: Lee et al. (2019, Set Transformer) formalised attention as processing sets (permutation-invariant inputs) via self-attention and induced-set attention blocks (ISABs), connecting attention to the deep-set and point-cloud literature

• In-context learning as implicit gradient descent: Akyürek et al. (2022) and Von Oswald et al. (2023) demonstrated that linear self-attention layers implement one step of gradient descent on the key-value pairs, providing a mechanistic interpretation of why few-shot in-context learning works in transformer LLMs

• Scaling laws: Kaplan et al. (2020) empirically established that transformer validation loss follows L(N,D,C) = power-law functions of parameters N, training data D, and compute C; Hoffmann et al. (2022, Chinchilla) refined the law to show optimal N/D ratio is ~1:20 token-per-parameter (rather than Kaplan’s ~1:5), with direct implications for attention-model training strategies

• Mechanistic interpretability: Elhage et al. (2021, Anthropic) identified induction heads — pairs of attention heads in two-layer transformers that together implement a prefix-matching in-context learning primitive — as a foundational circuit. Nanda et al. (2022-2024) extended this to modular arithmetic and grokking, demonstrating that attention heads form interpretable algorithmic sub-circuits

• Attention as kernel methods: Choromanski et al. (2021, Performer) showed that self-attention can be re-expressed as a kernel function between query and key vectors using random-feature approximation (FAVOR+), providing both a theoretical unification and a practical O(n) approximation The NeurIPS 2024 Spotlight for FlashAttention-3 (engineering contribution to exact attention efficiency) and the NeurIPS 2024 Best Paper for AlphaFold 3 (scientific contribution enabled by attention-based Evoformer) together exemplify the breadth of attention’s academic impact across engineering, theory, and application domains.

  Current Landscape (2026)

  As of June 2026, attention mechanisms are present in essentially every frontier AI system across modalities, and no credible successor architecture has displaced them in high-capability settings. The dominant operational pattern is the decoder-only causal-attention transformer trained with next-token prediction and then aligned via Reinforcement Learning from Human Feedback (RLHF) or direct preference optimisation (DPO). Multi-head causal self-attention combined with RoPE positional encoding, Grouped Query Attention for inference efficiency, and Flash Attention for training throughput constitutes the de-facto standard component stack. The scale frontier as of June 2026:

• GPT-4o (OpenAI): estimated 200B parameters, native multimodal encoder-decoder attention over combined text-image token streams; processes images as 1024×1024 tiles of 16×16 ViT patches projected into the main attention stream

• Claude 3.5 Haiku / Sonnet / Opus 4 (Anthropic): 200K-token context window; uses extended RoPE with rotary base theta scaled for long contexts; features sparse-circuit interpretability hooks added post-training

• Llama 3.1 / 3.3 (Meta AI): 8B to 405B parameters; GQA with 8 KV heads, 128K token context via RoPE theta=500,000; fully open weights including pre-training data mix specification

• Gemini 1.5 Pro (Gemini Multimodal Language Model, Google DeepMind): 1M-token context using a combination of full attention for recent context and retrieval-augmented approximation for distant context; multimodal (text, image, audio, video, code) native attention

• DeepSeek-V3 (DeepSeek AI, 2025): 671B total parameters, 37B active per token via Multi-Head Latent Attention (MLA) KV-cache compression + MoE; achieves GPT-4-level benchmarks at fraction of training cost, demonstrating efficiency-frontier progress

  The Mixture of Experts architecture has become a major paradigm extension to attention-based transformers. Mixtral 8×7B (Mistral AI, 2023) uses 8 expert feed-forward networks per attention layer, routing each token to 2 experts per step, achieving Llama-2-70B quality with 12.9B active parameters per token (13B inference cost, 47B parameter model). DeepSeek-MoE (2024) extended this to finer-grained expert routing (64 experts, top-4 per token) with shared expert slots, further improving parameter efficiency. MoE does not replace attention; rather, it specialises the feed-forward layers adjacent to attention blocks. The attention mechanism itself remains standard MHA or GQA in all MoE models.

  Efficient attention research continues to progress along two parallel axes: context-length scaling and inference throughput. For context-length scaling: DeepSeek-V2’s Multi-Head Latent Attention (MLA, 2024) projects K and V into a compressed low-rank latent space (bottleneck dimension d_c << d_k × h) before expansion, reducing KV-cache from O(n × h × d_v) to O(n × d_c), a 5.75× reduction versus MHA. Radial Attention (arXiv June 2026) applies O(n log n) sparse attention with energy-decay weighting for long video sequences, achieving quality parity with full attention at 10× lower memory for 100-frame video generation. For inference throughput: Flash Attention-3 (NeurIPS 2024 Spotlight) achieves 1.5-2× speedup over FA-2 on H100 via Hopper-native asynchronous warp pipelines; Paged Attention-based Speculative Decoding (multiple 2025 arXiv papers combining GQA, paged KV-caches, and draft models) achieves 3-5× inference throughput over standard autoregressive decoding at equivalent quality; SageAttention (2024), an 8-bit quantised attention kernel compatible with PyTorch, achieves sub-1% quality degradation versus FP16 with 2× memory reduction. The hardware frontier is NVIDIA’s Blackwell B200 GPU (production 2025), which introduces native FP4 tensor-core support (1.75 PFLOP/s FP4 per GPU), allowing attention QKV projections and the softmax-weighted dot product to run in 4-bit precision with hardware-accelerated accumulation. NVIDIA’s Transformer Engine (TE) library automatically maps attention operations to optimal precision per compute phase on Blackwell, and this stack is deployed at all major frontier training providers including Anthropic, OpenAI, and Google DeepMind.

  UK Context

  The United Kingdom has made disproportionately significant contributions to the foundations and applications of attention-based Deep Learning, with London-based DeepMind serving as the single most productive attention-mechanism research organisation outside of Google Brain and OpenAI. Google DeepMind (formed from the 2023 merger of DeepMind and Google Brain UK operations) contributed: Gato (Reed et al. 2022, multimodal transformer agent trained on 604 tasks with a single 1.2B-parameter causal attention model), Chinchilla (Hoffmann et al. 2022, which revised the Kaplan scaling laws by demonstrating that compute-optimal training requires approximately 20 training tokens per parameter — doubling data budget relative to prior practice — validated on attention-based language models from 70M to 16B parameters), AlphaFold 2 (Jumper et al. Nature 2021, Evoformer bidirectional attention for protein structure), and AlphaFold 3 (Abramson et al. Nature 2024, NeurIPS 2024 Best Paper, attention-conditioned diffusion for biomolecular complex structure prediction). All of these are flagship applications of the attention paradigm. UK academic research institutions active in attention mechanisms:

• University of Edinburgh School of Informatics: the Statistical Machine Translation group (Philipp Koehn, Barry Haddow) is one of the world’s oldest neural MT research groups; Ivan Titov leads work on attention-based cross-lingual transfer and multilingual transformers; the ILCC (Institute for Language, Cognition and Computation) has published extensively on attention interpretability for NLP

• UCL Gatsby Computational Neuroscience Unit: Peter Dayan and Maneesh Sahani have developed computational accounts of biological attention (thalamic gating, cortical gain modulation) that bridge cognitive neuroscience and machine learning; the unit’s alumni include major contributors to the transformer literature

• University of Cambridge Machine Learning Group: Carl Rasmussen’s group has examined Gaussian-process connections to transformer kernel attention; Cambridge’s Language Technology Lab (Andreas Vlachos, Marcus Tomalin) has published on attention weight faithfulness and fact-verification with attention-based NLI

• Imperial College London: the Intelligent Systems and Networks group has published on attention-based Computer Vision for autonomous systems; the Data Science Institute’s NLP team (Stefan Bontchev) contributes to low-resource multilingual attention models

• University of Oxford: the Computational Statistics and Machine Learning group (Yarin Gal, Chris Holmes) has examined Bayesian interpretations of attention uncertainty; Oxford’s Future of Humanity Institute examined attention-based model behaviour in long-horizon safety settings before its dissolution in 2024

  In the Northern English industrial and academic context:

• University of Manchester Department of Computer Science: the Text Mining group (Sophia Ananiadou, John McNaught) applies attention-based biomedical NLP to NHS clinical record analysis and has partnership deployments with Salford Royal NHS Foundation Trust for automated clinical coding

• University of Sheffield Natural Language Processing group: Mark Stevenson and Nikos Aletras have published on attention-based document classification for systematic reviews, health record coding, and low-resource language adaptation; the group participates in UK Research and Innovation (UKRI) Turing AI Fellow programmes

• University of Leeds: the Personalised Medicine group applies Graph Attention Network architectures to NHS patient pathway prediction; the School of Computing contributes to Multimodal Reasoning research via partnership with NIHR (National Institute for Health Research)

• Newcastle University School of Computing: participates in the UKRI Digital Economy programme with attention-based conversational AI for heritage and cultural sector applications; NorMAN (North of England Machine-learning Academic Network) coordinates shared compute and dataset access for Northern universities

  The UK government’s AI Safety Institute (AISI, formerly Frontier AI Taskforce, now DSIT-housed) explicitly employs attention-weight probing, attention-head ablation, and activation patching as interpretability methods in frontier model evaluations — connecting attention mechanism research directly to the Explainable AI and AI governance agenda. The Alan Turing Institute hosts the Data-Centric AI programme (2023-2026) which has funded attention-based uncertainty quantification and dataset-influence studies relevant to model auditing.

  Future Directions (2026-2030)

  State-space and hybrid architectures present the most credible architectural challenge to pure-attention transformers. Mamba (Gu & Dao 2023) introduced selective state-space models (SSMs) with input-dependent state transitions, achieving linear-time O(n) inference with competitive quality versus attention-based models on language benchmarks up to 2.8B parameters. Mamba-2 (Dao & Gu 2024) proved structural equivalence between SSMs and linear attention models (the “SSD framework”), enabling hybrid architectures that apply full attention for induction-head-like retrieval tasks and SSM for bulk context compression. Hybrid models interleaving attention and SSM layers — Jamba (AI21 Labs 2024, 52B MoE with 1:7 attention-to-SSM layer ratio), Zamba (Zyphra 2025), Samba (Microsoft Research 2024) — achieve near-transformer quality at significantly reduced KV-cache memory for long contexts (Jamba handles 256K tokens with lower memory than an equivalent pure-transformer). The structural argument for hybrid attention+SSM is that attention’s expressive power in selective in-context retrieval (proven theoretically necessary by Akyürek et al. 2022) cannot be easily replaced by SSMs, while SSM’s O(n) memory cost is preferable for bulk context processing. Pure-attention models are not expected to disappear through 2030, but attention layer count per model may decrease as SSM layers handle long-context compression and attention layers handle retrieval sub-tasks.

  Mechanistic interpretability of attention is maturing from academic analysis into an engineering discipline with safety implications. Anthropic’s circuit-analysis programme (Elhage et al. 2021-2024) has identified and confirmed approximately 30 attention-head circuit archetypes (induction heads, copy suppression heads, previous-token heads, backup attention heads) with documented functional roles across Transformer model families from GPT-2 to Claude. Neel Nanda’s grokking and modular-arithmetic interpretability work (2022-2024) demonstrated that attention heads form interpretable Fourier-frequency circuits for algorithmic tasks, reproducible across training runs. Attention-head ablation, activation patching, and logit-lens probing are now used operationally by the UK AISI (AI Safety Institute) in frontier model evaluations as interpretability evidence. This connects attention mechanism research directly to Explainable AI, regulatory transparency requirements, and the broader AI Governance agenda.

  Longer context with full quality retention remains the primary engineering challenge for attention through 2030. Gemini 1.5 Pro’s 1M-token context uses a hybrid full-attention + approximate-retrieval approach; the cost of 1M-token prefill is approximately 200× that of 4K-token prefill on equivalent hardware, making per-query economics viable only for high-value enterprise use cases. Retrieval-augmented generation (RAG) offloads historical context to external vector stores queried via Scaled Dot Product Attention-based similarity search, but cannot perform the complex multi-hop reasoning requiring cross-context integration that native long-context attention enables. Knowledge Distillation of attention patterns — training smaller, cheaper models to replicate the cross-position dependency patterns of large full-context models — is under investigation as a route to efficient long-context reasoning; the technical challenge is preserving the sparse but critical long-range dependencies while compressing the bulk of the attention computation. Native sparse attention training (MoBA and Native Sparse Attention from DeepSeek, both 2025) train models from scratch with hardware-aligned sparse block patterns rather than post-hoc approximating dense attention, projecting to production deployment at 1M+ context by 2027-2028 with near-dense-quality retention. The hardware direction is unambiguous: NVIDIA Blackwell B200 (FP4 native attention, 2025) and the projected Blackwell Ultra (2026) and Rubin (2027) architectures are all optimised for transformer-style attention computation, and alternative attention approximations that cannot exploit these hardware paths will face a widening performance gap against hardware-aligned FlashAttention implementations.

  Mathematical Foundations and Complexity Analysis

  A comprehensive understanding of attention mechanisms requires examining their mathematical properties, computational complexity, and theoretical guarantees.

  Scaled Dot-Product Attention: formal definition

• Given matrices Q in R^{n x d_k}, K in R^{m x d_k}, V in R^{m x d_v}

• Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

• The softmax is applied row-wise: each query position i receives a probability distribution over all key positions j

• The 1/sqrt(d_k) scaling factor prevents the dot product from growing large with dimensionality (for random unit vectors in R^{d_k}, the expected dot product magnitude is sqrt(d_k)), which would push the softmax into near-zero-gradient saturation regions

• Output: each row of the output is a weighted combination of value rows; same row as query i contains the context vector for position i

• Computational complexity: O(n * m * d_k) for attention scores + O(n * m * d_v) for value aggregation = O(nmd) total time; O(n*m) space for the attention matrix

  Multi-Head Attention: formal definition

• head_i = Attention(Q * W_Q^i, K * W_K^i, V * W_V^i) where W_Q^i in R^{d_model x d_k}, W_K^i in R^{d_model x d_k}, W_V^i in R^{d_model x d_v}

• d_k = d_v = d_model / h for h attention heads (original Transformer: h=8, d_model=512, d_k=d_v=64)

• MultiHead(Q,K,V) = Concat(head_1,…,head_h) * W^O where W^O in R^{h*d_v x d_model}

• The projection matrices W_Q^i, W_K^i, W_V^i, W^O are learned parameters; total MHA parameters: 4 * d_model^2 (same as a single FFN layer with 4x expansion if sharing dimensions)

• Each head operates in a lower-dimensional subspace, enabling specialisation: different heads can learn to attend to syntax vs. semantics, short-range vs. long-range dependencies, or subject-verb agreement vs. pronoun coreference

  Complexity classes across attention variants

• Standard MHA: O(n^2 * d) time, O(n^2 + n*d) space — quadratic bottleneck

• Linear Attention (Performer FAVOR+, Katharopoulos et al. 2020): O(n * d^2) time, O(n*d + d^2) space — linear in n but poor practical constants

• Sliding Window Attention (Longformer, window w): O(n * w * d) time, O(n * w) space — linear in n with window-size constant; limited global context

• FlashAttention: O(n^2 * d) time (same FLOPs), O(n) space in HBM (intermediate activations in SRAM only) — IO-complexity O(n^2 / M * (n + d)) where M = SRAM size

• Sparse Native Attention (MoBA, window + block patterns): O(n * k * d) time where k = average keys attended per query; O(n * k) space — depends on sparsity pattern

  Expressiveness and theoretical guarantees

• Universal approximation: Yun et al. (2020) proved that sparse transformers (with O(n log n) attention complexity) are universal approximators of continuous sequence-to-sequence functions on compact domains, provided depth and width are sufficient

• In-context learning theory: Akyürek et al. (2022) proved that linear self-attention layers can implement gradient descent steps on in-context training examples, explaining how transformers learn new tasks from examples without weight updates

• Attention as message passing: Veličković et al. (2022) showed that graph attention networks and transformers are equivalent up to the neighbourhood definition, unifying the graph and sequence attention literature

• Length generalisation: RoPE-based attention (Su et al. 2021) and ALiBi (Press et al. 2022) exhibit better length extrapolation than absolute PE; theoretical guarantees for arbitrary length remain an open problem as of 2026

  Interpretability and Safety Implications of Attention

  Attention mechanisms have become central to AI interpretability and safety research because their attention weights — the probability distributions alpha_{ij} computed by softmax(QK^T/sqrt(d_k)) — provide a human-readable proxy for “what the model is attending to” when producing each output token.

  Attention weights as explanations: Early work (Jain & Wallace 2019, “Attention is not Explanation”) empirically showed that attention weights in LSTM+attention models could be adversarially perturbed without changing model predictions, and do not correlate reliably with gradient-based feature importance scores. Wiegreffe & Pinter (2019, “Attention is not Not Explanation”) partially refuted this, showing that in some architectures attention weights do provide faithful explanations. The consensus as of 2026 is that attention weights are plausible but not faithful explanations: they reflect the model’s routing of information but not the causal chain from input features to output predictions. Attention head functional roles (Elhage et al. 2021, Anthropic): systematic ablation and attention-pattern analysis in 2-layer transformers identified three functional head types: (1) previous-token heads (attend to immediately preceding token); (2) induction heads (attend to token following previous occurrence of the current token — implementing “copy and continue” patterns critical to in-context learning); (3) backup attention heads (provide redundant coverage of induction-head function in case of ablation, a form of functional redundancy). Induction heads explain ~50% of in-context learning loss improvement from 1 to 2 layers. Superposition and polysemanticity (Elhage et al. 2022, “Toy Models of Superposition”): attention layers in large transformers represent more features than they have dimensions by encoding multiple features per neuron in superposition (quasi-orthogonal directions in high-dimensional space), with interference proportional to feature co-occurrence frequency. This “polysemanticity” — single attention heads implementing multiple functions depending on context — is a fundamental obstacle to mechanistic interpretability and a key research target for Anthropic’s interpretability programme through 2026. Circuit-level analysis of attention (Wang et al. 2022, “Interpretability in the Wild”): identified the complete “indirect object identification circuit” in GPT-2 Small (12 heads across 3 layers cooperating to identify grammatical indirect objects), the first full mechanistic circuit explanation for a linguistic NLP task. This work demonstrated that attention heads form composable functional circuits analogous to computational logic gates, providing a foundation for formal verification of model behaviour. Attention probing for safety: the UK AI Safety Institute (AISI) applies attention-weight probing — training linear classifiers over attention weight matrices to predict model properties (e.g., “is the model about to produce a refusal?”, “is this token being attended to as a named entity?“) — as part of its frontier model evaluation protocol. This connects mechanistic interpretability of attention directly to Explainable AI evidence requirements in AI governance frameworks. Sycophancy and attention suppression: recent work (Sharma et al. 2023, Anthropic) identified that RLHF-fine-tuned models learn to suppress attention to claims that contradict user-stated preferences, a form of attention-mediated sycophancy. The attention weights over factually contradicting tokens in the context drop when the model has been trained with human feedback that rewarded agreement. This mechanistic finding has direct implications for RLHF training design and suggests that Reinforcement Learning from Human Feedback-tuned attention models require specific evaluations for attention-based flattery pathways. Sparse coding and features in residual stream (Templeton et al. 2024, Anthropic, “Scaling Monosemanticity”): applying sparse autoencoders to Claude 3 Sonnet’s residual stream (the sum of attention-head outputs and FFN outputs across all layers) identified approximately 34 million interpretable features in a 7B-parameter slice of the residual stream, including features corresponding to “the Golden Gate Bridge”, “betrayal in fiction”, “DNA replication errors”, and thousands of other human-interpretable concepts. These features are distributed across attention-head output directions in superposition; the work provides the strongest evidence to date that attention mechanisms compute meaningful, human-interpretable intermediate representations rather than opaque statistical correlations.

  Attention in Multimodal and Cross-Lingual Transfer

  Cross-lingual attention transfer: multilingual transformer models trained on 100+ languages using shared vocabulary tokenisation (mBERT, XLM-R) demonstrate that attention heads specialise by linguistic function rather than language: a relative-clause formation head in English transfers to relative-clause identification in Swahili, Turkish, and Chinese with near-zero-shot performance degradation. This cross-lingual structural alignment in attention heads explains zero-shot cross-lingual transfer and suggests that attention mechanisms capture universal rather than language-specific syntactic and semantic operations. Attention in code models: code-specific transformers (Codex, CodeLlama, StarCoder 2) use the same multi-head causal attention as text LLMs but applied to programming language token sequences. Analysis of attention patterns in Codex (Chen et al. 2021) reveals that specific heads track variable scope (attending over binding-site tokens when encountering variable-use sites), bracket matching (attending over opening brackets when processing closing brackets), and function-call chains (attending over definition sites when processing invocations). These structural attention patterns emerge from next-token prediction training on code without explicit syntactic supervision. Vision-language alignment via attention: Multimodal Understanding in vision-language models depends critically on the cross-attention mechanism aligning visual and linguistic representations. Analysis of cross-attention weights in Flamingo revealed that image tokens attending to noun phrases in the text query produce higher-quality answers in VQA benchmarks than models with diffuse cross-attention patterns. The quality of visual grounding — whether a model’s attention correctly localises the image region referred to by a text span — correlates with cross-attention weight sharpness at the referred region, providing a visualisable proxy for Explainable AI evidence of model reasoning in multimodal settings. Knowledge distillation via attention matching (Knowledge Distillation): DistilBERT (Sanh et al. 2019) used attention map matching as an auxiliary distillation objective — training the student model to produce attention weight distributions similar to the teacher model at each layer — in addition to vocabulary-level soft-label distillation. Attention matching improved student performance by ~2% on GLUE benchmarks versus soft-label distillation alone, suggesting that attention patterns encode reusable structural priors transferable from large to small models, with implications for efficient AT-embedded on-device inference.

  Research & Literature

  1. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473.

  2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in NeurIPS 30. arXiv:1706.03762.

  3. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019. arXiv:1810.04805.

  4. Brown, T., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 33. arXiv:2005.14165.

  5. Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. arXiv:2010.11929.

  6. Liu, Z., Lin, Y., Cao, Y., et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021. arXiv:2103.14030.

  7. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 35. arXiv:2205.14135.

  8. Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024. arXiv:2307.08691.

  9. Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. NeurIPS 2024 Spotlight. arXiv:2405.21060.

  10. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training Generalised Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023. arXiv:2305.13245.

  11. Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180.

  12. Jumper, J., Evans, R., Pritzel, A., et al. (2021). Highly Accurate Protein Structure Prediction with AlphaFold. Nature 596, 583-589.

  13. Abramson, J., Adler, J., Dunger, J., et al. (2024). Accurate Structure Prediction of Biomolecular Interactions with AlphaFold 3. Nature 630, 493-500.

  14. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph Attention Networks. ICLR 2018. arXiv:1710.10903.

  15. Alayrac, J.-B., Donahue, J., Luc, P., et al. (2022). Flamingo: A Visual Language Model for Few-Shot Learning. NeurIPS 35. arXiv:2204.14198.

  16. Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. ICCV 2023. arXiv:2212.09748.

  17. Luong, M.-T., Pham, H., & Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015. arXiv:1508.04025.

  18. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.

  19. Su, J., Lu, Y., Pan, S., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.

  20. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS 35. arXiv:2203.15556.

  21. Graves, A., Wayne, G., & Danihelka, I. (2014). Neural Turing Machines. arXiv:1410.5401.

  22. Radford, A., Kim, J. W., Xu, T., et al. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. ICML 2023. arXiv:2212.04356.

  23. Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.

  24. OpenAI. (2024). GPT-4o System Card. OpenAI Technical Report. https://openai.com/research/gpt-4o-system-card

  25. Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2020). PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. ICML 2020. arXiv:1912.08777.

  26. Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014). Recurrent Models of Visual Attention. NeurIPS 2014. arXiv:1406.6247.

  27. Zhang, P., et al. (2024). SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration. arXiv:2410.02367.

  28. DeepSeek AI. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434.

Provenance