Automatic Differentiation

Automatic differentiation (AD) is a family of computational techniques for evaluating the derivative of a function specified by a computer program, by systematically applying the chain rule to elementary arithmetic operations rather than through symbolic algebra or finite-difference approximation. It operates in two principal modes: forward mode, which propagates tangent values alongside primal values, and reverse mode (backpropagation), which accumulates gradients in a backward pass over a recorded computation graph. AD produces machine-precision derivatives at a cost linear in the number of program operations, making it the computational backbone of modern deep learning frameworks.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:ForwardModeDifferentiation))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:Backpropagation))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:ComputationGraph))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:WengertTape))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:ChainRule))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:VJPOperator))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:hasPart ai:JVPOperator))

Dependency Relationships

SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:requires ai:DifferentiableFunction))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:requires ai:ComputationGraph))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:dependsOn ai:LossFunction))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:dependsOn ai:ActivationFunction))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:dependsOn ai:NeuralNetwork))

Capability Relationships

SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:Backpropagation))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:GradientDescent))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:DeepLearning))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:ScientificMachineLearning))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:ProbabilisticProgramming))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:DifferentiableRendering))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:PhysicsSimulation))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:enables ai:ReinforcementLearning))

Implementation Relationships

SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:implements ai:Backpropagation))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:implements ai:GradientDescent))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:uses ai:MachineLearningFramework))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:uses ai:JAX))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:uses ai:ComputationGraph))

Reduction Relationships

SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:reducesTo ai:NumericalMethod))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:contrastsWith ai:SymbolicDifferentiation))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:contrastsWith ai:FiniteDifferences))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:bridgesTo ai:DifferentiableRendering))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:bridgesTo ai:RoboticsControl))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:relatedTo ai:AutomatedReasoning))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:relatedTo ai:ProbabilisticProgramming))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:supports ai:TransformerArchitecture))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:supports ai:LargeLanguageModels))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:uses ai:GradientCheckpointing))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:uses ai:GradientClipping))
SubClassOf(ai:AutomaticDifferentiation
  ObjectSomeValuesFrom(ai:uses ai:GradientAccumulation))

About

Automatic differentiation occupies a singular position in the modern computational sciences: it is simultaneously a classical technique from numerical analysis, the technical foundation of the Deep Learning revolution, and an expanding infrastructure layer for scientific computing, physical simulation, and formal reasoning. The core insight — that any differentiable program can be decomposed into elementary operations each with a known derivative, and that the chain rule composes these derivatives without loss of precision — was first formalised by Robert Edwin Wengert in 1964 in his landmark paper on a simple automatic derivative evaluation program. Wengert described forward-mode accumulation over what he called a “Wengert list” (now universally called a computation tape or Computation Graph). The reverse-mode formulation, which is what underlies Backpropagation, was independently discovered multiple times: by Seppo Linnainmaa in his 1970 Master’s thesis at the University of Helsinki (published 1976), by Paul Werbos in his 1974 Harvard PhD thesis in the context of neural learning, and re-popularised by Rumelhart, Hinton, and Williams’ landmark 1986 Nature paper that demonstrated its utility for training multi-layer neural networks and ignited the neural network revival that culminated in modern Deep Learning.

The mathematical structure of reverse-mode AD is most elegantly understood through the lens of dual numbers and the cotangent space. In forward mode, a scalar function y = f(x₁, x₂, …, xₙ) is evaluated simultaneously with its directional derivative: each intermediate value v is augmented with a tangent value v̇ = ∂v/∂xⱼ for a chosen input direction xⱼ (the “seed” direction). For a composition of elementary operations v = g(u₁, …, uₖ), the tangent propagates as v̇ = Σᵢ (∂g/∂uᵢ) × u̇ᵢ by the chain rule, computed alongside the primal evaluation with negligible overhead (a constant factor of approximately 2–3x per operation). Full Jacobian computation costs O(n) forward passes (one per input dimension). In reverse mode, the forward pass records the computation graph (the “tape”), and a backward pass propagates adjoint values (also called cotangents or sensitivities): for each node v in the graph, the adjoint v̄ = ∂L/∂v represents how the output loss L depends on the intermediate value v. For a composition v = g(u₁, …, uₖ), the adjoint propagates backwards as ūᵢ += (∂g/∂uᵢ) × v̄ for each input uᵢ — the transpose of the forward Jacobian-vector product. Full gradient computation costs O(m) backward passes (one per output), so for m = 1 (scalar loss, the standard case in neural network training) the full gradient over all n inputs costs exactly O(1) backward passes regardless of n. The memory overhead of reverse mode is the storage of the computation tape — all intermediate values computed during the forward pass must be retained for the backward pass, costing O(number of operations) memory. For very deep networks with billions of activations, this becomes a bottleneck addressed by Gradient Checkpointing (recomputing selected activations during the backward pass rather than storing them, trading O(n) memory for O(√n) memory at a factor-of-2 increase in compute).

The most theoretically elegant formalisation of AD comes from categorical denotational semantics: Conal Elliott (2018) showed that forward-mode AD corresponds to the functor sending a differentiable function f: A → B to its derivative Df: A → (B × (A ⊸ B)), where A ⊸ B is the linear map space, and that reverse-mode AD corresponds to the transpose of this functor using the duality between tangent and cotangent spaces. This categorical view, extended by the CHAD framework (Vákár & Smeding 2021, University of Edinburgh) to higher-order functional programming languages with first-class functions and recursion, provides a mathematically rigorous foundation that guarantees the correctness of AD systems and guides the design of type systems for differentiable programming languages. The LAFI (Languages for Inference) workshop at POPL 2024 gathered 200+ researchers exploring these formal foundations, with active cross-pollination between programming language theory, probabilistic programming, and machine learning.

The JAX library (Google Brain, 2018; open-sourced 2020) represents the state of the art in AD system design as of 2026, introducing a purely functional AD interface based on composable function transformations: jax.grad for reverse-mode gradients, jax.jvp for forward-mode JVPs, jax.vjp for reverse-mode VJPs, jax.hessian for second-order matrices computed via forward-over-reverse composition (vjp(jvp(…)) or jacfwd(jacrev(…))), jax.vmap for vectorised batched AD computing all Jacobian columns or rows in parallel (SIMD over the batch dimension), and jax.jit for Just-In-Time Compilation of the entire differentiated computation graph to XLA IR for GPU/TPU kernel fusion and execution. The functional purity of JAX (no in-place mutation, immutable pytree arrays, pure functions with no side effects) enables arbitrary composition of transformations: one can compute grad(jit(vmap(grad(f)))) and JAX will correctly compile the doubly-differentiated, vectorised, JIT-compiled function to efficient hardware code. The 2025 MLPerf Training benchmark reported JAX SPMD (Single-Program Multiple-Data) setups achieving a 3.2x throughput advantage over PyTorch FSDP on H100 clusters for LLM training (72B parameter scale), and Optax (the JAX optimiser library, covering Adam, AdamW, SGD, Adagrad, Yogi, and 20+ other methods) was adopted by 65% of 2026 MLPerf submitters for large language model and vision tasks. PyTorch 2.x (torch.compile using dynamo tracing and TorchInductor code generation, 2023) responded with comparable performance gains for eager-mode workflows and closed much of the compilation performance gap with JAX for standard training pipelines. The Enzyme compiler (MIT, William Moses 2020) takes a fundamentally different architectural approach: operating at the LLVM IR level rather than the Python level, it differentiates programs in any LLVM-compiled language (C, C++, Rust, Fortran, Julia, Swift, Python via Numba) by performing AD directly on the optimised low-level intermediate representation, enabling compiler-level dead-code elimination, alias analysis, memory optimisation, and parallelisation for reverse-mode AD — particularly valuable for scientific computing and HPC simulation where codes are written in C++ or Fortran and need differentiation without rewriting. By 2025, Enzyme was integrated into LLVM mainline and supports MLIR-level AD passes targeting diverse hardware backends including FPGAs (AMD Xilinx), neuromorphic arrays (Intel Loihi 2), and quantum-classical hybrid systems (PennyLane’s Catalyst compiler).

The Three Modes of Differentiation: Forward, Reverse, and Beyond

Beyond the two classical modes, modern AD research has identified and implemented several additional differentiation modes with distinct computational profiles. Forward-over-reverse (or “reverse-over-forward”) AD computes Hessian-vector products at O(1) forward-pass cost per vector, enabling curvature approximations without storing the full n×n Hessian matrix; this is the computational basis of K-FAC (Kronecker-Factored Approximate Curvature), KFAC-JAX, and the shampoo optimiser used in training large language models at Google. Vectorised Jacobian computation using vmap(jacfwd(...)) or vmap(jacrev(...)) computes the full m×n Jacobian matrix at a cost of O(min(n,m)) passes, which is more efficient than n forward passes or m backward passes when n and m are of comparable magnitude — relevant for scientific computing, control systems, and sparse neural architectures. Operator overloading implementations (the approach used by PyTorch autograd and TensorFlow) intercept standard arithmetic operations at Python runtime to record the Computation Graph dynamically, while source-transformation implementations (Tapenade for Fortran, Enzyme for LLVM IR, Dex for a functional array language) transform the program source or IR into a differentiated equivalent, enabling whole-program optimisation that operator overloading cannot achieve. The tension between dynamic (define-by-run) and static (tracing-based) computation graph construction reflects a fundamental trade-off: dynamic graphs support Python-native control flow (if/else, loops with data-dependent iteration counts) but prevent global compilation optimisation; static graphs (JAX tracing under jit, TensorFlow 1.x graph mode, ONNX export) enable whole-graph kernel fusion and memory planning but require the control flow to be captured symbolically, typically via abstract value tracing or graph construction APIs.

Second-order AD — computing Hessian matrices or Hessian-vector products — has moved from research curiosity to practical tool between 2023 and 2026. The January 2025 paper “Sparser, Better, Faster, Stronger” (arXiv:2501.17737) addresses AD for sparse Jacobians and Hessians: most scientific computing Jacobians are highly sparse (finite-difference stencils, graph neural network message passing, contact mechanics), and exploiting sparsity structure reduces both compute and memory from O(n²) dense to O(nnz) sparse, where nnz is the number of non-zeros. The KFAC-JAX library (Google DeepMind, 2023) implements the K-FAC second-order optimiser that approximates the inverse Fisher information matrix using Kronecker factorisation of layer-wise gradient outer products, enabling curvature-scaled gradient updates that converge faster than Adam on transformer training. The somax library provides stochastic second-order methods in JAX-native format for research experimentation with Newton-type and quasi-Newton methods at scale. These developments signal that second-order optimisation — long theoretically superior but practically impractical for large models — will become computationally competitive as hardware costs fall and AD-computed curvature information becomes cheaper to exploit.

Computational Cost Reference

A quick-reference summary of the cost characteristics of the principal AD modes and operations:

Operation	Forward Mode (JVP)	Reverse Mode (VJP)	Notes
Scalar gradient ∂L/∂x₁	1 forward pass	1 backward pass	Both O(1) for scalar output
Full gradient ∇L ∈ ℝⁿ	n forward passes	1 backward pass	Reverse preferred when n ≫ 1
Full Jacobian J ∈ ℝᵐˣⁿ	n forward passes	m backward passes	Use vmap(jacfwd or jacrev)
Hessian H ∈ ℝⁿˣⁿ	n forward-over-reverse	n reverse-over-forward	O(n²); use HVP for large n
Hessian-vector product Hv	1 forward-over-reverse	1 reverse-over-forward	O(1) per vector v
Memory (reverse mode)	O(1) per step	O(n_ops) tape	Gradient Checkpointing: O(√n)
Sparse Jacobian nnz entries	nnz forward passes	nnz backward passes	Sparsity exploitation (arXiv:2501.17737)
Jacobian via vmap+jacfwd	n forward (vectorised)	—	min(n, m) passes with SIMD

These cost characteristics guide architectural choices: reverse mode dominates for scalar losses (ML training); forward mode is preferred for low-dimensional inputs (sensitivity analysis, control); vectorised Jacobian computation (vmap(jacrev(...))) is standard for intermediate-size Jacobians in SciML applications.

Components / Architecture

The internal architecture of an automatic differentiation system comprises the following layers:

1. Primitives Library

A catalogue of elementary differentiable operations — arithmetic (+, -, ×, ÷), comparison, trigonometric (sin, cos, tan, atan2), exponential (exp, log, pow), and linear algebra routines (matmul, conv, norm, solve) — each annotated with their VJP/JVP rules. In PyTorch autograd, these are torch.autograd.Function subclasses with forward and backward static methods; in JAX, they are registered jax.core.Primitive objects with associated jvp and transpose rules that permit arbitrary composition under the transformation machinery.

2. Computation Graph Construction

During the forward pass, the AD system records computational structure via one of two paradigms:

• Dynamic (define-by-run): PyTorch eager mode and TensorFlow 2.x with GradientTape build a fresh directed acyclic computation graph on each forward call by intercepting Python operator overloads. Every Tensor carries a grad_fn attribute pointing to the operation that produced it. This enables Python-native control flow (arbitrary if/else, while loops with data-dependent iteration counts) at the cost of preventing global compilation optimisation.

• Static (tracing-based): JAX under @jit, TensorFlow 1.x tf.Graph, and ONNX export trace the computation once using abstract value semantics to produce a static computation graph compiled to XLA or TensorRT. Control flow must be expressed via jax.lax.cond, jax.lax.while_loop, or tf.cond/tf.while_loop rather than Python conditionals, but static graphs enable whole-graph kernel fusion, memory planning, and cross-device sharding optimisation unavailable to dynamic graphs.

  3. Tape / Gradient Tape

  The Wengert tape is the data structure recording the operation sequence and intermediate values needed for the backward pass. In PyTorch, each forward operation appends a Node to the thread-local gradient computation graph; in TensorFlow, tf.GradientTape wraps the forward computation in a context manager accumulating operations. Memory for the tape — storing all intermediate activations — is the dominant cost of reverse-mode AD in large models, motivating Gradient Checkpointing (also called activation recomputation or rematerialisation): rather than storing all activations, only a subset of “checkpoint” activations are stored; during the backward pass, other activations are recomputed from the nearest upstream checkpoint. The classical scheme (Chen et al. 2016, arXiv:1604.06174) achieves O(√n) memory at a 33% increase in compute for depth-n computation graphs.

  4. VJP/JVP Dispatch

  The mechanism associating each registered primitive with its derivative rule and applying it during backward (VJP) or forward tangent propagation (JVP):

• In JAX, jax.linear_util.WrappedFun and the jax.interpreters.ad module implement forward-mode (jvp) and reverse-mode (vjp) transformations as composable interpreter passes over the JAX jaxpr (intermediate representation). The jax.vjp function returns both the primal output and a residual (the saved intermediate values needed for the backward pass), plus a callable vjp_fun that implements the VJP operator.

• In PyTorch, torch.autograd.Engine implements the backward pass as a C++ thread-pool executing graph nodes in topological sort order. accumulate_grad accumulates gradient contributions from multiple paths through the computation graph (when a tensor is used multiple times), and Node.apply() invokes the user-defined backward method.

  5. Gradient Accumulation, Clipping, and Reduction

  Gradient Accumulation across micro-batches (processing batches in sub-batches with loss.backward() accumulating .grad tensors before stepping the optimiser) enables effective large-batch training on memory-constrained hardware. Gradient Clipping — enforced by torch.nn.utils.clip_grad_norm_(parameters, max_norm) — projects the global gradient vector onto a ball of radius max_norm, preventing exploding gradient pathologies in deep recurrent or very deep feedforward networks. Distributed gradient reduction (AllReduce in data-parallel training) aggregates gradients across devices before the optimiser step; PyTorch DDP uses NCCL ring-AllReduce, while JAX uses jax.lax.pmean / psum under pmap.

  6. Higher-Order Differentiation

  Modern AD systems support nested differentiation for second-order optimisation and uncertainty quantification:

• jax.hessian(f) computes the full Hessian matrix ∂²f/∂xᵢ∂xⱼ as jacfwd(jacrev(f)) or jacrev(jacfwd(f)), with cost O(n²) — practical for small n (control systems, few-parameter physics models) but prohibitive for neural networks with millions of parameters.

• Hessian-vector products Hv = ∂(∇f · v)/∂x computed via forward-over-reverse (jvp over vjp) at O(1) cost per vector v, enabling conjugate-gradient Newton methods, natural gradient descent (Fisher information matrix-vector products), and influence function computation.

• Third-order derivatives (Jacobian of Hessian-vector products) arise in higher-order meta-learning and hyperparameter optimisation; JAX supports these through nested grad calls with no additional implementation overhead beyond the composition of existing primitives.

  7. Custom Derivative Registration

  Users register custom VJPs for non-differentiable or numerically ill-conditioned operations:

• @jax.custom_vjp decorates a function and provides alternative forward/backward implementations. Use cases include: argmax (not differentiable, replaced by softmax for gradient flow), SVD (numerically unstable backward pass for nearly-degenerate singular values, replaced by stabilised formula), implicit differentiation (differentiating the solution of an optimisation problem or ODE via the implicit function theorem rather than through the solver’s internal steps), and operations with analytically simpler gradients than the automatic chain-rule derivation (e.g., log-sum-exp, Cholesky solve).

• Custom derivative registration is essential for differentiating through external compiled code, sparse operations, and simulation loops where unrolled gradient computation would be prohibitively memory-intensive.

  Use Cases / Major Families

  1. Neural Network Training (Core Application)

  The canonical application of reverse-mode AD — computing gradients of loss functions over all parameters of neural networks from small MLPs through to trillion-parameter Large Language Models. Every modern training pipeline depends on AD: the Adam Optimiser (and AdamW, Adafactor, Lion, SOAP, and other variants) receives gradient tensors computed by AD and updates parameter tensors; Batch Normalisation layer gradients require AD through batch statistics; attention mechanisms in Transformer Architecture architectures require AD through softmax and scaled dot-product attention; Reinforcement Learning policy gradient methods (PPO, REINFORCE, SAC) require AD through stochastic policy log-probabilities. RLHF (Reinforcement Learning from Human Feedback) fine-tuning of Large Language Models requires AD through the entire model forward pass to compute KL-divergence gradients and reward model gradients simultaneously. Gradient Accumulation, Gradient Clipping, and Gradient Checkpointing are all infrastructure layers built on top of AD to make trillion-parameter training computationally feasible on finite hardware.

  Scale benchmarks: GPT-4 (estimated 1.8T parameters, mixture-of-experts) and Google Gemini Ultra (estimated 1.5T parameters) were trained with JAX-based or PyTorch-based reverse-mode AD on tens of thousands of TPUs/H100 GPUs over months of continuous training. A single parameter update step for a 70B parameter LLaMA 3 model requires computing and storing approximately 70B float32 gradient values (280GB), making Gradient Checkpointing and Gradient Accumulation non-optional.

  2. Scientific Machine Learning (SciML)

  Physics-informed neural networks (PINNs) embed physical PDE constraints into the Loss Function via AD-computed spatial and temporal derivatives: for a neural network u(x, t; θ) approximating the solution of the heat equation ∂u/∂t = α∇²u, the PDE residual loss L_PDE = ||∂u/∂t - α∇²u||² is computed by applying AD twice (first-order in t, second-order in x) to u(x, t; θ) with respect to spatial coordinates, then combining with boundary and initial condition losses. This approach requires no finite-difference mesh and naturally handles irregular geometries.

  The Julia SciML ecosystem (DifferentialEquations.jl, NeuralODE.jl, Zygote.jl, Enzyme.jl) and Python-based SciML (JAX + Diffrax for neural ODEs, Neural Tangent Kernel methods) are the dominant toolkits as of 2025–2026. The state-of-SciML report (June 2025) notes the 2024 joint Lux.jl/Enzyme.jl integration as a major advance, enabling full neural-network AD support within DifferentialEquations.jl. Applications span computational fluid dynamics (Navier-Stokes solutions), structural mechanics, geophysical inversion, weather prediction adjoint methods (used in data assimilation at the Met Office and ECMWF), and molecular dynamics.

  3. Differentiable Physics Simulation

  Differentiable physics simulators implement physical systems as differentiable programs, enabling policy gradient computation through simulation for Reinforcement Learning and inverse problem solving:

• JAX-MD (Schoenholz & Cubuk 2020, NeurIPS 2020): molecular dynamics in JAX, differentiating through MD trajectories to compute free energy sensitivities.

• JaxSim (IIT 2023): multi-body dynamics and contact mechanics for robot learning, computing policy gradients through rigid-body simulation.

• Brax (Google 2021): physics engine for robotic locomotion training, using JAX’s vmap for massively parallel environment rollouts.

• MuJoCo MJX (DeepMind/Google 2023): JAX-based implementation of MuJoCo differentiable physics for dexterous manipulation research.

• DaXBench (2022): deformable object manipulation via JAX-based differentiable FEM simulation.

  4. Differentiable Rendering

  Neural Radiance Fields (NeRF, Mildenhall et al. 2020), 3D Gaussian Splatting (Kerbl et al. 2023), and differentiable rasterisers (nvdiffrast, PyTorch3D, JAXRenderer) use reverse-mode AD to optimise 3D scene representations from 2D image observations:

• NeRF represents scenes as volumetric density and radiance functions parameterised by neural networks; reverse-mode AD differentiates the volume rendering integral to obtain gradients for network parameter updates.

• 3D Gaussian Splatting (winner of SIGGRAPH 2023 Best Paper) represents scenes as 3D Gaussians with learnable position, orientation, scale, opacity, and spherical harmonic colour; AD differentiates the α-blending rasterisation to optimise Gaussian parameters.

• Differentiable rasterisers enable single-image 3D reconstruction, augmented reality, and physics-based inverse rendering for material estimation.

  5. Probabilistic Programming

  Frameworks embed AD as the gradient engine for gradient-based Markov Chain Monte Carlo and variational inference:

• Hamiltonian Monte Carlo (HMC) and its adaptive variant NUTS (No-U-Turn Sampler, used in Stan and NumPyro) require the gradient ∇_θ log p(θ|data) to simulate Hamiltonian dynamics, enabling efficient sampling from complex high-dimensional posteriors. AD computes this gradient automatically for any differentiable probabilistic model.

• Variational inference (ELBO optimisation in Pyro, NumPyro, TensorFlow Probability) uses AD to compute ELBO gradients with respect to variational parameters, enabling scalable approximate Bayesian inference.

• Score-based diffusion models (DDPM, Score Matching) use AD to compute score function gradients ∇_x log p_t(x) for denoising, powering state-of-the-art image generation (Stable Diffusion, DALL-E 3, Imagen).

  6. Structural and Topology Optimisation

  AD-enabled sensitivity analysis replaces manual adjoint derivation in engineering design optimisation:

• JAXFEM and FEniCS-adjoint compute structural compliance sensitivities ∂C/∂ρᵢ with respect to material density fields ρ for density-based topology optimisation (SIMP method), enabling aerodynamic shape optimisation and lightweight structural design without manual adjoint code derivation.

• The FEniCS-adjoint library (dolfin-adjoint) annotates standard FEniCS finite-element solvers to enable automatic adjoint computation, used in geothermal energy optimisation, cardiac electrophysiology modelling, and ocean current simulation.

• DiffBreed (arXiv:2024, Bioinformatics 2025) treats multi-generational plant breeding schemes as differentiable programs, differentiating through genetic selection, recombination, and phenotypic evaluation to optimise breeding strategies end-to-end — demonstrating AD’s generalisation beyond purely computational domains.

  7. Second-Order Optimisation

  KFAC-JAX (Google DeepMind 2023), somax, and shampoo implement curvature-aware optimisers using AD-computed Hessian-vector products:

• K-FAC (Kronecker-Factored Approximate Curvature) approximates the inverse Fisher information matrix as Kronecker products of per-layer covariance factors, enabling natural gradient updates that account for parameter space curvature and converge faster than Adam on transformer architectures.

• Shampoo (Google 2018, adopted for LLM training at scale) approximates full-matrix preconditioning with Kronecker-factored preconditioners; combined with AD-computed gradients, it achieves 10–40% faster convergence than AdamW on language model training.

• The 2025 sparse AD paper (arXiv:2501.17737) reduces second-order computation from O(n²) dense Hessian to O(nnz) sparse for problems with sparse Jacobian structure, enabling practical Hessian computation for large-scale scientific computing PDE solvers.

  8. Quantum Computing

  PennyLane’s Catalyst compiler (2024) integrates Enzyme-based AD with quantum circuit parameter differentiation, combining the parameter-shift rule (exact gradient of quantum circuit expectation values with respect to gate parameters) with classical AD through pre/post-processing circuits, enabling end-to-end gradient-based optimisation of hybrid classical-quantum algorithms for variational quantum eigensolvers (VQE) and quantum machine learning circuits.

  Academic Context

  The intellectual lineage of automatic differentiation traverses three communities — numerical analysis, programming languages, and machine learning — each contributing distinct conceptual tools and theoretical frameworks.

  Numerical Analysis Tradition

  Wengert (1964) published the first computational implementation of forward-mode AD as a “simple automatic derivative evaluation program,” introducing the Wengert list (computation tape) as the data structure for recording elementary operations. Linnainmaa’s 1970 Helsinki Master’s thesis (published 1976) introduced the reverse accumulation algorithm as the basis for computing the “accumulated rounding error” of floating-point operations — the first description of what would later be called reverse-mode AD or backpropagation. Griewank (1989) formalised the operator-overloading approach to AD and, with Walther, produced the definitive textbook “Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation” (SIAM, 2nd ed. 2008), which systematically treats the complexity bounds (O(ω) forward mode, O(ω) reverse mode where ω is the operation count of the primal), the memory-time trade-offs of checkpointing, the theory of operator overloading versus source transformation, and the relationship to sparsity exploitation. The ad-hoc (Automatic Differentiation by Overloading in C++) and ADIFOR (AD for Fortran) systems of the 1990s demonstrated practical large-scale AD in scientific computing before the deep learning era.

  Programming Languages Theory

  Pearlmutter and Siskind (2008) developed a higher-order AD system operating on lambda calculus programs, using continuation-passing style transformation and first-class closures to implement both forward and reverse mode for programs with first-class functions and recursion — the first theoretically sound treatment of AD in a higher-order functional setting. Elliott (2018) provided a categorical denotational semantics: forward-mode AD corresponds to the functor D: Diff → Diff × Lin taking f: A → B to (f, Df): A → B × (A ⊸ B) (the function paired with its derivative, a linear map), while reverse-mode AD corresponds to the dual functor using the duality isomorphism A ⊸ R ≅ A* (cotangent). This categorical perspective, published at ICFP 2018, established AD as an instance of a more general structure theory and guided the design of type systems for differentiable programming languages. The CHAD (Combinatory Homomorphic Automatic Differentiation) framework (Vákár & Smeding, 2021, University of Edinburgh, published at PACMPL POPL) extended this categorical treatment to higher-order functional languages with general recursion, products, sums, and inductive types using a novel semantic framework of Cartesian reverse-differential categories. The JAX autodiff from a linear logic perspective (Danilecki et al., PACMPL 2025, doi:10.1145/3776654) connects JAX’s VJP/JVP design to the cut-elimination procedure of linear logic sequent calculi, providing a proof-theoretic interpretation of AD transformations.

  Machine Learning Community

  The crucial machine learning contributions were:

• Werbos (1974): first application of reverse-mode AD to neural networks, in a Harvard PhD thesis titled “Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences.”

• Rumelhart, Hinton, and Williams (1986): “Learning representations by back-propagating errors,” Nature 323, 533–536, the paper that popularised backpropagation and ignited the second neural network wave, demonstrating learning of internal representations in multi-layer networks.

• LeCun et al. (1989): convolutional neural networks trained via backpropagation, achieving 99%+ accuracy on handwritten digit recognition.

• Baydin, Pearlmutter, Radul, and Siskind (2018): “Automatic differentiation in machine learning: a survey,” JMLR 18(153), the definitive survey unifying forward/reverse mode, operator overloading, source transformation, and their connections to probabilistic programming and differentiable programming.

• Bradbury et al. (JAX team, Google 2018): JAX: composable transformations of Python+NumPy programs, operationalising composable AD transformations via the grad/jit/vmap/pmap transformation algebra.

• Paszke et al. (2019): PyTorch: An imperative style, high-performance deep learning library, NeurIPS 2019, the system that dominated ML research from 2019–2024.

• Moses and Churavy (2020): “Instead of rewriting foreign code for machine learning, automatically synthesize fast gradients,” SC 2020, introducing the Enzyme compiler for LLVM-level AD.

  Conference Venues and Community

  Primary venues include NeurIPS (Neural Information Processing Systems), ICML (International Conference on Machine Learning), ICLR (International Conference on Learning Representations) for machine learning AD research; POPL (Principles of Programming Languages) and PLDI (Programming Language Design and Implementation) for programming language theory; SIAM CSE (Computational Science and Engineering) and SC (Supercomputing) for scientific computing AD; and the LAFI (Languages for Inference) workshop series for the intersection of AD, probabilistic programming, and formal semantics. The AD community hosts the biennial International Conference on Algorithmic Differentiation (ICAD), which focuses on numerical analysis aspects, industrial applications, and tool development. Software ecosystems include the ad-hoc TAPENADE (Fortran/C source transformation, INRIA), ADOL-C (C++ operator overloading, Argonne National Laboratory), and Dex (a research functional array language with built-in AD semantics, Google Brain).

  The DL-ReasonSuite Connection

  An emerging academic intersection between AD and Automated Reasoning concerns the verification of AD implementations: just as formal methods verify the correctness of hardware arithmetic units, researchers are beginning to apply proof assistants to verify the correctness of AD transformation algorithms. The CHAD framework provides a proof of correctness for its higher-order AD macro transformation in the Agda proof assistant, and JAX’s VJP/JVP linear-logic interpretation (PACMPL 2025) provides a proof-theoretic basis for their compositional correctness. This convergence of AD correctness verification with Automated Reasoning and program verification tools represents a nascent but important research frontier, particularly as AD systems become infrastructure for safety-critical AI applications.

  Current Landscape (2026)

  As of mid-2026, automatic differentiation has matured from a research technique into critical infrastructure, embedded invisibly into every Machine Learning Framework and increasingly into scientific computing stacks. The PyTorch / JAX duopoly governs research: PyTorch retains a plurality share of ML research papers (approximately 55% in 2025 ICML/NeurIPS submissions) owing to its Pythonic ergonomics and extensive ecosystem, while JAX dominates at the frontier of large-scale distributed training and physical simulation, powered by XLA’s superior TPU/GPU kernel fusion. Google DeepMind’s Gemini training pipeline, Google’s TPU v5 deployments, and the Flax / Optax ecosystem exemplify JAX at scale; Meta AI’s PyTorch 2.x with torch.compile and FSDPv2 represents the competing paradigm. The Enzyme compiler project has seen rapid adoption in scientific computing HPC contexts where C++/Fortran codes require AD without Python runtime overhead — it is now integrated into the LLVM mainline and supports MLIR-level AD passes targeting FPGA and neuromorphic backends. The University of Southampton offers a dedicated graduate module COMP6258 “Differentiable Programming and Deep Learning,” reflecting the discipline’s academic canonisation. In the SciML ecosystem, the 2025 joint Lux.jl/Enzyme.jl integration in Julia extended full neural-network AD support within the DifferentialEquations.jl framework, enabling seamless neural ODE training with forward-sensitivity and adjoint methods. The A Common Interface for Automatic Differentiation (arXiv:2505.05542, May 2025) proposes a cross-language AD interoperability standard covering JAX, PyTorch, Tapenade (Fortran), and Enzyme, aiming to reduce the friction of mixing AD-enabled components across language boundaries. In agriculture, DiffBreed (2024, published in Bioinformatics 2025) applied AD to entire plant breeding pipeline optimisation, treating multi-generational selection schemes as differentiable programs and optimising them end-to-end with gradient descent — a striking example of AD’s reach beyond machine learning.

  UK Context

  UK universities have contributed to automatic differentiation at both theoretical and applied levels across the numerical analysis, programming languages, and machine learning traditions, with particularly notable contributions to the categorical foundations and to scientific computing applications.

  Edinburgh — Categorical Foundations and Probabilistic Programming

  The University of Edinburgh’s Laboratory for Foundations of Computer Science (LFCS) produced foundational work on the categorical semantics of AD through the CHAD framework (Vákár & Smeding, 2021, PACMPL POPL), which provides the first correct-by-construction formulation of reverse-mode AD for higher-order functional languages with recursive types, using Cartesian reverse-differential categories as the semantic foundation. This work appeared at POPL 2024 in the LAFI (Languages for Inference) workshop in its extended form. Edinburgh’s LFCS maintains ongoing research on AD in the context of probabilistic programming languages and linear type systems, with connections to the denotational semantics of quantum programming languages where linear types prevent copying of quantum states — a constraint structurally analogous to the linearity requirements in AD VJP computation.

  Oxford — Numerical Analysis and Weather Prediction

  The University of Oxford’s Numerical Analysis Group applies AD-enabled adjoint methods in atmospheric science and weather prediction, coupling differentiated simulation codes with the Met Office Unified Model (UM) for data assimilation — the process of incorporating observational data into forecast model initial conditions. AD computes sensitivities of forecast error with respect to initial conditions (the “adjoint sensitivity” or “influence function”), enabling 4D-Var (Four-Dimensional Variational) data assimilation that dramatically improves forecast accuracy. Oxford’s Department of Statistics applies AD within probabilistic programming for Bayesian inference over cosmological models, differentiating through N-body gravitational simulation codes.

  Cambridge — Theoretical and Applied Computation

  The University of Cambridge’s Applied Mathematics and Theoretical Physics (DAMTP) Department employs AD in cosmological simulation pipelines and differentiable N-body codes for gravitational dynamics, using PyTorch and JAX to optimise initial condition parameters for large-scale structure formation models. Cambridge’s Computer Laboratory contributes to the theory of differentiable programming through dependent type theory research and the connections between AD and parametric polymorphism in System F type theory.

  Southampton — Pedagogy and Differentiable Programming

  The University of Southampton offers a dedicated graduate module COMP6258 “Differentiable Programming and Deep Learning” (2025–2026), one of the first UK programmes to teach differentiable programming as a first-class computational paradigm rather than merely as a tool for neural network training. This pedagogical canonisation signals that AD has matured from a specialist technique into core computational science curriculum, analogous to how numerical linear algebra and finite element methods became standard graduate training in the 1980s–1990s.

  Northern England — Industrial and Applied Research

  The University of Manchester’s machine learning group applies AD-trained Deep Learning models in medical imaging analysis through NHS collaborations: differentiable image registration (registering follow-up CT scans to baseline scans for radiotherapy treatment planning at Christie NHS Foundation Trust, Manchester); differentiable segmentation networks (automatic tumour delineation in PET-CT images for lung cancer staging at Manchester University NHS Foundation Trust); and AD-based biomarker discovery (differentiating through genomic pathway models to identify candidate drug targets).

  The University of Leeds applies AD within supply chain and logistics Optimisation systems at the EPSRC-funded Institute for Data Analytics, specifically in routing and scheduling problems where differentiable surrogates of discrete combinatorial objectives are optimised via gradient-based methods (the “differentiable optimisation” paradigm using linear programming relaxations or Sinkhorn approximations to compute gradients through combinatorial decisions).

  The University of Sheffield’s Advanced Manufacturing Research Centre (AMRC) employs physics-informed neural networks — trained via AD-computed PDE residual gradients capturing the heat equation, elasticity equations, and fluid flow equations — for digital twin modelling of directed energy deposition (DED) and selective laser melting (SLM) additive manufacturing processes, enabling real-time quality prediction (residual stress, porosity, dimensional accuracy) in aerospace component fabrication for BAE Systems, Rolls-Royce, and Boeing UK.

  Newcastle University’s Digital Institute applies AD-enabled Reinforcement Learning to autonomous vehicle navigation, training policy networks via differentiable physics simulation (Brax, MuJoCo MJX) for motion planning in urban scenarios, in collaboration with Nissan Sunderland’s autonomous vehicle research programme and the Connected and Autonomous Mobility initiative of the UK’s Centre for Connected and Autonomous Vehicles (CCAV).

  Research Funding

  UK EPSRC programmes funding AD-adjacent research include: the “Mathematical Foundations of Intelligence” portfolio (£50M, 2023–2028) covering differentiable programming and its applications; the “Probabilistic AI” priority area funding AD-based inference; and the “Digital Twins for Complex Systems” programme (£25M) where AD-enabled differentiable simulators are a core technical approach. UK Research and Innovation’s ARIA (Advanced Research + Invention Agency) has funded exploratory research on “differentiable world models” for physical simulation, targeting 10–100x efficiency improvements over current JAX-based differentiable physics approaches via specialised compiler and hardware co-design.

  Future Directions (2026–2030)

  Six converging forces shape the trajectory of automatic differentiation over the next four years, spanning hardware, algorithms, theory, and application domains.

  1. Hardware-Level AD Acceleration (2027–2030)

  Dedicated AD hardware units — analogous to CUDA tensor cores for matrix multiplication — are under research at Intel (Loihi 2 spiking cores for sparse gradient computation), Cerebras (wafer-scale systolic arrays for all-reduce gradient aggregation), and Graphcore (IPU bulk-synchronous parallel for graph-structured AD) for neuromorphic and wafer-scale processors. XLA, MLIR, and Enzyme already perform AD at the compiler IR level, with MLIR-VPUX enabling AD for AMD/Intel NPU (Neural Processing Unit) targets. The ARIES MLIR-based FPGA compilation flow (Cornell, 2025, FPGA 2025 proceedings) maps ML workloads including differentiated codes to AMD Alveo FPGAs with automated resource allocation, demonstrating a 3.7x energy efficiency advantage over GPU for sparse attention gradients. GPU memory bandwidth — the primary bottleneck for backward-pass tape replay in large models — will be addressed by HBM4 (High-Bandwidth Memory generation 4, projected 2027, 2+ TB/s bandwidth) and 3D-stacked GPU architectures collocating memory and compute.

  2. Higher-Order and Sparse AD at Scale (2026–2028)

  Higher-order and sparse AD will transition from research to production. The 2025 sparse Jacobian/Hessian AD work (arXiv:2501.17737) enables efficient second-order computation for finite-element simulation codes and graph neural networks where Jacobians have less than 1% density. KFAC-JAX and shampoo demonstrate curvature-aware training at 70B+ parameter scale; as GPU memory costs fall (projected 50% cost reduction by 2028), storing Kronecker factors for full model layers will become feasible. The combination of sparse AD and curvature preconditioning is expected to reduce LLM training compute by 20–40% relative to Adam for convergence to equivalent loss — a significant economic impact given the $100M+ training costs of frontier models.

  3. AD and Automated Reasoning Convergence (2027–2030)

  The boundary between AD and Automated Reasoning will dissolve through differentiable logic programming and probabilistic theorem proving. AlphaProof’s MCTS proof search already guides gradient descent over proof-step embedding vectors; future “Lean/JAX” or “Isabelle/PyTorch” interfaces will allow neural networks trained by reverse-mode AD to guide proof search while the formal checker validates each step. The “differentiable ILP” (Inductive Logic Programming) paradigm — systems like Scallop, NeuroLog, and DeepProbLog — combines gradient-based learning with logic program synthesis by differentiating through soft logic operations, enabling learning of relational rules from examples. By 2028–2030, these systems are expected to handle scientific hypothesis generation tasks: learning differentiable programs that combine neural perception (via Backpropagation) with logical inference (via automated reasoning) in end-to-end trainable pipelines.

  4. Differentiable Rendering and Interactive Applications (2026–2028)

  Differentiable rasterisers already run at interactive frame rates on RTX 4090/H100 hardware for small scenes; 3D Gaussian Splatting achieves real-time rendering at 30+ FPS after a 30-second training time. By 2027, differentiable rendering will be integrated into professional 3D content creation pipelines (Blender, Maya, Unreal Engine) enabling material and lighting parameter optimisation from reference photographs in minutes rather than hours of manual adjustment. Differentiable physics engines for robotics and autonomous driving simulation (Brax 3.0, IsaacGym 3.0) are expected to operate at full-speed (millions of parallel simulation steps per second) within Reinforcement Learning training loops on TPU pods, with AD gradients flowing through the entire sensor-to-actuator pipeline for end-to-end policy learning.

  5. AD for Scientific Discovery (2026–2030)

  AD will become the standard computational infrastructure for scientific machine learning across physics, biology, materials science, and climate modelling. Physics-informed neural networks differentiated via AD will compete with traditional finite element solvers for fluid dynamics, structural mechanics, and multi-physics coupling problems, with projected 10–100x speedups for inverse problems (inferring material properties from surface measurements). Differentiable molecular dynamics (JAX-MD, OpenMM-ML) will accelerate drug discovery by enabling gradient-based optimisation of molecular force field parameters and protein structure prediction refinement. Differentiable climate models (combining neural network parameterisations of sub-grid processes with differentiable global circulation models) will enable optimal experimental design for climate intervention assessment.

  6. AD Verification and Correctness (2027–2030)

  As AD systems become infrastructure for safety-critical AI, formal verification of AD implementations will become important. The CHAD framework (Edinburgh, 2021) provides a Agda proof-of-correctness for its AD transformation; future work will extend this to the PyTorch and JAX implementation level, proving correctness of specific optimisations (gradient accumulation commutativity, checkpoint correctness, parallelisation correctness). The convergence with Automated Reasoning on this axis means that proof assistants will eventually be used to certify AD compilers (analogous to CompCert for C compilers), providing machine-checked guarantees that gradient computations in deployed AI systems are mathematically correct.

  Research & Literature

  1. Wengert, R. E. (1964). A simple automatic derivative evaluation program. Communications of the ACM, 7(8), 463–464. [First computational implementation of forward-mode AD: the Wengert list / computation tape.]

  2. Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146–160. [First description of reverse-mode AD as error accumulation, from his 1970 Helsinki Master’s thesis.]

  3. Werbos, P. J. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. PhD Thesis, Harvard University. [First application of reverse-mode AD to training neural networks, predating Rumelhart et al. by 12 years.]

  4. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. [The paper that popularised backpropagation, demonstrating learning of internal representations in multi-layer networks.]

  5. Griewank, A., & Walther, A. (2008). Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (2nd ed.). SIAM. [Definitive theoretical reference for AD: complexity bounds, operator overloading, source transformation, checkpointing.]

  6. Pearlmutter, B. A., & Siskind, J. M. (2008). Reverse-mode AD in a functional framework. TOPLAS, 30(2), Article 7. [Higher-order AD for lambda calculus programs via CPS transformation: foundation for functional AD theory.]

  7. Elliott, C. (2018). The simple essence of automatic differentiation. PACMPL, 2(ICFP), Article 70. [Categorical semantics of AD: forward mode as derivative functor, reverse mode as transposition via cotangent duality.]

  8. Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18(153), 1–43. [Definitive survey unifying forward/reverse mode, operator overloading, source transformation, and connections to ML.]

  9. Bradbury, J. et al. (2018). JAX: composable transformations of Python+NumPy programs. GitHub: google/jax. http://github.com/google/jax [JAX: the functional AD library with composable grad/jit/vmap/pmap transformations compiling to XLA for GPU/TPU.]

  10. Moses, W. S., & Churavy, V. (2020). Instead of rewriting foreign code for machine learning, automatically synthesize fast gradients. SC 2020. ACM. doi:10.1145/3458817.3476165. [Enzyme: LLVM-level AD compiler enabling differentiation of C/C++/Fortran/Rust without Python overhead.]

  11. Vákár, M., & Smeding, T. (2021). CHAD: Combinatory homomorphic automatic differentiation. PACMPL, 6(POPL). [Correct-by-construction reverse-mode AD for higher-order functional languages, from University of Edinburgh LFCS.]

  12. Rackauckas, C. et al. (2020). Universal differential equations for scientific machine learning. arXiv:2001.04385. [Universal differential equations: combining neural networks with AD-differentiated ODE/PDE solvers for SciML.]

  13. Schoenholz, S. S., & Cubuk, E. D. (2020). JAX MD: A framework for differentiable physics. NeurIPS 2020. [JAX-based differentiable molecular dynamics with full reverse-mode AD through simulation trajectories.]

  14. Kidger, P. (2021). On neural differential equations. PhD Thesis, University of Oxford. [Neural ODEs theory and practice: adjoint method, continuous normalising flows, controlled differential equations.]

  15. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimisation. ICLR 2015. [Adam optimiser: adaptive learning rate method built on AD-computed first and second gradient moments.]

  16. Martens, J., & Grosse, R. (2015). Optimizing neural networks with Kronecker-factored approximate curvature. ICML 2015. [K-FAC: second-order optimisation using Kronecker factorisation of Fisher information matrix via AD curvature.]

  17. Chen, T. et al. (2016). Training deep nets with sublinear memory cost. arXiv:1604.06174. [Gradient checkpointing: O(√n) memory at O(n log n) compute cost for reverse-mode AD in deep networks.]

  18. Paszke, A. et al. (2019). PyTorch: An imperative style, high-performance deep learning library. NeurIPS 2019. [PyTorch autograd: define-by-run dynamic computation graph AD engine, the dominant ML research framework.]

  19. Abadi, M. et al. (2016). TensorFlow: A system for large-scale machine learning. OSDI 2016, 265–283. [TensorFlow: static graph AD with XLA compilation, underpinning industrial ML at Google scale.]

  20. Innes, M. et al. (2019). Zygote: A differentiable programming system to bridge machine learning and scientific computing. arXiv:1907.07587. [Zygote: source-transformation AD for Julia, the basis of the SciML ecosystem’s gradient engine.]

  21. University of Southampton (2025–2026). COMP6258: Differentiable Programming and Deep Learning. Module descriptor, University of Southampton. https://www.southampton.ac.uk/courses/modules/comp6258 [UK postgraduate module on differentiable programming as a first-class computational paradigm.]

  22. Kasim, M. F. et al. (2025). DiffBreed: automatic differentiation enables efficient gradient-based optimization of breeding strategies. Bioinformatics, 41(11), btaf586. doi:10.1093/bioinformatics/btaf586. [AD applied to multi-generational plant breeding optimisation — demonstrating AD reach beyond ML.]

  23. Sparse AD Team (2025). Sparser, better, faster, stronger: Efficient automatic differentiation for sparse Jacobians and Hessians. arXiv:2501.17737. [Sparse AD reducing second-order computation from O(n²) to O(nnz) for sparse scientific computing Jacobians.]

  24. Common AD Interface Team (2025). A common interface for automatic differentiation. arXiv:2505.05542. [Cross-language AD interoperability standard covering JAX, PyTorch, Tapenade (Fortran), and Enzyme.]

  25. SciML Community (2025). The state of SciML: Open source software for scientific machine learning. SciML Blog, June 2025. https://sciml.ai/news/2025/06/26/state_of_sciml/ [State of Julia SciML ecosystem: Lux.jl/Enzyme.jl integration, neural ODE advances, SciML benchmarks 2025.]

  26. Danilecki, B. et al. (2025). JAX autodiff from a linear logic perspective. PACMPL, doi:10.1145/3776654. [Linear logic interpretation of JAX VJP/JVP: proof-theoretic basis for AD transformation correctness.]

  27. LAFI 2024 Programme Committee (2024). Languages for Inference 2024 — Workshop at POPL 2024. https://popl24.sigplan.org/home/lafi-2024 [LAFI: workshop at the intersection of AD, probabilistic programming languages, and formal type theory.]

  Key Terminology Glossary

• Primal computation: the standard forward evaluation of a function f(x) producing the output value — as distinguished from the tangent (forward mode) or adjoint (reverse mode) computations that accompany it.

• Tangent / JVP: the Jacobian-vector product Df(x) · v, propagated forward through the computation graph in forward-mode AD. Represents the directional derivative of f at x in direction v.

• Adjoint / VJP: the vector-Jacobian product uᵀ · Df(x), propagated backward through the computation graph in reverse-mode AD. For scalar outputs u = ∇L, this is the full gradient ∇_x L.

• Cotangent: the adjoint value ∂L/∂v for an intermediate variable v, accumulating contributions from all downstream uses of v in the Computation Graph. The accumulation of cotangents from multiple outputs is the multi-variable chain rule.

• Wengert tape: the historical term for the recorded sequence of elementary operations in the forward pass — also called the computation trace, evaluation trace, or gradient tape (JAX GradientTape, TensorFlow GradientTape).

• Checkpointing: selective discarding of intermediate activations during the forward pass and recomputing from stored checkpoints during the backward pass (Gradient Checkpointing). Reduces peak memory from O(n_ops) to O(√n_ops) at one additional forward recomputation.

• Operator overloading: the AD implementation strategy that intercepts standard arithmetic at runtime by overloading Python/C++ operators to record the Computation Graph dynamically. Used by PyTorch autograd and TensorFlow 2.x eager mode.

• Source transformation: the AD strategy that transforms program source (or IR) into a new program computing both primal and adjoint values, enabling compiler-level optimisation. Used by Tapenade (Fortran), Enzyme (LLVM IR), and Zygote (Julia).

• Jaxpr: the functional intermediate representation used internally by JAX to capture the computation structure of a traced Python function for transformation and compilation. A jaxpr is a typed, first-order program without side effects, enabling composition of grad, jit, vmap, and pmap.

• Reverse accumulation: the classical term for reverse-mode AD, emphasising the data structure — accumulated adjoint values flowing backward through the Computation Graph in reverse topological order. Equivalent to Backpropagation when applied to the loss function of a Neural Network.

• Implicit differentiation: an AD technique for differentiating through the solution of an optimisation problem or fixed-point iteration (e.g., ODE solution, iterative solver) by applying the implicit function theorem rather than unrolling all solver iterations through the Computation Graph, avoiding O(n_iterations) memory cost.

Provenance