DoRA

DoRA (Weight-Decomposed Low-Rank Adaptation) is a parameter-efficient fine-tuning method that decomposes pretrained weights into separate magnitude and direction components, applying low-rank updates only to the directional component while learning the magnitude independently. By separating these two degrees of freedom, DoRA more closely mirrors the learning dynamics of full fine-tuning than standard LoRA, improving accuracy on many tasks at comparable parameter cost and without added inference latency once merged. It is used to adapt large language and vision models efficiently.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:hasPart ai:MagnitudeComponent))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:hasPart ai:DirectionalComponent))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:hasPart ai:LowRankAdaptation))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:hasPart ai:WeightDecomposition))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:hasPart ai:ColumnWiseNormalisation))

Dependency Relationships

SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:requires ai:PretrainedModel))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:requires ai:Backpropagation))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:requires ai:GradientDescent))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:dependsOn ai:TransformerArchitecture))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:dependsOn ai:LowRankAdaptation))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:dependsOn ai:NeuralNetwork))

Capability Relationships

SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:enables ai:FineTuning))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:enables ai:LargeLanguageModel))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:enables ai:VisionLanguageModel))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:enables ai:ModelAdaptation))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:supports ai:NaturalLanguageProcessing))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:supports ai:ComputerVision))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:supports ai:MultimodalLearning))

Implementation Relationships

SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:implements ai:WeightDecomposition))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:implements ai:LowRankAdaptation))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:uses ai:PyTorch))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:uses ai:HuggingFacePEFT))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:reducesTo ai:LowRankAdaptation))

Reduction Relationships

SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:reducesTo ai:ParameterEfficientFineTuning))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:reducesTo ai:TransferLearning))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:contrastsWith ai:FullFineTuning))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:contrastsWith ai:PromptTuning))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:contrastsWith ai:AdapterModules))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:relatedTo ai:QDoRA))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:relatedTo ai:DVoRA))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:relatedTo ai:FoundationModel))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:relatedTo ai:IntrinsicDimensionality))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:supports ai:MultilingualLearning))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:supports ai:TextToImageGeneration))
SubClassOf(ai:DoRA
  ObjectSomeValuesFrom(ai:enables ai:LowDataAdaptation))

About

The development of DoRA grew directly out of a critical examination of why Low-Rank Adaptation (LoRA), despite its remarkable practical success, consistently falls short of full fine-tuning accuracy on harder benchmarks, particularly when using low adapter ranks. LoRA (Hu et al., 2022) addresses the prohibitive cost of full fine-tuning very large Transformer Architecture models by freezing all pretrained weights W_0 and learning a low-rank residual update W = W_0 + ΔW = W_0 + BA, where B ∈ ℝ^{d×r} and A ∈ ℝ^{r×k} are small matrices with rank r ≪ min(d, k). This reduces trainable parameters from the d×k of the full weight matrix to r(d+k), enabling adaptation of 7-billion-parameter models on 24 GB consumer GPUs where full fine-tuning would require several hundred gigabytes. The learned adapter matrices are initialised so that ΔW = 0 at the start of training (A is random Gaussian, B is zeros), ensuring training begins from the pretrained weights. Despite its efficiency advantages, LoRA consistently underperforms full fine-tuning on tasks requiring significant distributional shifts from the pretraining domain, suggesting that LoRA’s low-rank constrained update does not capture all the degrees of freedom exercised by full fine-tuning.

The DoRA authors (Liu et al., NVIDIA NVLabs and National Taiwan University, 2024) characterised this gap through a novel weight decomposition analysis. They decomposed each weight matrix W — whether from a fully fine-tuned model or from a LoRA-adapted model — into a magnitude component m = ||W||_c (the column-wise L2 norm, a vector of k scalars) and a directional component V/||V||_c (the column-unit-normalised weight matrix). By tracking how each component evolved separately during training from the same pretrained initialisation, they discovered that full fine-tuning makes structured separable updates to these components: at many weight matrices, full fine-tuning primarily adjusts the directional component (large directional change, small magnitude change), while at others it primarily adjusts the magnitude (large magnitude change, small directional change). These patterns are structured and layer-type-specific. LoRA, by applying a single low-rank perturbation to the full weight matrix, conflates these two degrees of freedom and cannot independently reproduce either pattern — particularly large directional changes, which require the rank-r subspace to span the entire space of updated column directions simultaneously.

The DoRA method resolves this fundamental limitation by parameterising each adapted weight as W’ = (m / ||W_0 + BA||_c) ⋅ (W_0 + BA), where m ∈ ℝ^{1×k} is the trainable magnitude vector, B ∈ ℝ^{d×r} and A ∈ ℝ^{r×k} are the LoRA matrices, and ||W_0 + BA||_c is the column-wise norm of the current directionally-adapted weight matrix (computed dynamically at each forward pass). The gradient of the loss with respect to m is therefore a simple vector gradient decoupled from the low-rank constraint on BA; the gradient of the loss with respect to BA flows through both the directional normalisation and the magnitude scaling. This decoupling means the optimiser can independently decide: (a) how far to shift the directional orientation of each column (controlled by BA), and (b) how to rescale the magnitude of that column (controlled by m). A large directional change with small magnitude correction, a large magnitude change with small directional change, or any arbitrary combination becomes representable within the same parameter budget as standard LoRA plus k additional scalar parameters per weight matrix.

Crucially, once training is complete, the learned components can be merged back into a single weight matrix: W_final = (m / ||W_0 + BA||_c) ⋅ (W_0 + BA). This merged weight is identical in shape and data type to the original pretrained weight W_0 and can be used as a direct replacement, making inference identical in compute, memory, and latency to using the original pretrained model without any adapter. This weight merging property is shared with LoRA and is a key practical advantage over adapter module approaches, which require keeping the adapter parameters separate and applying them at inference time, adding latency proportional to adapter width. The only overhead of DoRA relative to standard LoRA during inference is zero — the magnitude vector m is absorbed into the merged weight.

Components and Architecture

Weight Decomposition Layer

The foundational operation decomposes each target weight matrix W_0 ∈ ℝ^{d×k} into magnitude m ∈ ℝ^{1×k} (column norms) and directional matrix V = W_0 / ||W_0||_c ∈ ℝ^{d×k} (column unit vectors): W_0 = m ⋅ V. Only m and the LoRA matrices A and B are trainable; V itself is not stored as a separate parameter but is implicitly defined as (W_0 + BA) / ||(W_0 + BA)||_c during the forward pass. The column norm ||W_0 + BA||_c must be recomputed at each forward pass over the current value of the directionally-adapted matrix W_0 + BA, as BA changes during training. This dynamic recomputation adds a small but non-zero computational overhead relative to LoRA, typically less than 1% of total training FLOPs for reasonable adapter ranks.

Low-Rank Directional Update

The directional update ΔV = BA follows exactly the LoRA formulation: B ∈ ℝ^{d×r} and A ∈ ℝ^{r×k}, with A initialised using the Kaiming uniform distribution and B initialised to zero, so that ΔV = 0 at the start of training and the adaptation begins from the pretrained directional configuration. The rank r is a hyperparameter (common values: 8, 16, 32, 64) that controls the capacity of the directional update; the set of representable directional changes is the column span of B, a d-dimensional subspace of rank r. DoRA’s key theoretical claim is that separating magnitude from direction allows this rank-r subspace to be used more efficiently for directional updates alone, rather than having it simultaneously serve both directional and magnitude update purposes.

Magnitude Vector Parameterisation

The magnitude vector m ∈ ℝ^{1×k} is initialised to the column norms of W_0 — the exact magnitudes of the pretrained weight columns — so that the decomposition is exact at initialisation (W_0 = m_init ⋅ (W_0 / ||W_0||_c)). During training, m is updated by the optimiser (Adam, AdaFactor, or SGD with momentum) independently of BA. Being a dense but very narrow vector with only k elements (where k is the output dimension of the weight matrix, e.g., 4096 for LLaMA-7B’s Q projection), m adds minimal parameter overhead: for a single attention projection matrix of shape 4096×4096, m has 4096 elements compared to r×4096 + r×4096 = 8192r for the LoRA matrices. At rank r=16, m is 4096 parameters versus 131,072 for BA — about 3% of the LoRA parameter count — making it a negligible addition even when applied to all weight matrices in the model.

Integration with Quantisation (QDoRA)

DoRA is fully compatible with quantisation-aware training approaches. In the QDoRA configuration, the base weights W_0 are stored in 4-bit NF4 (Normal Float 4) quantisation as in QLoRA (Dettmers et al., 2023), while the trainable magnitude vector m and the LoRA matrices A and B are maintained in full bfloat16 precision. During the forward pass, W_0 is dequantised on-the-fly to bfloat16 for the matrix multiplication, while m and BA are applied in the DoRA formulation. After training, the adapter can be merged into dequantised weights for deployment. QDoRA has been characterised as the leading PEFT standard for 2025 by practitioners, achieving near-full-fine-tuning quality on consumer hardware (24 GB VRAM for 7B models, 48 GB for 13B models) at 4-bit quantisation, making frontier model fine-tuning accessible without access to data centre GPU resources.

Applicable Weight Matrices and Targeting Strategy

Following established LoRA convention, DoRA is most commonly applied to the query (Q), key (K), value (V), and output projection matrices in multi-head attention sub-layers, and to the up-projection and down-projection (or gate-projection in SwiGLU architectures) matrices of feed-forward sub-layers within each Transformer Architecture block. Empirical studies across LLaMA and LLaVA model families suggest that applying DoRA to all linear weight matrices in the model (including the output projection and the LM head) generally improves task performance over the LoRA default of applying adapters only to Q and V projection matrices. Embedding matrices and layer normalisation gain and bias parameters are typically left unchanged because their structure differs fundamentally from the column-oriented weight matrices to which the DoRA decomposition is most naturally applied.

Variants and Extensions

DVoRA (DoRA + VeRA) combines DoRA’s magnitude-direction decomposition with the Vector-based Random Matrix Adaptation (VeRA) approach, which uses shared frozen random matrices for B and A across layers and learns only small per-layer scaling vectors. DVoRA achieves comparable or superior performance to DoRA at significantly fewer trainable parameters on certain tasks, making it attractive for very constrained parameter budgets. SSMLoRA (2025) integrates state-space model (SSM) components into the adaptation structure for improved modelling of long-range dependencies, showing particular benefit on long-context tasks. Structured task-aware low-rank adaptation methods (2025, arXiv:2603.14228) extend DoRA’s directional analysis to identify which directions within the low-rank subspace are most task-relevant and concentrate the rank budget there, further improving parameter efficiency. High-rank structured modulation approaches (arXiv:2601.07507, 2025) complement DoRA by exploiting structured high-rank updates that complement the low-rank directional adaptation.

Use Cases / Major Families

Language Model Instruction Fine-Tuning

DoRA has been extensively evaluated on instruction following and commonsense reasoning benchmarks using LLaMA-7B, LLaMA-13B, and LLaMA-3-8B base models. On the eight-dataset commonsense reasoning evaluation suite (CommonsenseQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, PIQA, SIQA, and OpenbookQA), DoRA outperforms LoRA by an average of 3.7 percentage points on LLaMA-7B at matched adapter rank, and achieves commonsense accuracy exceeding ChatGPT-3.5 on several individual benchmarks — a widely used performance threshold since ChatGPT’s instruction-tuned training gives it a strong commonsense baseline. DoRA maintains this superiority at lower adapter ranks (r=8) relative to LoRA at higher ranks (r=32), suggesting that DoRA’s improved representational capacity partially compensates for rank reduction, making it particularly valuable when the parameter budget is tightly constrained.

Mathematical Reasoning

On mathematical reasoning benchmarks including GSM8K and MATH, DoRA shows consistent improvements over LoRA at matched rank and learning rate configurations. The decomposition appears particularly beneficial for mathematical tasks because the reasoning capabilities encoded in weight directions may need larger directional updates (representing genuinely new reasoning patterns) while magnitude adjustments calibrate confidence, a separation that full fine-tuning exploits and that DoRA’s architecture enables.

Vision-Language Model Adaptation

DoRA has been applied to LLaVA-1.5 (a Transformer Architecture vision-language model combining a CLIP vision encoder with a Vicuna LLM decoder) and VL-BART for image-text and video-text understanding tasks. On the LLaVA visual instruction tuning benchmark suite (VQAv2 for visual question answering, GQA for compositional question answering, TextVQA for text recognition in images, MMBench for multimodal reasoning, MM-Vet for complex VQA), DoRA consistently outperforms LoRA by 1–3 percentage points at matched rank and training budget. For video understanding tasks (ActivityNet-QA, MSVD-QA, MSRVTT-QA), where the adapter must generalise from fine-tuning data to temporally extended video sequences, DoRA’s improved directional control shows particularly consistent benefit over LoRA, suggesting that the magnitude-direction decomposition captures properties of the weight update geometry that generalise across modalities.

Text-to-Image and Diffusion Model Fine-Tuning

DoRA applied to diffusion model U-Net weight matrices (Stable Diffusion, SDXL) has demonstrated improved subject fidelity and style consistency relative to LoRA at matched parameter budgets. This application domain is commercially significant: virtually all commercial LORA-based style transfer, character customisation (DreamBooth-LoRA), and aesthetic fine-tuning workflows could in principle be improved by substituting DoRA for LoRA, provided the training toolchain supports DoRA. The Hugging Face Diffusers library DoRA integration is progressing following the PEFT library native support.

Low-Data and Resource-Constrained Settings

Research from 2026 (arXiv:2603.17782) on adapting billion-parameter vision models to agricultural imagery under a severely limited data regime (98:1 test-to-train ratio, approximately 50 training samples for 5,000 test images) found QDoRA achieving approximately 83% test accuracy — competitive with QLoRA and significantly above zero-shot baseline — demonstrating that DoRA’s quality advantage extends to severely data-limited fine-tuning scenarios without requiring additional parameters. This is directly relevant to applied AI deployment in domains where annotated training data is expensive or scarce: medical device manufacturers, materials scientists, industrial inspection, and agricultural monitoring applications all commonly face this constraint.

Domain Adaptation for Professional Applications

DoRA has been applied to immigration law and insurance domain adaptation tasks (2025 research), where fine-tuning of a general-purpose LLM to professional domain terminology, reasoning patterns, and factual accuracy requirements demands more expressive adaptation than standard LoRA can provide within a constrained parameter budget. The larger distributional shift between general web text (the pretraining domain) and highly specialised professional language appears to particularly benefit from DoRA’s decoupled directional and magnitude updates, which allow the model’s column directions to shift substantially (capturing new vocabulary distributions and reasoning patterns) while magnitude adjustments calibrate output confidence to domain-specific certainty levels.

Academic Context

DoRA was authored by Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen, spanning NVIDIA NVLabs, National Taiwan University, and National Yang Ming Chiao Tung University. The paper was accepted as an oral presentation at ICML 2024 — at a 1.5% acceptance rate, the highest distinction in the conference programme — placing it among the most highly selected contributions of the year. The official PyTorch implementation is maintained at https://github.com/NVlabs/DoRA.

The intellectual context for DoRA is the broader Parameter-Efficient Fine-Tuning literature that emerged from the practical necessity of adapting enormous foundation models for specific tasks without the resource requirements of full fine-tuning. The adapter approach (Houlsby et al., ICML 2019) inserted small bottleneck modules (two fully-connected layers with a low-rank bottleneck) into frozen Transformer Architecture blocks, reducing trainable parameters from billions to millions while preserving pretrained knowledge. Prefix tuning (Li & Liang, ACL 2021) and prompt tuning (Lester et al., EMNLP 2021) moved entirely to soft trainable token representations prepended to inputs, avoiding architectural modification altogether. LoRA (Hu et al., ICLR 2022) achieved the most practical impact by combining targeted parameter updates in weight space (as in adapters) with weight mergeability (zero inference overhead), becoming the dominant PEFT method within one year of publication.

The progression from LoRA to DoRA was not direct: several intermediate methods characterised LoRA’s limitations and proposed partial solutions. LoRA+ (Hayou et al., ICML 2024) showed that different learning rates for the A and B matrices substantially improve LoRA performance, with the optimal ratio being the square root of the rank dimension. rsLoRA (Kalajdzievski, 2023) corrected a scaling factor in LoRA’s rank initialisation that caused instability at high ranks. VeRA (Kopiczko et al., ICLR 2024) reduced trainable parameters further by sharing frozen random matrices across layers and learning only scaling vectors. LoftQ (Liu et al., 2023) improved DoRA/LoRA initialisation for quantised models, aligning adapter initialisation with the quantisation error. The (IA)³ method (Liu et al., NeurIPS 2022) applied even more lightweight adaptation using rescaling vectors applied to key, value, and feed-forward weights.

DoRA’s theoretical contribution connects to a broader research direction in understanding the intrinsic geometry of fine-tuning. Aghajanyan et al. (ACL 2021) showed that fine-tuning mainly occupies a low-dimensional subspace of the full parameter space, providing theoretical motivation for low-rank adaptation methods. OFT (Orthogonal Fine-Tuning, Qiu et al., NeurIPS 2023) constrained the directional update to be a rotation (orthogonal transformation), explicitly preserving hyperspherical energy and magnitude. DoRA generalises OFT by removing the rotation constraint and replacing it with the less restrictive low-rank constraint on the directional update, while explicitly learning the magnitude separately — a strictly more general family that subsumes OFT-style rotation as a special case when m is fixed and BA is orthogonal.

Current Landscape (2026)

As of mid-2026, DoRA is widely adopted in research fine-tuning workflows and is gaining commercial adoption for tasks where LoRA’s accuracy deficit is operationally significant. The Hugging Face PEFT library — the dominant Python package for parameter-efficient fine-tuning across the ecosystem — integrates DoRA natively as of PEFT version 0.11 (released early 2024 following the ICML 2024 oral presentation), enabling drop-in replacement of LoRA with DoRA through a single use_dora=True configuration flag. The Hugging Face Transformers SFTTrainer and TRL library both support DoRA through the PEFT configuration, making it accessible to practitioners without requiring custom code.

QDoRA — the combination of 4-bit NF4 base weight quantisation (as in QLoRA) with DoRA’s magnitude-direction adaptation — has emerged as what practitioners and practitioners’ guides characterise as the leading PEFT standard for 2025. QDoRA substantially narrows the quality gap to full fine-tuning while remaining trainable on single-GPU consumer hardware (24 GB VRAM for 7B models), democratising access to high-quality fine-tuning without data centre GPU resources. The comparison article “Comparing Fine-Tuning Optimization Techniques (LoRA, QLoRA, DoRA, and QDoRA)” by Encora (2025) documents the practical trade-offs across these methods with empirical benchmarks across several task families.

The broader PEFT landscape in 2026 features ongoing competition and complementarity among methods. LoRA remains the baseline for its simplicity and lowest overhead. QLoRA addresses memory-critical 4-bit training scenarios. DoRA improves quality for tasks where the accuracy gap to full fine-tuning matters operationally. (IA)³ provides extremely sparse adaptation at even lower parameter count. Adapter modules support multi-task serving where per-task adapters can be hot-swapped from a shared weight server without merging, at the cost of inference latency. The 2025 survey “PEFT A2Z” (arXiv:2504.14117) provides a systematic taxonomy positioning DoRA as a tier-1 method alongside LoRA, QLoRA, and (IA)³ in the current PEFT landscape.

Research extensions published in 2025–2026 include: structured task-aware adaptation building on DoRA’s directional analysis to concentrate rank budget in the most task-relevant subspaces (arXiv:2603.14228); QDoRA for adapting billion-parameter vision foundation models (ViT-L, ViT-H) in low-data agricultural and medical imaging regimes (arXiv:2603.17782); DVoRA combining DoRA’s magnitude separation with VeRA’s shared random matrix approach for further parameter efficiency; and applications of DoRA to diffusion model fine-tuning for personalisation and style transfer. ICML and NeurIPS 2025 programmes included multiple papers building on the DoRA framework or proposing alternatives motivated by the same weight decomposition analysis.

UK Context

UK research groups have contributed to both the theoretical foundations and applied contexts for DoRA and Parameter-Efficient Fine-Tuning more broadly. UCL’s Centre for Artificial Intelligence, the Oxford Internet Institute, and the Edinburgh Natural Language Processing group have all published research on efficient adaptation of large language models to domain-specific and resource-constrained settings. The EPSRC Programme Grant “Robust and Reliable AI” (led by Edinburgh, in collaboration with Manchester and Imperial) includes work on understanding the geometry of fine-tuning, directly relevant to DoRA’s theoretical foundation.

The NHS AI Lab and NHSX have motivated UK-specific research into PEFT-based adaptation of medical language models for clinical note processing, radiology report generation, and clinical coding (ICD-10 classification). NHS information governance frameworks prohibit moving sensitive patient data to external training clusters, making fine-tuning only a small adapter on locally held clinical data the operationally mandated approach for domain specialisation. Imperial College London’s Department of Computing has applied LoRA and DoRA to NHS clinical data for automated ICD coding, demonstrating that DoRA’s improved accuracy is material in a medical context where classification errors have clinical and financial consequences.

Edinburgh’s School of Informatics has integrated DoRA-based fine-tuning into research pipelines for biomedical NLP, applying it to models trained on MIMIC-IV clinical notes and the UK Biobank phenotype data for rare disease prediction and genetic association studies. The Alan Turing Institute’s national AI research centre coordinates PEFT methodology research across its university network, with a focus on federated adaptation — applying LoRA and DoRA within federated learning frameworks where each NHS trust or research institution fine-tunes only on its local data and aggregates adapter parameters (rather than raw data) across sites.

Manchester’s AI research hub and the N8 Research Partnership (the consortium of eight research-intensive universities in Northern England: Durham, Lancaster, Leeds, Liverpool, Manchester, Newcastle, Sheffield, and York) have applied DoRA to industrial text analytics, engineering maintenance log analysis, and supply chain documentation processing for the Northern England manufacturing sector. These applications are characterised by highly domain-specific vocabulary and reasoning patterns that diverge substantially from general web text, making the improved directional control of DoRA particularly valuable for domain adaptation quality.

The UK Government’s National AI Strategy and the AI Safety Institute (established 2023) have highlighted efficient fine-tuning and model adaptation as a key enabling technology for safe AI deployment, noting that PEFT methods allow organisations to adapt foundation models to their specific operational contexts while minimising the security and safety risks associated with full retraining on sensitive datasets. The AI Safety Institute’s evaluations of frontier model fine-tuning techniques include DoRA as a reference method alongside LoRA and full fine-tuning.

Future Directions (2026–2030)

Several research directions will shape the evolution of DoRA and weight-decomposition-based adaptation over the next four years. The mathematical framework of magnitude-direction decomposition is generalisable beyond column-wise L2 norms: future variants may exploit full singular value decomposition (SVD) of weight matrices to identify principal components and apply low-rank updates selectively in the most informative singular directions. This connects to theoretical work on the intrinsic dimensionality of fine-tuning (Aghajanyan et al., 2021) and to the ROSA (Random Orthogonal Subspace Adaptation) line of methods.

Continual learning and multi-task adaptation are natural extensions of DoRA: the magnitude component provides a natural control variable for interpolating between task-specific and pretrained knowledge, and DoRA adapters can be merged via task arithmetic and linear mode connectivity with potentially better geometric behaviour than LoRA adapters, given that the magnitude-direction decomposition has cleaner gradient geometry. Multi-task DoRA ensembles serving thousands of domain-specific adapters over a shared base model — where per-request adapter selection is performed without weight merging — may become a dominant deployment pattern for LLM API services, with the magnitude vector enabling adaptive confidence calibration per task without architecture changes.

The intersection of DoRA with model pruning and structured sparsity is relatively unexplored: if the directional update BA is concentrated in a sparse set of columns, structured pruning of adapter matrices may be applicable, further reducing parameter count. Post-training quantisation applied to merged DoRA weights may have better numerical properties than merged LoRA weights due to the magnitude normalisation preserving uniform column norms, potentially improving quantisation accuracy for low-bit deployment. DoRA applied to state space models (Mamba, RWKV) and linear attention variants is a natural extension as these architectures gain adoption for long-context tasks where Transformer Architecture attention is computationally impractical.

As Transformer Architecture models grow through Mixture-of-Experts (MoE) architectures, DoRA applied to expert weight matrices raises new questions: whether magnitude and direction should be decomposed per-expert (capturing expert-specific scaling) or shared across experts within a layer (capturing layer-level scaling). The interaction between DoRA’s weight decomposition and token routing in MoE fine-tuning may enable expert-level specialisation with fewer trainable parameters than per-expert LoRA adapters. UK-specific regulatory drivers — NHS data governance requirements, the EU AI Act’s transparency provisions applicable to UK providers serving EU markets, and the ICO’s guidance on AI model documentation — create strong incentives for PEFT-based fine-tuning approaches that limit sensitive data exposure, audit adapter provenance, and maintain clear attribution between base model and task-specific adaptation.

Formal Algorithm

DoRA Forward Pass

Given pretrained weight W_0 ∈ ℝ^{d×k}, trainable magnitude m ∈ ℝ^{1×k}, and trainable LoRA matrices A ∈ ℝ^{r×k}, B ∈ ℝ^{d×r}:

1. Compute adapted direction: W_adapted = W_0 + BA ∈ ℝ^{d×k}
2. Compute column norms: c_j = ||W_adapted[:, j]||_2 for j = 1, …, k
3. Normalise: V_normalised = W_adapted / c (broadcasting c ∈ ℝ^{1×k})
4. Apply magnitude: W’ = m ⊙ V_normalised (element-wise broadcast multiplication)
5. Forward: h = W’ x + bias (or equivalently h = (m / c) ⊙ (W_adapted x))

DoRA Merge (Inference)

After training:

1. W_final = (m / ||W_0 + BA||_c) ⊙ (W_0 + BA)
2. Replace W_0 with W_final in the pretrained model
3. Remove m, A, B — zero additional inference parameters

Comparison with LoRA

LoRA update: W’ = W_0 + BA (additive, unconstrained, d×r + r×k parameters) DoRA update: W’ = (m / ||W_0 + BA||_c) ⊙ (W_0 + BA) (d×r + r×k + k parameters) Parameter overhead of DoRA over LoRA: k parameters (column magnitudes), negligible relative to r(d+k).

Benchmark Results and Evaluation

• LLaMA-7B commonsense reasoning (8-task suite): DoRA (r=16): 82.1% average; LoRA (r=16): 78.4% average; Full FT: 83.7%; ChatGPT-3.5: 80.2%

• LLaMA-7B commonsense reasoning (r=8 vs LoRA r=32): DoRA at r=8 outperforms LoRA at r=32 on 6 of 8 tasks, demonstrating superior parameter efficiency

• LLaVA-1.5-7B visual instruction tuning: DoRA gains 1.2–2.8 points over LoRA across VQAv2, GQA, TextVQA, MMBench benchmarks at matched rank r=16

• VL-BART video understanding (ActivityNet-QA): DoRA achieves 2.1 points over LoRA at rank r=16

• DVoRA vs DoRA: DVoRA at 10× fewer parameters than DoRA achieves within 0.3 points of DoRA on most commonsense tasks

• QDoRA on agricultural vision models (2026): ~83% test accuracy at 98:1 test-to-train ratio, competitive with QLoRA, significantly above zero-shot baseline

  Theoretical Connections and Related Methods

  DoRA’s theoretical contribution connects to several adjacent lines of research on the geometry of neural network weight space and fine-tuning. The intrinsic dimensionality hypothesis (Aghajanyan et al., ACL 2021) demonstrated that the effective dimensionality of the fine-tuning update for many NLP tasks is surprisingly low — as few as 200 dimensions for some GLUE tasks — providing the theoretical basis for the claim that low-rank updates can capture the essential fine-tuning signal. DoRA extends this understanding by observing that the low-dimensional update can be further decomposed into separable magnitude and directional components, and that exploiting this separation improves the efficiency of the low-dimensional parameterisation.

  The connection to orthogonal fine-tuning (OFT, Qiu et al., NeurIPS 2023) is particularly illuminating. OFT constrains the directional component of weight updates to be a rotation (an orthogonal transformation preserving vector norms), motivated by the hypothesis that fine-tuning should preserve hyperspherical energy — the angular relationships between weight vectors that encode pretrained knowledge — while allowing magnitude to adapt freely. DoRA can be seen as a strict generalisation of OFT: if the LoRA matrices BA are constrained to produce an orthogonal directional update (a strong constraint), DoRA specialises to an OFT-like method; but without this constraint, DoRA allows arbitrary low-rank directional changes, which is more expressive for tasks requiring large distributional shifts.

  The adapter module family (Houlsby et al., 2019; Pfeiffer et al., AdapterFusion, 2021; He et al., UniPELT, 2022) represents a structurally different approach: rather than modifying existing weight matrices, adapters insert new small networks (bottleneck MLP layers) between frozen pretrained layers. Adapter methods require keeping the adapter parameters separate from pretrained weights at inference time, adding latency proportional to adapter width. LoRA and DoRA’s key practical advantage over adapters is weight mergeability — zero inference overhead — making them more appropriate for deployment scenarios where inference latency matters. Adapter methods retain advantages in multi-task serving where different adapters must be rapidly swapped for different requests without weight re-merging.

  The prefix tuning family (Li & Liang, 2021; P-tuning v2, Liu et al., 2022) operates in the input token space rather than the weight space, prepending soft trainable token embeddings to the input and relying on the frozen model’s attention mechanism to incorporate these signals into its computation. Prefix tuning sidesteps the weight modification problem entirely but introduces its own limitations: the soft prefix tokens consume context window length, and the method is less effective for tasks requiring large distributional shifts where weight-space updates are necessary. For tasks with tight context constraints or long-document inputs, weight-space PEFT methods (DoRA, LoRA) are preferred.

  The (IA)³ method (Few-Shot Parameter-Efficient Fine-Tuning, Liu et al., NeurIPS 2022) takes an even more extreme approach: it learns only three vector-valued scaling factors per transformer layer (for keys, values, and feed-forward activations), achieved by injecting learned rescaling multiplied element-wise into the attention and feed-forward computations. (IA)³ uses fewer trainable parameters than any other mainstream PEFT method but correspondingly has less representational capacity for large distributional shifts. DoRA occupies an intermediate position in this space: more parameters and expressiveness than (IA)³, comparable parameters to LoRA, and improved quality over LoRA.

  Standards, Tooling, and Ecosystem Integration

  DoRA is available through several major ecosystem entry points that ensure broad practitioner access. The Hugging Face PEFT library (https://github.com/huggingface/peft) is the primary Python package: DoRA is available as a LoraConfig option (use_dora=True), making it a one-line change to enable in any script that already uses LoRA via PEFT. The PEFT library’s integration with Hugging Face Transformers, Accelerate, and TRL means that DoRA is usable out-of-the-box in supervised fine-tuning (SFT) and RLHF (via TRL’s SFTTrainer and PPOTrainer) workflows.

  The Axolotl fine-tuning framework — a popular third-party tool for training and fine-tuning LLMs that wraps PEFT and Transformers — added DoRA support shortly after PEFT integration, making it accessible to practitioners who prefer YAML-configured training pipelines over Python scripting. Unsloth, a GPU-kernel-level optimised fine-tuning library targeting consumer-GPU training, similarly integrated DoRA support, including quantised DoRA (QDoRA) configurations optimised for single-GPU training.

  Within the research community, the NVLabs official implementation (https://github.com/NVlabs/DoRA) provides standalone PyTorch code for reproducing the ICML 2024 results and for extending DoRA to new architectures. This implementation serves as the canonical reference for verifying claimed performance improvements and understanding implementation details that may differ from the PEFT library’s production implementation.

  No formal standards body governs PEFT methods; the Hugging Face PEFT library serves as the de facto standard through adoption breadth. The library’s model card and adapter card specifications (which include metadata about the adapter method used, the base model, and the training configuration) provide a lightweight provenance framework for published fine-tuned model adapters, enabling users to identify whether a shared adapter uses LoRA, DoRA, or another method.

  Economic and Deployment Context

  The practical economics of DoRA are favourable relative to both full fine-tuning and alternative high-quality adaptation methods. A full fine-tuning run of a 7B parameter model requires approximately 16 GB VRAM for the model weights alone (at bfloat16 precision) plus gradient storage and optimiser states, totalling 120 GB for the Adam optimiser in full precision — requiring either an 8×A100 GPU server or distributed ZeRO training. DoRA at rank r=16 applied to all attention and MLP projections adds approximately 6–8 million trainable parameters to a 7B model, compared to approximately 7 billion for full fine-tuning — a 1000× reduction in trainable parameter count, and a proportional reduction in gradient and optimiser state memory, enabling single-GPU training on a 24 GB consumer GPU (RTX 3090, RTX 4090, or A10G).

  The cost differential between full fine-tuning and QDoRA on a 7B model is approximately 10–20× in GPU-hours, depending on the dataset size and number of epochs. For organisations fine-tuning multiple task-specific adapters over a shared base model, DoRA’s parameter efficiency enables maintaining dozens of separately fine-tuned adapters at low storage cost (each adapter being 10–50 MB for a 7B base model at rank 16), versus the cost of maintaining dozens of full 7B model checkpoints at 14 GB each. The multi-adapter serving architecture — where a single base model in GPU memory is augmented with per-request adapter selection — is an emerging deployment pattern for LLM API services that makes DoRA and LoRA economically attractive relative to fine-tuned model serving.

  The DoRA paper’s zero-overhead inference claim holds exactly for merged adapter deployment: after merging, the model is indistinguishable from a differently pretrained model of the same architecture and precision. This makes DoRA-fine-tuned models directly deployable with any inference engine that supports the base model architecture (vLLM, TGI, TensorRT-LLM, llamacpp) without modification, a significant practical advantage over adapter-based methods that require framework-specific adapter loading support.

  Training Stability and Practical Considerations

  DoRA introduces one practical consideration that does not arise in standard LoRA training: the column-norm recomputation in the forward pass. At each forward step, the column-wise L2 norms of the dynamically updated weight matrix W_0 + BA must be computed before the normalisation and magnitude scaling. For a weight matrix of shape d×k = 4096×4096 with bfloat16 precision, this column-norm computation requires a reduction over 4096 elements for each of 4096 columns — approximately 16 million FLOPs per weight matrix. For a 7B parameter LLaMA model with approximately 200 adapted weight matrices, this adds roughly 3.2 billion FLOPs per forward pass, compared to approximately 14 trillion FLOPs for the full transformer forward pass at typical sequence lengths — a computational overhead below 0.025%, effectively zero from a throughput perspective.

  Training stability with DoRA is generally reported as equal to or better than LoRA at the same hyperparameters. The separation of magnitude and direction provides more independent gradient signals to the two sets of parameters, which can reduce gradient interference. One potential numerical concern is column-norm collapse: if BA causes some column directions to become very small in magnitude during early training, the normalisation step could amplify small numerical errors. Empirically, this is avoided by the initialisation convention (B=0 at start, so W_0 + BA = W_0 at step 0 with healthy column norms) and by standard weight decay and learning rate schedules.

  The optimal learning rate for DoRA may differ from the optimal LoRA learning rate because m and BA have different gradient scales and different effective parameterisation. The DoRA paper found that standard LoRA hyperparameters (learning rate 1e-4 to 3e-4, α=r, batch size 128 sequences, rank r=16 to 64) transferred well to DoRA without additional tuning. However, the LoRA+ finding (Hayou et al., 2024) that different learning rates for A and B matrices improve performance is likely to apply to DoRA as well, and practitioners experimenting with DoRA may benefit from treating m as a third parameter group with its own learning rate multiplier.

  Applying DoRA selectively to subsets of weight matrices is a common practical strategy when the total trainable parameter budget is tightly constrained. The standard LoRA convention of targeting only Q and V attention projections can be applied to DoRA as well, with slightly lower parameter counts than full-model coverage. Empirical findings from the DoRA paper and follow-up work suggest that the benefit of DoRA over LoRA is largest when applied to the feed-forward MLP projections (up and down projections), which have larger weight matrices and may carry more of the task-specific knowledge that requires directional updates.

  Distributed fine-tuning with DoRA is fully supported through the PEFT library’s integration with Accelerate and FSDP: the magnitude vector m and the LoRA matrices A and B are treated as standard PyTorch parameters and are automatically distributed across data-parallel ranks using DDP or FSDP with no DoRA-specific changes to the distributed training code. The additional parameter count from m is negligible relative to the base model parameters, so the memory overhead of DoRA in distributed fine-tuning settings is effectively identical to LoRA.

  Multimodal and Cross-Architecture Applications

  While DoRA was originally evaluated on decoder-only Large Language Models (LLaMA family) and vision-language models (LLaVA), the weight decomposition principle is architecture-agnostic and has been explored across a range of model families. For encoder-only Transformer Architecture models (BERT, RoBERTa, DeBERTa) used in classification, named entity recognition, and question answering tasks, DoRA provides similar improvements over LoRA, with particularly consistent gains on tasks requiring fine-grained lexical sensitivity (NER, slot filling) where directional adjustment of embedding-like weight columns is important.

  For diffusion model architectures (U-Net-based and DiT-based), DoRA applies to the attention and convolutional weight matrices within the denoising network. Early experiments with DreamBooth-DoRA (applying DoRA instead of LoRA in DreamBooth-style subject customisation) reported improved subject identity preservation and better prompt fidelity relative to DreamBooth-LoRA at matched training steps. The text encoder component of diffusion pipelines (typically CLIP or T5) can also be adapted with DoRA, with potential benefits for style and compositional accuracy.

  For state space models (Mamba, S4, RWKV), which use structured recurrent parameterisations rather than attention weight matrices, the DoRA decomposition applies straightforwardly to the input and output projection matrices but requires careful adaptation for the structured recurrence matrices (A, B, C in SSM parameterisation), where column-wise magnitude-direction decomposition may not be the most natural structure-preserving factorisation. Research on PEFT for state space models is an emerging area where DoRA variants may be developed.

  For code generation models (CodeLlama, DeepSeek-Coder, StarCoder2), DoRA fine-tuning on domain-specific codebases (e.g., proprietary API documentation, internal library styles) has been reported to outperform LoRA in tool-use accuracy and API call correctness, domains where distributional shift from general programming corpora is large and the directional component of weight updates must carry significant new information about domain-specific syntax and idiom.

  Research and Literature

  1. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. ICLR 2022. arXiv:2106.09685.
  2. Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., & Chen, M.-H. (2024). DoRA: Weight-decomposed low-rank adaptation. ICML 2024 (Oral). arXiv:2402.09353. https://arxiv.org/abs/2402.09353
  3. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2024). QLoRA: Efficient fine-tuning of quantised LLMs. NeurIPS 2023. arXiv:2305.14314.
  4. Houlsby, N., Giurgiu, A., Jastrzebski, S., et al. (2019). Parameter-efficient transfer learning for NLP. ICML 2019. arXiv:1902.00751.
  5. Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimising continuous prompts for generation. ACL-IJCNLP 2021. arXiv:2101.00190.
  6. Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. EMNLP 2021. arXiv:2104.08691.
  7. Qiu, S., Chen, J., Zhu, C., et al. (2023). Controlling text-to-image diffusion by orthogonal fine-tuning. NeurIPS 2023 (OFT). arXiv:2306.07280.
  8. Kopiczko, D., Blankevoort, T., & Asano, Y. M. (2024). VeRA: Vector-based random matrix adaptation. ICLR 2024. arXiv:2310.11454.
  9. Hayou, S., Ghosh, N., & Yu, B. (2024). LoRA+: Efficient low-rank adaptation of large models. ICML 2024. arXiv:2402.12354.
  10. Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., & Raffel, C. (2022). Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. NeurIPS 2022 ((IA)³). arXiv:2205.05638.
  11. Aghajanyan, A., Gupta, S., & Zettlemoyer, L. (2021). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ACL 2021. arXiv:2012.13255.
  12. Kalajdzievski, D. (2023). A rank stabilisation scaling factor for fine-tuning with LoRA. arXiv:2312.03732.
  13. Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. NeurIPS 2020 (GPT-3). arXiv:2005.14165.
  14. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. NeurIPS 2017.
  15. Devlin, J., Chang, M. W., Lee, K., & Toutanova, (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019.
  16. Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: Open and efficient foundation language models. arXiv:2302.13971.
  17. Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2024). Visual instruction tuning. NeurIPS 2023 (LLaVA). arXiv:2304.08485.
  18. NVLabs. (2024). DoRA: Official PyTorch implementation. GitHub. https://github.com/NVlabs/DoRA
  19. NVIDIA Developer Blog. (2024). Introducing DoRA, a high-performing alternative to LoRA for fine-tuning. https://developer.nvidia.com/blog/introducing-dora-a-high-performing-alternative-to-lora-for-fine-tuning/
  20. Hugging Face. (2024). PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft
  21. Franco, A. V. (2025). QDoRA explained: The new PEFT standard for 2025. Medium. https://medium.com/@AntonioVFranco/qdora-explained-the-new-peft-standard-for-2025-5cf59afeb6ba
  22. arXiv. (2025). PEFT A2Z: Parameter-efficient fine-tuning survey for large language and vision models. arXiv:2504.14117.
  23. arXiv. (2026). Exploring parameter-efficient fine-tuning of billion-parameter vision models with QLoRA and DoRA. arXiv:2603.17782.
  24. arXiv. (2025). Not all directions matter: Toward structured and task-aware low-rank adaptation. arXiv:2603.14228.
  25. Encora. (2025). Comparing fine-tuning optimisation techniques: LoRA, QLoRA, DoRA, and QDoRA. https://www.encora.com/interface/comparing-fine-tuning-optimization-techniques-lora-qlora-dora-and-qdora
  26. ICML 2024. DoRA: Weight-decomposed low-rank adaptation — oral presentation. https://icml.cc/virtual/2024/oral/35576
  27. Liu, Y., et al. (2023). LoftQ: LoRA-fine-tuning-aware quantisation for large language models. arXiv:2310.08659.
  28. Liner AI. (2024). Quick review: DoRA — weight-decomposed low-rank adaptation. https://liner.com/review/dora-weightdecomposed-lowrank-adaptation

  DoRA in the Broader Foundation Model Adaptation Landscape

  The challenge that DoRA addresses — adapting very large foundation models to specific tasks, domains, and users efficiently — is one of the defining engineering problems of the 2024–2030 AI landscape. Foundation models have become sufficiently large (7B to 1T+ parameters) and expensive to pre-train (millions to hundreds of millions of USD) that maintaining multiple specialised model variants through full fine-tuning is economically impractical for most organisations. The solution architecture that has emerged is a shared foundation model with per-task or per-customer adapters — either merged (for latency-sensitive serving) or dynamically applied (for multi-tenant serving).

  DoRA occupies an important position in this landscape as the quality-optimised variant of LoRA: when the accuracy gap between LoRA and full fine-tuning would cause operationally significant degradation (in medical diagnosis, legal text analysis, code generation accuracy, or mathematical reasoning), DoRA provides a technically superior and economically viable alternative to full fine-tuning. The empirical finding that DoRA at rank r=8 outperforms LoRA at r=32 on several tasks is particularly consequential: it means that DoRA can achieve LoRA-r=32 quality at roughly 1/4 the trainable parameter count (since parameter count scales as r), reducing adapter storage, training time, and gradient communication in distributed fine-tuning proportionally.

  The trajectory of PEFT methods suggests continued specialisation and composability: foundation model providers (OpenAI fine-tuning API, Anthropic fine-tuning API, Google Vertex AI fine-tuning) are increasingly offering PEFT-based fine-tuning as managed services, with LoRA as the standard mechanism. As DoRA gains adoption in the research community and PEFT library ecosystem, it is likely to become an alternative option in these managed fine-tuning APIs, giving enterprise customers the ability to select their PEFT method based on task requirements.

  Multi-adapter composition (LoRA Hub, AdapterSoup, task arithmetic) allows combining adapters trained for different tasks or domains through linear operations on adapter weights. DoRA adapters, like LoRA adapters, can in principle be composed this way, with the magnitude vector participating in the composition alongside the low-rank matrices. Whether DoRA’s decomposition structure has better or worse composability properties than LoRA is an open research question with practical implications for multi-task serving architectures.

  Personalisation at scale — where individual users have personalised model behaviours encoded in small personal adapters — is an emerging application domain for PEFT methods. DoRA’s low parameter count (8–50 MB per adapter for 7B models at rank 16) makes per-user personalisation adapters economically feasible at scale if combined with efficient adapter serving infrastructure. This connects to privacy-preserving personalisation research where the adapter is learned on-device and never leaves the user’s device, with the base model served centrally and the personal adapter applied locally at inference time.

  Practical Training Recipes

  The following training recipes reflect community-established best practices for DoRA fine-tuning across common deployment scenarios. These are not from the original DoRA paper but represent the practical crystallisation of experience in the PEFT community as of 2025–2026.

  DoRA Fine-Tuning Recipe for LLaMA-3-8B on Commonsense Reasoning

  Hardware: Single NVIDIA RTX 4090 (24 GB) or A10G (24 GB) Base model: Meta-Llama-3-8B (bfloat16, 16 GB VRAM at full precision; 8 GB at 4-bit NF4) Configuration:

• PEFT method: DoRA (use_dora=True in LoraConfig)

• Adapter rank r: 16

• Adapter alpha: 32 (effective learning rate scaling factor r/alpha = 0.5)

• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

• Dropout: 0.05

• Quantisation: 4-bit NF4 (QDoRA) if VRAM < 24 GB, else bfloat16 base weights

• Optimizer: AdamW (weight_decay=0.01, lr=2e-4 for r=16)

• Scheduler: cosine with warmup (warmup_ratio=0.03)

• Per-device batch size: 4 sequences

• Gradient accumulation steps: 4 (effective batch = 16 sequences)

• Max sequence length: 2048 tokens

• Training epochs: 3

• Gradient norm clipping: 1.0

  QDoRA Recipe for 70B Models on 2×A100 80GB

  Hardware: 2× NVIDIA A100 80GB SXM4 with NVLink Base model: Meta-Llama-3-70B-Instruct (4-bit NF4, ~37 GB across 2 GPUs with FSDP) Configuration:

• PEFT method: DoRA (use_dora=True)

• Adapter rank r: 64

• Adapter alpha: 128

• Target modules: all linear layers

• Quantisation: 4-bit NF4 with double quantisation

• Optimizer: paged AdamW 8-bit (bitsandbytes) for memory efficiency

• Framework: FSDP with LoRA wrapper OR DeepSpeed ZeRO-2

• Per-device batch size: 2, gradient accumulation: 8, effective batch: 32

• Learning rate: 1e-4 with cosine schedule

• Training duration: typically 1–3 epochs for instruction fine-tuning datasets

  DoRA for Vision-Language Model Fine-Tuning (LLaVA-1.5-7B)

  Hardware: Single A100 40GB or 2× RTX 4090 Configuration:

• Apply DoRA to the LLM decoder component (Vicuna-7B) only

• LoRA rank r: 128 for visual instruction tuning (higher rank benefits multimodal tasks)

• Target modules: attention and MLP projections in LLM decoder

• Learning rate: 2e-4, cosine schedule, 1 epoch over visual instruction dataset

• MLP connector (vision-to-text projection): can be fine-tuned fully or with DoRA

• Vision encoder (CLIP ViT-L): typically frozen during instruction tuning

• Training data: LLaVA-1.5 instruction dataset (665K conversation samples)

  Common Failure Modes and Diagnostics

  Practitioners using DoRA report a consistent set of failure modes, most of which are inherited from LoRA and are not specific to DoRA’s additional components:

  Loss divergence in early training: Usually caused by learning rate that is too high (>5e-4 for most 7B models), insufficient warmup steps, or rank too high (r≥128) creating optimisation instability. Solution: reduce learning rate, increase warmup, or reduce rank. DoRA is slightly more robust to this than LoRA at equal settings due to the normalised directional parameterisation, but can still diverge at extreme learning rates.

  Loss plateau without convergence: Usually indicates learning rate too low (<1e-5), insufficient training epochs, rank too low (r<4) for the task complexity, or wrong target modules. Solution: increase learning rate by 2–5×, increase epochs, or extend to all linear layers. If DoRA shows similar plateau to LoRA at matched settings, check whether the task genuinely requires full fine-tuning quality.

  GPU OOM during forward pass: In QDoRA, OOM during the DoRA column-norm computation occasionally occurs because the dequantised W_0 + BA matrix (bfloat16) briefly occupies memory during norm computation before the quantised representation is reconstructed. Solution: reduce max_sequence_length, reduce per-device batch size, or enable gradient checkpointing.

  Merged adapter produces different behaviour than training: The merge operation W_final = (m / ||W_0 + BA||_c) ⊙ (W_0 + BA) can accumulate small floating-point errors if performed at low precision. Solution: perform the merge in float32 then convert the result to bfloat16 for deployment. The PEFT library’s merge_and_unload() function handles this correctly by default.

  DoRA performs worse than LoRA at the same rank: This occasionally occurs when the base model is already well-suited to the fine-tuning distribution (small distributional shift), in which case LoRA’s simpler parameterisation may have a slight advantage from reduced parameter interaction. Solution: evaluate whether the task genuinely benefits from DoRA’s additional expressiveness, or whether LoRA is sufficient. DoRA’s advantage grows with distributional shift.

  Ablation Studies and Design Choices

  The original DoRA paper and subsequent work have conducted systematic ablation studies that clarify which design choices are essential and which are incidental. The most fundamental ablation compares DoRA against LoRA with matched parameter counts by sweeping adapter ranks: DoRA at r=8 versus LoRA at r=8, r=16, r=32, and r=64. Across LLaMA-7B commonsense reasoning tasks, DoRA at r=8 outperforms LoRA at r=32 on 6 of 8 tasks — a result that demonstrates DoRA’s improved parameter efficiency is genuine and not merely a training dynamics advantage at matched rank.

  The necessity of the separate magnitude component m is confirmed by an ablation where m is fixed at its initialisation value (the column norms of W_0) and only BA is trained — effectively the Directional-LoRA baseline without magnitude learning. This directional-only variant performs worse than full DoRA but better than standard LoRA at matched rank, confirming that directional normalisation itself provides a benefit, and that adding the trainable magnitude m provides further improvement. The ablation further shows that training m with a higher learning rate than BA (similar in spirit to LoRA+‘s recommendation) provides marginal additional improvement.

  Column-wise normalisation (the default in DoRA) is compared against row-wise normalisation and full Frobenius-norm normalisation in ablations from the DoRA paper: column-wise normalisation consistently performs best, consistent with the observation that individual column vectors correspond to the output projection direction for individual input features, making column-wise semantics the natural scale of interest for language model weight matrices.

  The choice of which weight matrices to apply DoRA to is ablated systematically. Applying DoRA to all linear weight matrices in the model (Q, K, V, O projections plus up, gate, down MLP projections) provides the best performance. Restricting to only attention projections (LoRA’s default) is consistently inferior. The feed-forward MLP projections appear to be particularly important: ablating DoRA from MLP projections while retaining it on attention projections significantly degrades performance on reasoning-heavy benchmarks. This finding may reflect that feed-forward layers in Transformer Architecture models store factual knowledge (Geva et al., 2021), making directional updates to these matrices particularly important for domain adaptation tasks.

  Training duration ablations show that DoRA achieves higher accuracy than LoRA at the same number of training steps across all evaluated training durations (1, 2, 4, and 8 epochs on commonsense reasoning data), confirming that DoRA’s advantage is not merely a function of faster convergence to the same quality level but represents a higher achievable quality ceiling at comparable parameter budgets.

  Interaction with Model Quantisation

  The interaction between DoRA and model quantisation is richer than the standard QLoRA recipe and merits detailed treatment. In the standard QLoRA approach (Dettmers et al., 2023), the base model weights W_0 are quantised to 4-bit NF4 (Normal Float 4) format, while LoRA matrices A and B remain in bfloat16. During the forward pass, W_0 is dequantised block-by-block to bfloat16 as needed for matrix multiplication, then immediately discarded. The quantisation error — the difference between the dequantised approximation and the true bfloat16 weight — acts as a structured noise source during training, which the LoRA adapter learns to partially compensate for.

  In QDoRA, the same quantisation scheme applies: W_0 is stored in 4-bit NF4, while m and BA remain in bfloat16. The DoRA forward pass then computes the column norms of (W_0_dequantised + BA) — the dequantised weight perturbed by the current directional update — and applies the magnitude normalisation with m. This means that the column norm computation occurs in bfloat16 after dequantisation, which is correct and numerically stable. The quantisation error in W_0 perturbs the column norms of the directional matrix, but since this error is small and structured (NF4 is specifically designed to minimise the quantisation error for normally-distributed weight values typical of large pretrained models), the effect on column norm estimation is minor.

  LoftQ (Liu et al., 2023) proposes an improved initialisation for quantised LoRA training by alternating SVD-based initialisation and quantisation to find initial A and B matrices that compensate for the quantisation error from the start of training. This approach is directly compatible with DoRA: applying LoftQ initialisation to the BA matrices while initialising m to the column norms of the LoftQ-compensated initial weight provides a better starting point for QDoRA training. Experiments with LoftQ-DoRA (not yet widely reported) are expected to show improvements particularly at very low bit-widths (3-bit or 2-bit quantisation) where quantisation error is large enough to meaningfully affect training quality.

  Historical Context and Intellectual Lineage

  The development of DoRA is best understood as the latest step in a decade-long progression from full fine-tuning through increasingly efficient adaptation methods, each step motivated by the growing scale of the models being adapted. When BERT (Devlin et al., 2018, 340M parameters) was released, fine-tuning the full model on a single GPU for a few hours was standard practice. By 2020, GPT-3 (175B parameters) made full fine-tuning economically impractical for most organisations. By 2023, Llama 2 (7B–70B parameters) was accessible to academic researchers, but full fine-tuning of even the 7B variant required 40GB+ VRAM and multiple GPUs, while LoRA made it accessible on a single consumer GPU.

  The intellectual lineage of weight decomposition in neural networks extends back to weight factorisation research in model compression: singular value decomposition (SVD)-based weight factorisation (Sainath et al., 2013; Denton et al., 2014; Zhang et al., 2015) was used to compress trained networks by decomposing weight matrices into low-rank products, discarding small singular values. These compression methods operated post-hoc on already-trained networks; LoRA and DoRA transposed the concept to the adaptation setting, learning the low-rank update from scratch rather than compressing a completed model. The polar decomposition that DoRA exploits (W = m ⋅ V/||V||) is conceptually related to the SVD in that both exploit the structure of weight matrices as elements of a Riemannian manifold, but DoRA’s column-wise rather than spectral decomposition is a simpler factorisation that avoids the computational cost of full SVD.

  The orthogonal fine-tuning direction (OFT, Qiu et al., 2023; BOFT, Liu et al., 2023) applied group-theoretic constraints — specifically requiring weight updates to be elements of the orthogonal group O(d) — to preserve Euclidean geometry of pretrained representations. DoRA can be read as a relaxation of OFT’s orthogonal constraint: where OFT requires ΔV to be a rotation, DoRA requires only that ΔV lie in a low-rank subspace. This relaxation is necessary because some fine-tuning tasks require directional changes that no rotation can represent (e.g., tasks requiring dramatic reorientation of some weight columns), and the low-rank constraint alone provides sufficient inductive bias for the parameter efficiency goal.

  Security, Privacy, and Governance Considerations

  Fine-tuning with DoRA on sensitive datasets introduces security and governance considerations that are particularly salient in UK and EU regulatory contexts. The EU AI Act (applicable to UK providers serving EU markets, as of August 2024) requires transparency about fine-tuning data and adaptation methods for high-risk AI systems, including those used in employment, education, healthcare, and law enforcement. DoRA fine-tuned models must maintain clear provenance — which base model was used, on what data, with what PEFT method — to satisfy AI system documentation requirements under the Act.

  Gradient leakage from fine-tuning (analogous to the gradient inversion attacks described in Zhu et al., 2019, for full model training) applies to PEFT fine-tuning: gradient information about the magnitude vector m and the LoRA matrices A and B can, in principle, be inverted to reconstruct training examples. However, because DoRA (and LoRA) gradients are extremely compressed relative to full model gradients (the adapter has 0.1% of the full model’s parameters), the practical gradient leakage risk is substantially lower than for full fine-tuning. For federated DoRA fine-tuning across NHS trusts, applying differential privacy noise to the adapter gradient updates before aggregation provides a quantifiable privacy guarantee; the small size of DoRA adapters relative to full model gradients means the DP noise-to-signal ratio is more favourable than for full-gradient federated learning.

  Model watermarking for DoRA-fine-tuned models is an emerging concern: if a fine-tuned adapter is shared publicly (e.g., on the Hugging Face Hub) and an adversary downloads the adapter, merges it with the base model, and deploys the merged model commercially, proving that the deployed model is derived from your fine-tuning without inspecting its weights is difficult. Research on post-hoc adapter watermarking (embedding a detectable signal in the adapter weights that survives the merge operation) is an active area directly motivated by the proliferation of PEFT adapters on model sharing platforms.

  UK AI safety evaluation frameworks (UK AI Safety Institute, DSIT) apply to fine-tuned models as well as base models: a DoRA-adapted model used in a safety-critical application must be evaluated for harmful capability uplift introduced by the fine-tuning. The UK AI Safety Institute’s model evaluation protocols include fine-tuning red-teaming assessments that specifically test whether domain adaptation via PEFT methods can bypass safety refusals trained into the base model — a documented concern since early 2024 demonstrations that even small LoRA adapters could partially remove safety training.

  Comparative Evaluation Framework

  A comprehensive evaluation of DoRA versus other PEFT methods should assess multiple dimensions beyond aggregate accuracy:

  Parameter Efficiency Ratio (PER): Accuracy improvement per additional trainable parameter relative to base model. DoRA typically achieves higher PER than LoRA because its accuracy improvement is greater at matched parameter count; at the same PER target, DoRA often requires fewer parameters than LoRA.

  Rank Sensitivity: How much does performance vary as rank r is changed from 4 to 128? DoRA is empirically more rank-robust than LoRA, maintaining high performance at lower ranks. This means practitioners can use smaller adapters with DoRA while maintaining equivalent accuracy to larger LoRA adapters.

  Distributional Shift Sensitivity: PEFT methods perform differently depending on how similar the fine-tuning distribution is to the pretraining distribution. For small shifts (e.g., style adaptation of a general English model to formal English), all PEFT methods perform similarly and LoRA suffices. For large shifts (e.g., adapting an English model to Mandarin scientific text, or a general model to medical clinical notes), DoRA’s improved directional control provides measurable advantages.

  Inference Latency (merged): After weight merging, all PEFT methods (LoRA, DoRA, VeRA) produce identical inference latency to the unmodified base model, since the adapter is absorbed into the weight. This is in contrast to adapter module methods (Houlsby adapters) which add latency proportional to adapter width and number of adapter layers.

  Multi-Adapter Serving Latency (unmerged): When adapters must be applied dynamically without merging (e.g., in multi-tenant serving where different requests use different adapters), LoRA and DoRA require running the base model forward pass plus an additional low-rank matrix multiplication per adapted layer. The latency overhead is proportional to r × (d + k) arithmetic operations per layer — for r=16 and d=k=4096, approximately 131,072 additional multiply-accumulate operations per layer, adding roughly 1% to per-layer latency.

  Training Stability (rank scaling): At very high ranks (r=256 or r=512), LoRA training can become unstable due to the large number of trainable parameters in the A matrix interacting with the fixed random initialisation. DoRA inherits this instability for the BA matrices but may be somewhat more robust because the magnitude vector m provides an additional degree of freedom that absorbs some of the optimisation pressure.

  Key Terminology Glossary

• Weight decomposition: Factorisation of a weight matrix W into magnitude m (column L2 norms, a 1×k vector) and direction V (unit-normalised columns, a d×k matrix): W = m ⋅ (V / ||V||_c).

• Magnitude component: The trainable vector m ∈ ℝ^{1×k} holding the per-column scale of the adapted weight. Initialised to column norms of W_0; adds only k parameters per weight matrix relative to LoRA.

• Directional component: The unit-normalised column direction matrix (W_0 + BA) / ||W_0 + BA||_c; updated via the low-rank perturbation BA during training.

• Column-wise norm ||V||_c: The L2 norm of each column of V, used to normalise direction. Recomputed at each forward pass over the current adapted direction W_0 + BA.

• Low-Rank Adaptation (LoRA): The mechanism used within DoRA for the directional update: ΔV = BA, B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, r ≪ min(d, k).

• QDoRA: DoRA combined with 4-bit NF4 base weight quantisation as in QLoRA. Achieves near-full-fine-tuning quality on consumer GPUs with 24 GB VRAM.

• DVoRA: Combination of DoRA’s magnitude-direction decomposition with VeRA’s shared frozen random matrices. Achieves comparable performance to DoRA at much fewer trainable parameters.

• Full fine-tuning (FT): Updating all model parameters; the gold-standard quality baseline against which all PEFT methods are benchmarked.

• PEFT (Parameter-Efficient Fine-Tuning): The family of methods (adapters, LoRA, DoRA, prefix tuning, (IA)³, prompt tuning) that adapt pretrained models by updating only a small fraction of total parameters.

• Weight merging: Post-training fusion of DoRA adapter weights (m, A, B) back into the base weight matrix W_final = (m / ||W_0 + BA||_c) ⋅ (W_0 + BA) for zero-overhead inference; the merged weight replaces W_0 entirely.

• Adapter rank r: The dimensionality of the low-rank matrices A and B; controls the capacity of the directional update subspace. DoRA maintains high quality at lower r than LoRA.

• Polar decomposition: The mathematical factorisation W = m ⋅ V that DoRA exploits, where m ≥ 0 and V has unit column norms; analogous to the polar form of complex numbers or the polar decomposition of matrices in numerical linear algebra.

• Column-wise adaptation: The property of DoRA that each column of the adapted weight matrix W’ has independently learned magnitude and directional update, enabling fine-grained control over individual output projections.

• LoftQ: LoRA Fine-Tuning-aware Quantisation; improved initialisation strategy for LoRA (and DoRA) adapters on quantised base models by alternating SVD and NF4 quantisation to minimise initialisation error.

• NF4 (Normal Float 4): 4-bit quantisation format designed for normally-distributed neural network weights; used in QLoRA and QDoRA to store base model weights at 4-bit precision with minimal information loss.

• Adapter card: Metadata specification on the Hugging Face Hub for published PEFT adapters, documenting base model, adapter method (LoRA, DoRA), rank, target modules, training data, and task type.

• use_dora: Boolean flag in Hugging Face PEFT LoraConfig that enables DoRA instead of standard LoRA; the primary configuration change required to use DoRA in Hugging Face-based training pipelines.

• dvora: Combination of DoRA (magnitude-direction decomposition) with VeRA (Vector-based Random Matrix Adaptation); achieves DoRA-level quality at significantly fewer trainable parameters.

• Hyperspherical energy: The distribution of weight vector orientations on the unit sphere, which OFT and DoRA both aim to preserve or control during fine-tuning; concept from the sphere packing literature adapted to neural network fine-tuning.

• Learning rate multiplier: In PEFT frameworks, a per-parameter-group scaling factor applied to the base learning rate; recommended to use different values for A, B, and m in DoRA following the LoRA+ findings.

• Rank-r update subspace: The r-dimensional column span of B within the d-dimensional weight output space; limits the directional changes DoRA can represent to this subspace, with r determining the update capacity.

• Gradient inversion attack: A security attack (Zhu et al., 2019) that reconstructs training examples from observed gradient values; relevant to federated DoRA fine-tuning over sensitive datasets, motivating DP-SGD adapter training.

• Parameter Efficiency Ratio (PER): Composite evaluation metric measuring accuracy improvement per additional trainable parameter; DoRA typically achieves higher PER than LoRA at matched adapter rank.

• Weight merging: Post-training operation that absorbs DoRA adapter parameters (m, A, B) into the base weight matrix, producing a standard model weight with zero inference overhead; implemented via PEFT merge_and_unload().

• Distributional shift: The degree to which the fine-tuning data distribution differs from the pretrained model’s training distribution; larger shift generally increases DoRA’s advantage over LoRA by requiring larger directional updates.

• Multi-tenant serving: Deployment pattern where a single base model in GPU memory serves multiple requests using different per-request adapters selected without weight re-merging; supports hot-swapping DoRA/LoRA adapters between requests.

• SVD (Singular Value Decomposition): Matrix factorisation W = UΣV^T decomposing a weight matrix into left singular vectors U, singular values Σ, and right singular vectors V; related to but distinct from DoRA’s column-wise polar decomposition.

• Chinchilla optimal: Compute allocation strategy prescribing roughly 20 tokens of training data per parameter; guides data collection for DoRA fine-tuning (though at adapter scale the prescription is less relevant than at full pretraining scale).

• LoRA Hub: A framework and model hub for composing multiple LoRA adapters through linear combination; applicable in principle to DoRA adapters, enabling multi-skill model composition without separate per-skill models.

• Task arithmetic: The operation of adding and subtracting LoRA/DoRA adapter weight vectors to compose capabilities; W_multi-task = W_0 + λ_1 ΔW_task1 + λ_2 ΔW_task2 for multiple task adapters weighted by λ.

• RLHF fine-tuning with DoRA: Application of DoRA during PPO, DPO, or GRPO alignment training; enables efficient alignment of 7B–70B models on human preference data without full parameter fine-tuning, compatible with TRL and OpenRLHF frameworks.

• Activation memory: GPU memory consumed by storing intermediate activation tensors during the forward pass for use in backpropagation; not directly affected by DoRA (which reduces parameter memory, not activation memory) but managed alongside adapter training via gradient checkpointing.

• Effective rank: The numerical rank of the adapted weight change ΔW = (m/||W_0+BA||_c)⊙(W_0+BA) - W_0; DoRA’s effective rank is bounded by r (from the BA matrices) but the column-wise magnitude scaling can increase the effective rank of the total adaptation relative to the directional update alone.

  Supplementary Notes on Model Families

  DoRA has been validated primarily on the LLaMA family (LLaMA-2-7B, LLaMA-2-13B, LLaMA-3-8B) and the LLaVA vision-language family, but its applicability across Transformer Architecture model families has been confirmed by independent practitioners. The following notes document family-specific considerations:

  LLaMA 3 family (Meta, 2024–2026): The LLaMA 3.1 and 3.2 families introduced a GQA (Grouped Query Attention) attention mechanism where the number of K and V heads is smaller than Q heads. DoRA applies naturally to Q, K, V projections regardless of GQA grouping; the smaller K and V projections (if k_dim < q_dim) have correspondingly smaller m vectors. LLaMA 3.2 Vision and 3.3 text models are fully supported by PEFT DoRA.

  Mistral and Mixtral families (Mistral AI, 2023–2025): Mistral 7B and Mixtral 8×7B (MoE variant) are fully compatible with DoRA via PEFT. For Mixtral, DoRA can be applied to each expert’s MLP projections independently (per-expert DoRA adaptation) or with shared adapters across experts (parameter-efficient but less expressive). The routing mechanism in Mixtral is typically left unchanged during DoRA fine-tuning.

  Gemma family (Google DeepMind, 2024–2025): Gemma 2B, 7B, and 9B are supported by PEFT DoRA through the Hugging Face Transformers Gemma model class. The Gemma architecture uses logit soft-capping (tanh-based output scaling) and interleaved local and global attention in Gemma 2; DoRA adapts naturally to both attention variants.

  Qwen family (Alibaba, 2023–2025): Qwen2 and Qwen2.5 (7B, 14B, 32B, 72B) are supported. Qwen models use a tied embedding (the input embedding and the language model head share weights), which requires careful handling in DoRA fine-tuning: the tied weights should either both be adapted with DoRA (using the same adapter parameters) or both left frozen to avoid inconsistency.

  Phi family (Microsoft Research, 2023–2025): Phi-3-mini and Phi-3-medium are small but highly capable models where DoRA fine-tuning has been applied for specialised coding and reasoning tasks. The smaller parameter count (3.8B and 14B) makes DoRA fine-tuning particularly accessible, with QDoRA for Phi-3-mini runnable on 16 GB VRAM consumer hardware.

  Adapter Sharing and Community Ecosystem

  A substantial ecosystem of publicly shared DoRA and LoRA adapters has developed on the Hugging Face Hub since mid-2024, spanning thousands of fine-tuned adapters for specific domains, languages, and tasks. The Hub’s PEFT adapter card specification (introduced 2023) provides standardised metadata enabling users to identify whether a shared adapter uses DoRA or LoRA, which base model it targets, and its training configuration. As of mid-2026, DoRA adapters represent an estimated 15–20% of newly uploaded PEFT adapters on the Hugging Face Hub, reflecting rapid adoption after PEFT library native support was established.

  The Open LLM Leaderboard and lm-evaluation-harness (EleutherAI) provide standardised evaluation infrastructure that enables fair comparison of DoRA-fine-tuned models against LoRA-fine-tuned models and full fine-tuned baselines across commonsense, mathematical, code generation, and instruction following benchmarks. Systematic community evaluations submitted to these leaderboards have independently validated the DoRA paper’s claimed improvements across diverse model families and task types, establishing DoRA’s performance advantage as a reproducible finding rather than a cherry-picked result from the original paper’s evaluation setup.

  The practitioner community has developed several DoRA-specific tooling extensions beyond the core PEFT library: adapter merging scripts that correctly handle the DoRA normalisation during merge; rank selection heuristics based on task difficulty and distributional shift estimates; and monitoring dashboards that track DoRA-specific training metrics including magnitude vector evolution and directional update magnitude over training. These community extensions reflect the maturity of DoRA as a production-ready tool, not merely an academic technique.

  Commercial fine-tuning APIs (OpenAI fine-tuning API, Anthropic fine-tuning API, Together AI, Replicate, Modal) have begun offering DoRA as a fine-tuning option alongside LoRA, providing managed infrastructure that handles all aspects of DoRA fine-tuning including hyperparameter selection, training monitoring, and adapter storage. The pricing model for DoRA fine-tuning on these platforms is typically the same as LoRA (per-token training cost plus adapter storage) since the computational overhead is negligible. This managed API access further democratises DoRA adoption, particularly for organisations without ML engineering capacity to configure and monitor their own fine-tuning infrastructure.

  Connections to Continual Learning and Catastrophic Forgetting

  Fine-tuning large pretrained models on task-specific data creates the risk of catastrophic forgetting: the adapted model improves on the target task but degrades on the general capabilities encoded during pretraining. This trade-off is particularly acute for PEFT methods because the limited parameter budget constrains how much the model can specialise to the task without modifying the pretrained knowledge representation. DoRA’s weight decomposition offers a potential mechanism for mitigating catastrophic forgetting: the magnitude vector m can be regularised to stay close to its initialisation (the column norms of W_0), while the directional update BA is given more latitude to adapt. This asymmetric regularisation strategy — loose on direction, tight on magnitude — could preserve the scale of pretrained weight features while allowing directional specialisation, a potentially better trade-off than symmetric L2 regularisation on ΔW applied indiscriminately to all components.

  Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2017) and progressive neural networks address catastrophic forgetting in continual learning by penalising changes to weights deemed important for previously learned tasks. Applied to DoRA, an EWC-inspired approach would estimate Fisher information for the magnitude vector m and the directional component separately, applying stronger penalties to the component carrying more pretrained information. Research at Edinburgh’s School of Informatics (in collaboration with the Alan Turing Institute continual learning programme) is exploring this direction for NHS clinical AI applications where a foundation model must be continually adapted to new clinical protocols without forgetting established clinical reasoning capabilities.

  The adapter merging perspective on DoRA connects naturally to continual learning: by maintaining separate DoRA adapters for each sequential task and merging them with a weighted combination, a practitioner can construct a model that balances capabilities across all tasks without a single fine-tuning run that must simultaneously optimise for all tasks. Task arithmetic applied to DoRA adapters (conceptually: W_multi = W_0 + λ_1 ΔW_task1 + λ_2 ΔW_task2 where ΔW_i is the merged delta for each task) provides a simple and practical approach to continual adaptation that avoids explicit catastrophic forgetting mitigation during training.

  Theoretical Open Problems

  Several theoretical questions about DoRA remain open and are active research targets in the PEFT community. The relationship between DoRA’s empirical advantage over LoRA and the intrinsic dimensionality of the fine-tuning update is not fully characterised: while Aghajanyan et al.’s intrinsic dimensionality results explain why low-rank updates are sufficient in principle, they do not predict why separating magnitude and direction within the low-rank update improves performance. A theoretical account connecting DoRA’s decomposition to the geometry of the pretrained weight manifold and the structure of fine-tuning objectives would strengthen the method’s theoretical foundation.

  The optimal magnitude parameterisation is not definitively established: DoRA uses column-wise L2 norms, but row-wise norms, per-element scaling (a full d×k magnitude matrix), or spectral-norm-based scaling might provide better performance on certain tasks. Systematic ablations across these variants have been conducted for specific model families but not comprehensively across all relevant architectures and tasks.

  The composability of DoRA adapters under task arithmetic and adapter merging is not well-characterised theoretically. For LoRA adapters, linear composition (W_composed = W_0 + Σ_i λ_i B_i A_i) is natural because the update is a linear sum. For DoRA adapters, the magnitude normalisation makes linear composition in the full adapted weight space nontrivial: merging two DoRA adapters trained on different tasks requires deciding how to combine not only the BA matrices but also the m vectors, and the interaction between the two sets of magnitude vectors is not commutative in general. Developing a principled composition algebra for DoRA adapters is an open theoretical problem with direct practical implications for multi-task model serving.

  The computational overhead of the DoRA normalisation step at extreme model scale is a practical concern not yet rigorously analysed. For models with weights of shape d×k where d, k ≫ 4096 (as in some MoE expert matrices or very wide model variants), the column-norm computation across k columns each of d dimensions may contribute measurably to training step latency. Kernel fusion — combining the norm computation, normalisation, and magnitude application into a single CUDA kernel that reads the combined weight matrix only once from HBM — is a natural optimisation analogous to FlashAttention’s IO-aware kernel design. A fused DoRA kernel that eliminates the HBM read-write bottleneck in the normalisation step would further reduce DoRA’s overhead below the already negligible levels reported in the original paper.

  The interaction of DoRA with model quantisation at inference time (post-training quantisation of the merged DoRA-adapted weight) has not been comprehensively studied. The column-wise magnitude normalisation in DoRA may produce merged weight matrices with better quantisation properties than equivalent LoRA-merged weights, because the normalisation ensures that column vectors have unit norm before magnitude scaling, potentially reducing outlier values that degrade quantisation accuracy. Empirical validation of this hypothesis across GPTQ, AWQ, SmoothQuant, and GGUF quantisation methods applied to DoRA-merged models would be practically valuable for deployment engineers targeting low-bit deployment.

  Research into mechanistic interpretability of DoRA-adapted models — understanding which circuits and attention head behaviours change as a result of the DoRA adaptation, and whether the directional versus magnitude decomposition corresponds to interpretable distinctions in model behaviour — is an emerging area. Anthropic’s mechanistic interpretability work (sparse autoencoders, circuit analysis) provides tools that could in principle be applied to compare DoRA-adapted versus LoRA-adapted versus full-fine-tuned models, with the hypothesis that DoRA’s decoupled update geometry produces circuits that are more cleanly interpretable or that differ from pretrained circuits in more structured ways than LoRA adaptations.

  Summary of Advantages and Limitations

  Advantages of DoRA over LoRA:

• Higher accuracy on tasks with large distributional shift from pretraining domain (commonsense reasoning, domain-specific NLP, visual instruction tuning)

• Better parameter efficiency: achieves LoRA-r=32 quality at DoRA-r=8 on several benchmarks

• More robust training dynamics at matched hyperparameters due to normalised directional parameterisation

• Zero inference overhead after weight merging, identical to LoRA deployment

• Negligible training overhead (column-norm computation < 0.025% of total FLOPs)

• Compatible with QLoRA, enabling QDoRA for consumer-GPU fine-tuning

• Native Hugging Face PEFT library support with single-flag enablement

  Limitations of DoRA relative to LoRA:

• Marginal additional memory: magnitude vector m adds k parameters per adapted weight matrix

• Slightly more complex implementation than LoRA (requires column-norm computation and dynamic normalisation)

• Column-norm computation requires dequantised weight matrix in memory transiently during forward pass

• Theoretical composition properties (task arithmetic, adapter merging) are less well-characterised than for LoRA

• Less adoption in commercial fine-tuning APIs and tools compared to the more established LoRA

• May not outperform LoRA for tasks with small distributional shift where LoRA already achieves near-full-FT accuracy

  Tasks where DoRA is strongly recommended (large distributional shift, quality-critical):

• Medical NLP (clinical note processing, ICD coding, radiology report generation)

• Legal domain adaptation (contract analysis, regulatory compliance, case summarisation)

• Mathematical reasoning (GSM8K, MATH, proof generation)

• Code generation for specialised APIs and internal codebases

• Multilingual adaptation for low-resource languages underrepresented in pretraining data

• Vision-language instruction tuning with complex compositional understanding requirements

  Tasks where LoRA is sufficient (small distributional shift, efficiency-critical):

• Style adaptation within general English (formal vs informal tone)

• Instruction following for tasks well-represented in instruction fine-tuning datasets

• Format adherence (JSON output, structured response formatting)

• Minor factual update (adding recently published information to a general model)

Provenance