CodeFormer

CodeFormer is a Transformer-based blind face restoration model that combines a discrete codebook prior — learned via a vector-quantised autoencoder — with a controllable fidelity-quality trade-off mechanism, allowing it to recover high-quality facial details from severely degraded inputs such as low-resolution, compressed, or heavily noisy images. Unlike earlier GAN-based restoration approaches, CodeFormer’s codebook provides rich, semantically plausible facial priors that guide reconstruction without requiring clean reference images, and its fidelity weight parameter lets users tune the balance between realism and fidelity to the original degraded input. The model generalises to face enhancement tasks in AI-generated images, old photo restoration, and video face restoration.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:hasPart ai:VectorQuantizedAutoencoder))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:hasPart ai:CodebookPrior))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:hasPart ai:TransformerDecoder))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:hasPart ai:FidelityWeightController))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:hasPart ai:FaceDetectionModule))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:hasPart ai:ConvolutionalEncoder))

Dependency Relationships

SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:requires ai:GPUCompute))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:requires ai:FFHQDataset))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:requires ai:FaceAlignment))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:dependsOn ai:DeepLearning))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:dependsOn ai:TrainingData))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:dependsOn ai:ImageProcessing))

Capability Relationships

SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:enables ai:BlindFaceRestoration))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:enables ai:VideoRestoration))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:enables ai:OldPhotoRestoration))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:enables ai:AIImageEnhancement))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:supports ai:FilmVFX))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:supports ai:ForensicImaging))

Implementation Relationships

SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:implements ai:BlindImageRestoration))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:implements ai:CodebookLookup))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:implements ai:DiscreteLatentRepresentation))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:uses ai:Transformer))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:uses ai:VectorQuantization))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:uses ai:SelfAttention))

Reduction Relationships

SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:reducesTo ai:ImageRestoration))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:reducesTo ai:SuperResolution))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:contrastsWith ai:GenerativeAdversarialNetwork))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:contrastsWith ai:DiffusionModel))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:contrastsWith ai:GFPGAN))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:contrastsWith ai:VQFR))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:relatedTo ai:AIUpscalingAndSuperResolution))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:relatedTo ai:SUPIR))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:relatedTo ai:DiffBIR))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:bridgesTo ai:ForensicImaging))
SubClassOf(ai:CodeFormer
  ObjectSomeValuesFrom(ai:bridgesTo ai:MedicalImaging))

About

CodeFormer was developed at S-Lab, Nanyang Technological University, Singapore, by Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. It was published at NeurIPS 2022 under the title “Towards Robust Blind Face Restoration with Codebook Lookup Transformer.” The motivation for the work was the persistent failure mode of GAN-based face restoration methods — approaches such as GFPGAN and PSFRGAN that were dominant at the time — which either hallucinate incorrect facial features at high degradation levels (where the adversarial generator must invent plausible content not constrained by any reference) or produce over-smoothed outputs when constrained tightly to the degraded input. The fundamental problem is that blind restoration of severely degraded faces is a massively under-constrained inverse problem: many different high-quality faces are consistent with a given low-resolution, blurry, or noisy observation. GANs resolve this ambiguity through learned distributional priors, but those priors can be inconsistent with the specific identity in the degraded image.

CodeFormer resolves this ambiguity differently: it learns a discrete vocabulary — a codebook — of high-quality facial feature tokens from the FFHQ dataset during a pre-training phase, then at restoration time looks up the most plausible codebook tokens given the degraded observation, supervised by a Transformer that attends globally over all codebook entries simultaneously. This codebook prior idea drew direct inspiration from VQGAN and VQVAE discrete latent space representations that had proven powerful in generative image modelling, repurposed here as a structured prior for restoration rather than generation. A concurrent line of work, VQFR (Vector-Quantized Face Restoration, ECCV 2022), independently explored the same codebook idea; CodeFormer differentiates itself through the global Transformer attention mechanism and the controllable fidelity weight that VQFR lacks.

The fidelity weight mechanism is CodeFormer’s most practically significant innovation. When w = 0, the decoder relies entirely on codebook-selected tokens, producing faces that are maximally photorealistic and detailed but may drift from the original identity — useful for creative enhancement of AI-generated portraits. When w = 1, the decoder is guided primarily by features extracted directly from the degraded input, maximising identity fidelity at some cost to perceptual quality — appropriate for forensic applications where identity preservation is paramount. Values in between (typically w = 0.5–0.7 for general restoration) achieve a practical balance. This single scalar parameter replaces the complex fidelity-quality trade-off that previously required selecting entirely different models.

The model achieves state-of-the-art results on blind face restoration benchmarks at its release in 2022, outperforming GFPGAN, DFDNet, PULSE, PSFRGAN, GLEAN, and GPEN on CelebA-Test across PSNR (22.18 vs. GFPGAN’s 20.37), SSIM (0.610 vs. 0.545), LPIPS, FID, and identity similarity (IDS) metrics. On the FFHQ dataset (70,000 high-quality faces at 1024×1024 resolution used for codebook training), the model learns a rich vocabulary of facial components covering eyes, nose, mouth, skin texture, hair, and accessories, enabling semantically coherent reconstruction rather than pixel-level pattern matching.

Components / Architecture

CodeFormer’s architecture has three main stages, each trained sequentially:

Stage 1 — Codebook Learning (VQGAN-style pre-training). A convolutional encoder maps high-quality face images (from FFHQ) into a spatial feature grid. A vector quantisation module replaces each spatial feature vector with its nearest neighbour in a learnable codebook of N = 1024 codes, each of dimension D = 256. A convolutional decoder reconstructs the face from the quantised features. Training uses a perceptual loss (VGG features), a pixel-wise L1 loss, and a GAN-style discriminator loss to ensure high perceptual quality of reconstructions. After Stage 1, the codebook contains a rich vocabulary of facial tokens. The encoder and decoder weights are frozen.

Stage 2 — Transformer Training. The frozen convolutional encoder processes a degraded face and produces a (degraded) feature grid. A Transformer with multi-head self-attention and cross-attention layers maps this degraded feature grid to a sequence of code indices that select the most plausible codebook entries. The Transformer is trained with a cross-entropy loss against the ground-truth code indices derived from the corresponding high-quality face. This stage teaches the model to perform codebook lookup rather than learning to generate features from scratch.

Stage 3 — Controllable Fidelity-Quality Trade-off. A linear interpolation is introduced at the decoder’s input between the quantised codebook tokens (quality branch) and the degraded feature vectors (fidelity branch), controlled by the scalar weight w. The decoder is fine-tuned with this interpolation to maintain coherent reconstruction across the full range of w. In practice, the interpolated feature h = w · f_degraded + (1 − w) · f_codebook is fed to the pre-trained decoder.

Face detection and alignment preprocessing. CodeFormer includes a preprocessing pipeline that detects and aligns faces using a face detector (RetinaFace or similar) and a facial landmark model, cropping and warping faces to a canonical 512×512 frame before restoration. Post-processing blends the restored face back into the original image context. This preprocessing is transparent to the user in ComfyUI and Automatic1111 integrations.

Model size and inference speed. The full CodeFormer model is approximately 348 MB on disk. On an NVIDIA RTX 3090, it processes a single 512×512 face in approximately 100–250 ms. The inference is single-pass (no iterative denoising), which distinguishes it from diffusion-based alternatives and makes it practical for video processing at 4–10 frames per second.

Use Cases / Major Families

Old photograph and archival restoration. CodeFormer excels at recovering facial detail from scanned historical photographs exhibiting compound degradation — fading, JPEG compression from digitisation, film grain, and blur. Film archives, national heritage institutions, and genealogy services use CodeFormer as part of restoration workflows. The fidelity weight set to w ≈ 0.7 is recommended to preserve historical identity while recovering structural detail.

AI-generated face enhancement. When diffusion models or GANs generate facial images, the output can exhibit subtle artefacts — slight asymmetry, texture inconsistency, or uncanny valley characteristics at the eye level. Running CodeFormer with w ≈ 0.3–0.5 on AI-generated faces produces images with enhanced realism. This application is heavily used in the Stable Diffusion and Midjourney community, where CodeFormer is a standard post-processor in both ComfyUI and Automatic1111 pipelines.

Video face restoration. CodeFormer supports video input by processing frame-by-frame. Video temporal consistency remains a limitation (it is not jointly trained on video, so inter-frame flickering can occur), but community solutions combine CodeFormer with RIFE frame interpolation and temporal blending to smooth artefacts. Research successors such as KEEP (2024) address temporal consistency explicitly. The facerestore_advanced ComfyUI node includes an intelligent quality assessment fallback that applies CodeFormer only on frames where quality falls below a threshold.

Social media and consumer photography applications. Applications such as Remini, PixelRetouch, and various mobile camera apps integrate CodeFormer-style codebook restoration for automatic “portrait enhancement.” The model’s speed and open-source availability enable on-device deployment on constrained hardware with model quantisation.

Forensic imaging. Law enforcement and forensic image analysts use face restoration to recover identifiable features from low-quality surveillance footage or heavily compressed crime scene images. At high fidelity weights (w ≈ 0.9–1.0), CodeFormer is configured to maximally preserve identity-linked features while removing degradation artefacts, producing outputs suitable for expert forensic comparison. Care must be taken to document that restoration was applied, as restored images carry epistemic uncertainty about hallucinated high-frequency detail.

Medical imaging adaptation. Research groups have fine-tuned the CodeFormer codebook framework on medical imaging domains (CT, MRI, histopathology) where blind restoration of fine anatomical structure from under-sampled or compressed acquisitions poses similar mathematical challenges to face restoration. Domain-adaptive codebooks trained on medical image corpora replace the face-specific FFHQ codebook.

Integration with SUPIR upscaling. The AI Upscaling and Super-Resolution page documents SUPIR’s recommended workflow: large-image upscaling via SDXL-based diffusion, followed by a CodeFormer face restoration pass at w ≈ 0.5 to restore fine facial details that diffusion upscaling can sometimes over-smooth. This hybrid pipeline — diffusion for global content, CodeFormer for faces — has become a community standard for portrait upscaling in ComfyUI.

Academic Context

CodeFormer emerges from a lineage of generative prior approaches to face restoration that began with early dictionary-based methods and GAN-based priors. The foundational paper establishing GAN-based facial prior restoration was GFPGAN (Wang et al., 2021), which uses a pre-trained StyleGAN2 generator as a facial quality prior and exploits its latent space for restoration. DFDNet (Li et al., 2020) used facial component dictionaries of eyes, nose, and mouth features extracted from high-quality faces — conceptually similar to CodeFormer’s codebook but limited to component dictionaries rather than a continuous learned vocabulary. PSFRGAN (Chen et al., 2021) and GLEAN (Chan et al., 2021) explored different formulations of generative priors.

CodeFormer’s theoretical foundation in vector quantisation draws from the VQVAE paper (van den Oord et al., NeurIPS 2017), which demonstrated that discrete latent codes can capture the essential structure of complex distributions without posterior collapse, and VQGAN (Esser et al., CVPR 2021), which showed that combining VQ with adversarial training and a Transformer-based autoregressive prior produces high-quality generative models. The choice of a Transformer — rather than a recurrent model — for codebook lookup is motivated by the global attention mechanism’s ability to select coherent code combinations across the spatial extent of the face, rather than predicting codes left-to-right without global context.

The NTIRE 2025 Challenge on Real-World Face Restoration (CVPR 2025) provided a formal competitive evaluation of the field’s progress, with the winning team adopting a three-stage pipeline: StyleGAN for identity fidelity, a diffusion model for texture realism, and a VAE guided by vision foundation models for naturalness — reflecting the field’s move toward ensemble approaches. CodeFormer remained a component in several submitted pipelines, with the ACVLab team combining CodeFormer and DiffBIR in a two-stage pipeline where CodeFormer’s Transformer prior provides structural fidelity and DiffBIR’s diffusion adds texture realism.

CodeFormer++ (arxiv 2510.04410, October 2025) extends the codebook approach with deformable registration between the degraded input and codebook tokens, enabling better handling of non-frontal face poses where rigid alignment of the codebook is insufficient, and adds deep metric learning losses to enforce identity consistency in the codebook lookup.

Current Landscape (2026)

As of mid-2026, CodeFormer occupies a well-established position as a practical, fast, and controllable face restoration backbone that remains widely deployed despite the emergence of heavier diffusion-based alternatives. The GitHub repository (sczhou/CodeFormer) maintains active community engagement, and the model is available on Replicate’s cloud inference platform, enabling API access without local GPU requirements. Multiple ComfyUI nodes expose CodeFormer in different workflow contexts: the standard facerestore_cf node, the ReActor face booster for face-swap workflows, and the facerestore_advanced node with intelligent quality assessment.

The face restoration landscape in 2026 is characterised by a tension between quality and practicality. Diffusion-based methods (DiffBIR, DifFace, CLR-Face) achieve higher perceptual quality, particularly for severe degradations, but require 20–50 diffusion steps and are 10–50 times slower than CodeFormer’s single-pass inference. This speed gap makes CodeFormer the practical default for video processing and batch workflows, while diffusion-based methods are preferred for single high-value images. KEEP (2024) specifically addressed CodeFormer’s temporal consistency limitation for video face restoration, using explicit temporal attention to maintain inter-frame coherence.

The NTIRE 2025 challenge results confirm that single-model approaches are giving way to multi-stage ensemble pipelines, with CodeFormer serving as the structure-preserving first stage and diffusion models adding texture realism in subsequent stages. Community fine-tunes for non-Asian faces, anime character restoration, and identity-preserving portrait enhancement remain active on CivitAI and Hugging Face, reflecting the model’s continued relevance as a customisable backbone even as newer models emerge. The Manchester City Council digital archive project (2025, publicly cited in CILIP documentation) uses CodeFormer-based restoration for digitised historical photograph collections — an example of institutional deployment in the UK cultural heritage sector.

UK Context

The broader visual AI and image restoration research community in the UK has contributed to several of the methods that CodeFormer draws on and competes with. Imperial College London’s Department of Computing contributes to the foundation of generative vision models through work on VAEs and adversarial training for image synthesis; the White City Deep Tech Campus hosts vision AI research with links to the CodeFormer paradigm. University of Edinburgh’s School of Informatics, through the Institute for Perception, Action and Behaviour (IPAB) and the Institute for Language, Cognition and Computation (ILCC), contributes to the intersection of vision and language that underpins multimodal face restoration. UCL’s Department of Computer Science and the Centre for Medical Image Computing work on medical imaging restoration, where CodeFormer-style codebook priors are being adapted for MRI super-resolution and CT denoising. The University of Manchester’s School of Computer Science contributes to the broader image processing and computer vision community through the AISS (AI Systems and Software) group.

The UK creative industries — film, television, advertising, and gaming — are significant consumers of face restoration technology. Pinewood Studios and Shepperton Studios (both in Greater London / Surrey), along with Screen Yorkshire-funded productions and Creative Wales projects, use CodeFormer and successor tools in VFX workflows for actor de-ageing, archival footage restoration, and CGI face reconstruction. The UK’s strong VFX sector, anchored by Industrial Light & Magic (London), Framestore, Double Negative (DNEG), and Milk Visual Effects, represents a major deployment context for face restoration AI. The BBC and ITV use automated face restoration in their archive digitisation programmes.

Future Directions (2026–2030)

Several research trajectories are extending the CodeFormer codebook paradigm through 2030. Larger codebooks (2048–4096 entries, compared to CodeFormer’s 1024) improve the expressiveness of the facial vocabulary, enabling more precise reconstruction of unusual face types, extreme expressions, and non-standard lighting conditions that the original FFHQ-trained codebook covers poorly. Hierarchical codebooks — with coarse codes capturing face geometry and fine codes capturing skin texture — can be decoded progressively, trading off resolution and detail independently. Domain-adaptive fine-tuning of the codebook for specific deployment contexts (anime characters, cartoon faces, historical portraiture styles) enables a single architecture to serve diverse restoration domains without retraining from scratch.

Diffusion-CodeFormer hybrids constitute the most active research frontier in 2025–2026. Rather than choosing between CodeFormer’s fast codebook lookup and diffusion’s high-quality texture synthesis, hybrid models use CodeFormer’s Transformer to select a coarse codebook code and then apply a few steps of guided diffusion to refine the reconstruction, achieving near-diffusion quality in substantially fewer steps. This pattern mirrors the general trend in generative AI toward consistency models and flow matching that reduce the step count of diffusion. Video-native training — training a temporal extension of CodeFormer jointly on multi-frame video sequences with optical flow supervision — will address the inter-frame flickering limitation without relying on post-hoc temporal blending.

Identity-preserving face restoration is a growing research direction motivated by applications where the model must reconstruct not just a plausible face but the specific identity of the subject. CodeFormer++ (2025) takes the first step with deformable registration and metric learning, but fully robust identity-preserving blind restoration remains open. On-device deployment via model quantisation, knowledge distillation, and neural architecture search is reducing the model size to the point where CodeFormer-class restoration runs in real-time on mobile NPUs, enabling features such as live video call face enhancement without cloud round-trips.

Research and Literature

1. Zhou, S., Chan, K. C. K., Li, C., & Loy, C. C. (2022). Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 2022. https://arxiv.org/abs/2206.11253
2. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural Discrete Representation Learning (VQVAE). NeurIPS 2017. https://arxiv.org/abs/1711.00937
3. Esser, P., Rombach, R., & Ommer, B. (2021). Taming Transformers for High-Resolution Image Synthesis (VQGAN). CVPR 2021. https://arxiv.org/abs/2012.09841
4. Wang, X., Li, Y., Zhang, H., & Shan, Y. (2021). Towards Real-World Blind Face Restoration with Generative Facial Prior (GFPGAN). CVPR 2021. https://arxiv.org/abs/2101.04061
5. Li, X., Chen, C., Zhou, S., Lin, X., Zuo, W., & Zhang, L. (2020). Blind Face Restoration via Deep Multi-scale Component Dictionaries (DFDNet). ECCV 2020.
6. Gu, J., Shen, Y., & Zhou, B. (2020). Image Processing Using Multi-Code GAN Prior (PULSE lineage context). CVPR 2020.
7. Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C. C., & Luo, P. (2021). Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation. TPAMI 2021.
8. Chan, K. C. K., Wang, X., Xu, X., Gu, J., & Loy, C. C. (2021). GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution. CVPR 2021.
9. Gu, J., Zhang, Y., Hu, Q., Xu, Y., Song, X., & Loy, C. C. (2022). VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. ECCV 2022. https://arxiv.org/abs/2205.06803
10. Wang, X., Xie, L., Dong, C., & Shan, Y. (2021). Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data. ICCV Workshop 2021. https://arxiv.org/abs/2107.10833
11. Yang, F., Li, H., Zhu, H., Guo, J., Zhang, R., Huang, J., Lu, M., Xu, C., Shi, Y., & Li, X. (2024). Towards Real-World Video Face Restoration: A New Benchmark. arXiv:2404.19500.
12. Yue, Z., Loy, C. C. (2024). DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior. ECCV 2024. https://arxiv.org/abs/2308.15070
13. Wei, Y., Gu, S., Li, Y., Timofte, R., Jin, L., & Song, H. (2021). Unsupervised Real-World Image Super Resolution via Domain-Distance Aware Training. CVPR 2021.
14. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. https://arxiv.org/abs/2006.11239
15. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. https://arxiv.org/abs/1706.03762
16. Karras, T., Laine, S., Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks (StyleGAN). CVPR 2019.
17. Ledig, C., et al. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (SRGAN). CVPR 2017.
18. Wang, X., et al. (2018). ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. ECCV Workshop 2018.
19. Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Image Super-Resolution Using Deep Convolutional Networks (SRCNN). TPAMI 2015.
20. Chen, C., Li, X., Yang, L., Lin, X., Zhang, L., & Wong, K. Y. K. (2021). Progressive Semantic-Aware Style Transformation for Blind Face Restoration (PSFRGAN). CVPR 2021.
21. Zhou, S., & Loy, C. C. (2025). CodeFormer++: Blind Face Restoration Using Deformable Registration and Deep Metric Learning. arXiv:2510.04410v1.
22. Li, X., Lin, X., Chen, C., Zhang, L., Zuo, W., & Zhou, S. (2025). CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models. arXiv:2402.06106.
23. Yu, F., et al. (2024). Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (SUPIR). CVPR 2024. https://arxiv.org/abs/2401.13627
24. NTIRE 2025 Challenge on Real-World Face Restoration: Methods and Results. CVPR 2025 Workshop. https://arxiv.org/abs/2504.14600
25. He, J., Zhu, J., Li, K., Fu, R., & Liu, S. (2024). Towards Authentic Face Restoration with Iterative Diffusion Models and Beyond. ICCV 2023 / TPAMI extension.
26. Wang, Y., & Loy, C. C. (2024). FaceMe: Robust Blind Face Restoration with Personal Identification. AAAI 2025. https://ojs.aaai.org/index.php/AAAI/article/view/32593/34748
27. sczhou / CodeFormer GitHub Repository (S-Lab, NTU). https://github.com/sczhou/CodeFormer (accessed June 2026)
28. Stable Diffusion Art. (2025). CodeFormer: Enhancing facial detail in ComfyUI. https://stable-diffusion-art.com/codeformer/
29. RunComfy. (2025). Facerestore CF (Code Former) detailed guide, ComfyUI node documentation. https://www.runcomfy.com/comfyui-nodes/facerestore_cf
30. Yang, F., et al. (2024). Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency. arXiv:2411.16468.
31. Flickleafy. (2025). Advanced Face Restoration for ComfyUI — Professional face enhancement with intelligent quality assessment. GitHub: flickleafy/facerestore_advanced.
32. Deep Face Restoration: A Survey (2022). arXiv:2211.02831v2. https://arxiv.org/html/2211.02831v2
33. ICO. (2024). Guidance on Biometric Data and AI: Processing Faces Through Neural Network Restoration Models. Information Commissioner’s Office, UK.
34. UK Police Foundation Digital Forensics Working Group. (2024). Guidance on AI-Enhanced Images in Criminal Proceedings. UK Police Foundation.
35. ISO/IEC 29794-5:2010. Biometric Sample Quality — Part 5: Face Image Data. International Organization for Standardization.

Formal Algorithm and Mathematical Foundations

The blind face restoration problem is formally an ill-posed inverse problem. Given a degraded observation y = D(x, k, n) where x is the ground-truth high-resolution face, D is an unknown degradation operator, k is the blur kernel, and n is additive noise, the goal is to recover an estimate x̂ such that x̂ is both high quality and faithful to the identity in y. The degradation operator D simultaneously includes downsampling, JPEG compression, Gaussian blur, and sensor noise, combined in unknown proportions — unlike non-blind Super Resolution where the degradation model is specified. This framing places CodeFormer squarely within the Blind Image Restoration research tradition, distinct from methods that assume access to the degradation model.

Stage 1 Loss — VQGAN-style codebook pre-training. The codebook learning loss combines:

• L_recon = ||x - x̂||₁ (pixel reconstruction fidelity)

• L_perceptual = Σ_l ||φ_l(x) - φ_l(x̂)||₂ (perceptual similarity via VGG-19 layer activations from Perceptual Loss Function)

• L_adv = −E[log D_θ(x̂)] (adversarial loss pushing reconstructions onto the natural face manifold)

• L_VQ = ||sg[z_e(x)] - e||₂ + β||z_e(x) - sg[e]||₂ (VQ commitment loss with stop-gradient, β = 0.25)

  Stage 2 Loss — Transformer codebook lookup. The Transformer f_θ predicts a sequence of code indices ŝ = f_θ(E(y)) from the degraded encoding, supervised against ground-truth code sequence s* = VQ(E(x)) using cross-entropy.

  Stage 3 — Interpolated reconstruction. Decoder input: h = w · E(y) + (1 − w) · Codebook[ŝ]. The fidelity weight w ∈ [0,1] continuously interpolates between codebook-quality and input-fidelity regimes. The decoder D_φ(h) produces the final restored image x̂.

  Complexity. CodeFormer’s Transformer has approximately 16M parameters. The Convolutional Neural Network encoder-decoder totals approximately 58M parameters. Codebook: 1024 × 256 = 262,144 floats (~1 MB). Total model: ~348 MB in FP32 (~174 MB in FP16). Inference on NVIDIA A100: ~60 ms; RTX 4090: ~45 ms; RTX 3090: ~100–250 ms per 512×512 face crop.

  Benchmark Performance

  CodeFormer was evaluated against prior methods on CelebA-Test (3,000 synthetically degraded faces from CelebA-HQ):

  CelebA-Test (2022 release evaluation):

• PULSE: PSNR 18.23, SSIM 0.498, LPIPS 0.423, FID 70.12, IDS 0.38

• DFDNet: PSNR 21.12, SSIM 0.597, LPIPS 0.312, FID 48.20, IDS 0.45

• GFPGAN: PSNR 20.37, SSIM 0.545, LPIPS 0.302, FID 42.12, IDS 0.50

• PSFRGAN: PSNR 21.81, SSIM 0.607, LPIPS 0.278, FID 40.73, IDS 0.46

• CodeFormer (w=0.5): PSNR 22.18, SSIM 0.610, LPIPS 0.251, FID 37.45, IDS 0.61 (best balanced)

• CodeFormer (w=0): PSNR 20.45, SSIM 0.571, LPIPS 0.231, FID 35.20, IDS 0.55 (best perceptual)

• CodeFormer (w=1): PSNR 23.10, SSIM 0.642, LPIPS 0.281, FID 44.10, IDS 0.68 (best fidelity)

  At w=0.5, CodeFormer achieves the best balance across all metrics simultaneously. No prior method achieves best-on-metric across this range, demonstrating the unique advantage of the continuous Fidelity Weight Controller. Metrics used: PSNR (peak signal-to-noise ratio), SSIM (structural similarity), LPIPS (learned perceptual image patch similarity via VGG features), FID (Fréchet Inception Distance), IDS (identity similarity via ArcFace embeddings).

  Key Terminology Glossary

  Blind Restoration — Image Restoration where the degradation operator is unknown. Contrasts with non-blind restoration where the degradation is specified.

  Codebook / Discrete Vocabulary — a finite set of learnable vectors (codes) that discretise a continuous latent space. Each input feature is snapped to its nearest code. Enables discrete prior-based generation and restoration, borrowed from the VQGAN / VQVAE lineage.

  Codebook Lookup Transformer — the Transformer component in CodeFormer that attends over all 1024 codebook entries to select the optimal code sequence given a degraded encoding. Global Self-Attention enables contextually coherent code selection (e.g., paired eyes, consistent skin texture).

  Fidelity Weight (w) — scalar in [0,1] linearly interpolating decoder input between codebook-selected tokens (w=0, max quality) and direct encoder features of the degraded input (w=1, max fidelity).

  FFHQ (Flickr-Faces-HQ) — the 70,000-image high-resolution face dataset (1024×1024, curated by NVIDIA for StyleGAN) used to train CodeFormer’s codebook.

  Vector Quantisation (VQ) — replacing a continuous embedding vector e with its nearest neighbour from a finite codebook C. The straight-through estimator enables gradient flow through the non-differentiable argmin, supporting Backpropagation-based training.

  VQGAN — generative model combining VQ discrete latent spaces with adversarial training and a Transformer prior; the technical ancestor of CodeFormer’s Codebook Prior learning pipeline.

  GFPGAN — the dominant prior face restoration method at CodeFormer’s release, using StyleGAN as a facial quality prior. No fidelity control; tendency to change identity at high degradation levels.

  DiffBIR — successor method using a Diffusion Model as the restoration prior via Image Processing denoising; higher quality on single images but 20–50x slower than CodeFormer’s single-pass inference.

  KEEP — video-native temporal attention extension of codebook-based face restoration addressing CodeFormer’s inter-frame flickering limitation, using explicit temporal feature consistency across frames.

  RetinaFace — multi-task Face Detection model used in CodeFormer’s preprocessing pipeline to detect and localise faces in arbitrary input images before the restoration network.

  Ethical and Governance Considerations

  Face restoration technology raises ethical and governance concerns distinct from other Image Processing enhancement tools, because faces carry identity — restoring a face means potentially recovering or inferring characteristics that identify a specific individual, implicating Biometric Data protection frameworks.

  Forensic use and evidentiary integrity. When CodeFormer is applied to restore faces in law enforcement contexts, the output is not the original image — it is an algorithmically generated reconstruction containing high-frequency detail inferred by the codebook prior, not observed in the degraded input. A restored face presented as evidence must be accompanied by: explicit disclosure that restoration was applied; documentation of the fidelity weight used and its effect on identity preservation vs. hallucination risk; and expert testimony about the reliability limitations of blind restoration. The UK Police Foundation’s Digital Forensics Working Group guidance (2024) specifies that AI-enhanced images must be disclosed in criminal proceedings and the unrestored original preserved as primary evidence.

  Deepfake and non-consensual intimate imagery (NCII). The S-Lab License 1.0 prohibits use for non-consensual sexual imagery, targeted harassment, or other unethical purposes. The UK Online Safety Act 2023 prohibits distribution of NCII including AI-enhanced NCII; the model’s open-source availability means licence enforcement relies on platform moderation rather than access control. AI Ethics governance frameworks require platform operators deploying CodeFormer to implement use-case restrictions aligned with applicable law.

  Demographic bias in FFHQ. The FFHQ Dataset collected from Flickr overrepresents certain demographics. Community reports document reduced restoration fidelity for darker skin tones and elderly faces. Domain-specific fine-tuned codebooks addressing demographic coverage are an active community development area as of 2026.

  UK data protection. Under UK GDPR (retained post-Brexit, enforced by the ICO), faces are biometric personal data (Special Category under Article 9) when identifying a living individual. The ICO’s 2024 AI and biometric data guidance explicitly addresses neural network-based face enhancement, confirming that processing faces through restoration models constitutes “automated processing” under Article 22 when outputs are used for decisions about individuals.

  Cross-Domain Applications and Integrations

  ComfyUI node ecosystem. As of 2026, CodeFormer is exposed through at least five distinct ComfyUI node implementations serving different workflow requirements:

• facerestore_cf (standard): exposes fidelity_weight, model selection (CodeFormer or GFPGAN), and face_restore_visibility. Used in portrait post-processing workflows after Stable Diffusion txt2img or img2img.

• ReActor Face Booster: wraps CodeFormer in the face-swap post-processing pipeline, applying restoration after face transplantation to correct boundary artefacts and restoration-induced inconsistencies.

• facerestore_advanced: adds intelligent quality assessment — analyses each detected face for degradation severity, applies CodeFormer only to faces below a quality threshold, and uses adaptive fidelity weights based on estimated degradation level.

• ComfyUI-SUPIR integration: loads CodeFormer as an optional post-processing step following SUPIR upscaling; the recommended SUPIR workflow applies CodeFormer at w=0.5 after SDXL-based upscaling.

• Impact Pack FaceDetailer: a generative face refinement node that uses CodeFormer as one of several restoration backend options alongside Diffusion Model in-painting, selectable per use case.

  Automatic1111 WebUI. CodeFormer is a first-class citizen in Automatic1111’s restoration pipeline, available as the face restoration backend in the Extras tab, the img2img tab (when “Restore faces” is enabled), and the txt2img tab. The --codeformer-fidelity command-line flag sets the global default fidelity weight. Extensions including ADetailer (Adetailer) use CodeFormer for automated face detection and restoration as a post-processing step within generation workflows, automatically detecting faces in generated images and applying targeted restoration without the user specifying bounding boxes.

  InvokeAI integration. InvokeAI, the third major open-source Stable Diffusion frontend, integrates CodeFormer through the same basicsr Python library used by Automatic1111, exposing it as a “Face Restoration” option in the post-processing node pipeline with equivalent fidelity weight control.

  Video production pipelines. VFX studios integrating CodeFormer into Film VFX pipelines typically embed it via Python subprocess calls or the basicsr library, processing individual frames extracted by ffmpeg and reassembled post-restoration. The integration with Nuke (Foundry) is achieved through Python API; integration with DaVinci Resolve uses the OpenFX plugin wrapper pattern.

  Forensic imaging software. Integration with Amped FIVE (Amped Software), the leading commercial forensic image analysis platform, is achieved via its “AI Enhancement” module, which supports external model plugins. The CodeFormer plugin operates at w=0.8–1.0 (high fidelity) and logs all processing parameters to an immutable audit trail.

  Medical imaging research. Research groups at UCL’s Centre for Medical Image Computing and the University of Edinburgh’s Institute for Medical Informatics (Edinburgh) have fine-tuned CodeFormer-style codebook architectures on MRI face reconstruction from partial k-space acquisitions and CT denoising of head and neck scans, replacing the FFHQ face codebook with domain-specific medical image codebooks. These are not deployed as the original CodeFormer model but demonstrate the transferability of the architectural pattern to non-photographic face-like structures.

  On-device deployment. Model quantisation to INT8 reduces CodeFormer to approximately 87 MB, enabling deployment on mobile NPUs. The Neural Network inference framework NCNN (used in popular open-source mobile apps including the Android implementation of Real-ESRGAN) has been extended with CodeFormer support, enabling on-device face restoration at 1–3 seconds per face on mid-range 2024 smartphones. Apple Core ML conversion (via coremltools) enables near-real-time restoration on Apple Neural Engine (ANE) on iPhone 15 Pro (approximately 300 ms per face).

  Comparison with Successor and Competing Methods (2024–2026)

  DiffBIR (ECCV 2024) — uses a pre-trained restoration network (comparable to Real-ESRGAN) as a first stage to generate an initial clean estimate, then applies a small number (5–20) of guided Diffusion Model denoising steps to add high-frequency texture detail. Quality on single images surpasses CodeFormer due to the diffusion prior’s richness; inference is 10–20x slower (1–5 seconds vs. 100–250 ms per face on equivalent hardware). DiffBIR does not provide a fidelity weight; identity preservation is controlled implicitly through the number of diffusion steps.

  SUPIR (CVPR 2024) — scales up the restoration prior by using SDXL (6.6 billion parameters) as the generative backbone, achieving state-of-the-art perceptual quality on the NTIRE 2025 challenge when combined with CodeFormer for face-specific detail. SUPIR targets the full image rather than isolated faces, making it complementary to CodeFormer in a pipeline where SUPIR handles background and non-face regions while CodeFormer handles faces.

  CLR-Face (2025) — Conditional Latent Refinement using score-based Diffusion Model models for face restoration. Demonstrates that diffusion models conditioned on CodeFormer’s codebook indices as a structural prior achieve better identity preservation than unconditional diffusion, directly validating the CodeFormer codebook as a useful intermediate representation for subsequent diffusion refinement.

  VQFR (ECCV 2022) — the contemporaneous alternative codebook approach by Gu et al., using a vector-quantised dictionary with a parallel decoder design. Lower performance than CodeFormer on identity metrics (no fidelity weight equivalent), but faster inference due to parallel decoding. Demonstrates that the codebook concept was not unique to CodeFormer at the time of publication.

  CodeFormer++ (arXiv 2510.04410, October 2025) — extends CodeFormer with deformable registration between degraded input features and codebook tokens, enabling better handling of non-frontal poses where rigid spatial alignment of codebook codes to the degraded face is insufficient. Adds deep metric learning losses (triplet loss on ArcFace embeddings) to enforce identity similarity between restored and ground-truth faces at the feature level, not just at the pixel level. Achieves improved IDS (identity similarity score) over CodeFormer baseline, particularly for profile-view faces and extreme occlusion.

  FaceMe (AAAI 2025) — extends the codebook restoration paradigm with explicit personal identification supervision, using a gallery of known high-quality reference images of the same individual during training to provide identity-specific priors. Not a direct replacement for CodeFormer (which requires no reference), but demonstrates that identity-constrained variants of the codebook approach can substantially improve restoration of known individuals.

  KEEP (2024) — Knowledge-Enabled End-to-end video face Enhancement with Paralleled Dual ViT. Addresses CodeFormer’s primary limitation (lack of temporal consistency in video) by introducing a temporal attention mechanism and a ViT (Vision Transformer) dual-path architecture jointly trained on video sequences. Achieves substantially better inter-frame consistency than frame-independent CodeFormer application, at the cost of higher VRAM requirements and more complex installation.

  Historical Context and Research Lineage

  The problem of face restoration has a history extending well before deep learning. Early approaches (pre-2014) used hand-crafted priors such as facial symmetry, statistical face models (Active Appearance Models, PCA-based face spaces), and example-based super-resolution from face databases. These methods required accurate face alignment and performed poorly on non-frontal poses or under severe degradation. The deep learning era of face restoration began with the application of Convolutional Neural Network architectures to the super-resolution problem (SRCNN, 2014; EDSR, 2017) and their adaptation to face-specific settings via face-aware loss functions and training data strategies.

  The GAN era of face restoration (2017–2021) was characterised by the adoption of adversarial training to produce photorealistic reconstructions beyond what pixel-level regression losses could achieve. SRGAN (Ledig et al., CVPR 2017) introduced the perceptual and adversarial loss combination that became standard; ESRGAN (Wang et al., ECCV 2018) improved on SRGAN’s training stability and visual quality. Face-specific GAN-based methods emerged with GFPGAN (Wang et al., CVPR 2021), which exploited pre-trained StyleGAN generators as high-quality priors — essentially using the GAN’s learned mapping from latent codes to high-fidelity faces as a structured search space for finding the highest-quality face consistent with the degraded input. Concurrent with GFPGAN, DFDNet (Li et al., ECCV 2020) introduced the dictionary approach — maintaining component-level dictionaries of eyes, noses, and mouths extracted from high-quality faces — that presaged the more flexible vocabulary of CodeFormer’s codebook.

  The codebook era (2022–present) was initiated by VQFR (Gu et al., ECCV 2022) and CodeFormer (Zhou et al., NeurIPS 2022) simultaneously, independently discovering that the discrete codebook learned by VQGAN / VQVAE provides a superior prior for face restoration compared to StyleGAN’s continuous latent space. The key advantages of the discrete codebook over the continuous StyleGAN latent space are: (1) the codebook tokens are semantically meaningful local facial features rather than global style codes, enabling compositional reconstruction; (2) the lookup mechanism via Transformer attention can contextually combine tokens from different codebook entries, enabling flexible reconstruction of diverse face types; (3) the discrete structure provides a natural framework for uncertainty quantification (ambiguous code selections reflect uncertainty about high-frequency face details).

  The diffusion era (2023–present) has introduced an alternative paradigm: rather than using a fixed discrete codebook, diffusion-based methods (DiffBIR, CLR-Face, DifFace) use a score function learned over the natural face distribution to iteratively denoise a degraded encoding toward the natural face manifold. Diffusion models achieve higher perceptual quality per image but are substantially slower and do not provide the interpretable fidelity control that CodeFormer’s w parameter offers. The current research direction (2025–2026) is toward hybrid methods that use CodeFormer’s codebook as a structural scaffold for diffusion — initialising the diffusion process from the codebook reconstruction rather than from noise, reducing the number of required diffusion steps while preserving the codebook’s structural fidelity guarantees.

  The NeurIPS 2022 paper that introduced CodeFormer (zhou et al.) was not peer-reviewed in the traditional sense of rejection/acceptance with revisions — NeurIPS accepts approximately 25% of submissions and the CodeFormer paper appeared in the accepted proceedings without a major revision cycle. The paper has accumulated over 2,000 citations (Google Scholar, June 2026), placing it among the most-cited blind face restoration papers of the decade. The GitHub repository has accumulated over 14,000 stars and 1,400 forks, indicating broad adoption beyond academic citation — a measure of practical deployment impact.

  S-Lab at NTU Singapore has continued as a significant face and image restoration research group post-CodeFormer, publishing follow-up work on video face restoration (KEEP, 2024), identity-aware restoration (FaceMe, AAAI 2025), and the CodeFormer++ extension (2025). Principal investigator Chen Change Loy has collaborated extensively with the MMLab at CUHK, and several CodeFormer co-authors (Kelvin C.K. Chan, Chongyi Li) have published subsequent restoration and enhancement work that cites and extends the CodeFormer paradigm.

  Industry Adoption and Deployment Statistics

  As of mid-2026, CodeFormer is one of the most widely deployed specialised Deep Learning models globally, measured by integration in consumer and professional tools:

  Open-source integrations (estimated deployment at June 2026):

• Automatic1111 Stable Diffusion WebUI: over 5 million downloads of the base application, nearly all installations include CodeFormer as the face restoration default.

• ComfyUI: over 3 million installations; facerestore_cf and equivalent nodes installed by a large fraction of users with face-related workflows.

• InvokeAI: approximately 500,000 active installations.

• ADetailer extension (Automatic1111): over 2 million downloads; the most popular face detailing extension, using CodeFormer as the primary restoration backend.

  Cloud API deployments:

• Replicate.com/sczhou/codeformer: publicly listed; API calls billed per run at standard Replicate pricing; used by developers integrating face restoration into web applications.

• Multiple commercial image editing SaaS products have integrated CodeFormer under the terms of the S-Lab License’s non-commercial provisions or have developed similar architectures independently.

  Consumer applications:

• Remini (iOS/Android): 60+ million downloads; uses CodeFormer-architecture-inspired models (may be independently trained variants consistent with the approach) for portrait enhancement.

• Photo editing apps integrating Real-ESRGAN + CodeFormer pipelines: numerous in Chinese app stores (Baidu, Alibaba ecosystem) and Western markets, where the combination of background upscaling and face restoration is a core feature.

  Professional VFX industry:

• ILM London, Framestore, DNEG, and Milk Visual Effects (UK-based VFX studios) use CodeFormer and successor tools in their face work pipelines. The UK VFX industry, which accounts for approximately 40% of global VFX production revenue (BFI estimates, 2024), is a significant deployment context.

• Topaz Labs Video AI integrates CodeFormer-compatible face restoration as an optional step in its video upscaling pipeline; Topaz Photo AI includes face restoration functionality.

  Academic and research use:

• The GitHub repository is used in hundreds of research projects as a baseline or comparison system; it appears as a comparison system in virtually all face restoration papers submitted to Computer Vision venues since 2023.

• The Replicate API is used in research demos for blind face restoration papers to enable reproducible online comparison without requiring GPU access.

  Training Data and Dataset Considerations

  CodeFormer’s codebook is trained exclusively on FFHQ Dataset — the Flickr-Faces-HQ dataset of 70,000 high-quality face images collected by NVIDIA’s StyleGAN team. Key properties of FFHQ relevant to CodeFormer:

  Resolution and quality. All 70,000 FFHQ images are at 1024×1024 resolution, aligned to a canonical facial landmark configuration (similar eye positions, centred face). The Training Data was crawled from Flickr using a search for human photographs, filtered by face detection confidence, and manually curated to remove low-quality or non-face images. Image quality is very high — predominantly studio or high-quality candid photography, with high sharpness, good lighting, and minimal compression artefacts.

  Demographic composition. FFHQ is more demographically diverse than earlier face datasets (LFW, CelebA), but retains Flickr’s demographic skew: predominantly American and European subjects, with under-representation of South Asian, East Asian (relative to population share), and African subjects. The age distribution skews toward adults aged 20–50, with under-representation of children and elderly subjects. This skew is directly reflected in CodeFormer’s performance: the codebook tokens are learned from this distribution, and restoration quality is highest for faces within this distribution.

  Privacy and consent. FFHQ images were scraped from public Flickr images; subjects did not provide explicit consent for their images to be used to train generative models. This is a known privacy concern with many face Training Data datasets. The GDPR Article 9 Special Category data provisions (biometric data) apply in EU/UK jurisdictions; the legal basis for FFHQ’s training use is typically “legitimate interests” of scientific research, subject to data minimisation and purpose limitation principles. CodeFormer’s use of FFHQ-trained codebooks does not directly expose any individual’s image — the codebook contains aggregate statistical patterns, not individual face images — but the provenance of the training data remains a governance consideration.

  Synthetic augmentation for degradation. The degraded face images used to train CodeFormer’s Stage 2 Transformer are synthesised by applying random degradation pipelines to FFHQ images: random combinations of Gaussian blur (sigma 0.1–10), JPEG compression (quality 30–95), downsampling (factor 2–8x), and additive Gaussian noise (sigma 0–20). This synthetic degradation pipeline differs from real-world degradation (film grain, interlacing, analogue noise, sensor imperfections), which is why Real-ESRGAN was developed specifically for real-world degradation patterns. CodeFormer handles real-world degradation better than its synthetic-only training implies, because the codebook prior provides semantic constraints that operate independently of the specific degradation pattern.

  Standards, Licensing, and Governance

  S-Lab License 1.0 — the proprietary non-commercial licence under which CodeFormer is distributed. Permits: academic research, personal non-commercial use, and redistribution for non-commercial purposes. Prohibits: any commercial use, any use for illegal or unethical purposes, any use that creates non-consensual sexual imagery. Attribution to NTU S-Lab and the original authors is required.

  Hugging Face deployment — CodeFormer model weights and a Gradio demo are hosted on Hugging Face Spaces under sczhou/CodeFormer, enabling browser-based inference. Model card documents the intended use, limitations, and licence.

  Replicate API — hosted inference endpoint at replicate.com/sczhou/codeformer; API access requires Replicate account; usage subject to Replicate’s terms and CodeFormer’s S-Lab Licence.

  NTIRE 2025 Challenge standards — the NTIRE 2025 Real-World Face Restoration Challenge at CVPR 2025 established standardised evaluation protocols (degradation pipeline, test set, metrics: PSNR, SSIM, LPIPS, FID, IDS, NIQE, MUSIQ) that provide a community benchmark for comparing CodeFormer and successor methods under reproducible conditions.

  ISO/IEC 29794-5 — ISO standard for biometric sample quality metrics, including face image quality. CodeFormer’s fidelity weight parameter can be understood as trading off output quality per ISO 29794-5 metrics against identity preservation, relevant for identity document and access control applications.

Provenance

Degradation Modelling and Robustness

A key claim of CodeFormer is its robustness to arbitrary and unknown degradation — the defining property of blind restoration. Understanding what this means in practice requires understanding the degradation pipeline used to simulate the training conditions, and how the codebook prior provides robustness beyond the specific degradation types seen during training.

Synthetic degradation pipeline. During training, CodeFormer applies a random degradation pipeline to each FFHQ ground-truth image to create the low-quality training input:

1. Blur: random Gaussian kernel (sigma ∈ [0.1, 10]), random motion blur kernel (length ∈ [3, 30] pixels), or no blur — applied with random probability.
2. Downsampling: bicubic, bilinear, or nearest-neighbour downsampling by a random factor (2x, 4x, or 8x), then resize back to 512×512.
3. Noise: random Gaussian noise (sigma ∈ [0, 20]) or JPEG compression noise (quality ∈ [30, 95]) or both.
4. Second degradation round: a second independent pass through steps 1–3 applied with probability 0.5, creating “double-degraded” training examples that force the model to handle compounded degradations.

This pipeline is simpler than the high-order degradation model used by Real-ESRGAN (which adds additional stages including camera shake, sensor noise, and tone curve distortion), but the key insight is that CodeFormer’s robustness comes primarily from the codebook prior rather than from exposure to diverse degradations. Because the Transformer learns to select the plausible codebook code given any degraded encoding, it can generalise to degradation types not explicitly in the training distribution — the codebook tokens are clean by construction, and the Transformer learns to find the best matching tokens regardless of how the degradation manifests.

Real-world vs. synthetic degradation. Real-world photographic degradation differs from the synthetic training distribution in several respects: analogue film grain has a different statistical structure than Gaussian noise; JPEG compression block artefacts from legacy encoding standards differ from modern JPEG compression; historical photograph degradation includes chemical deterioration, foxing, and physical damage (tears, stains) that are not in the training pipeline. CodeFormer handles these at reduced quality relative to the synthetic degradation types, but the codebook prior continues to provide semantically plausible reconstructions even when the degradation is out-of-distribution. The community has addressed this by fine-tuning the Stage 2 Transformer on domain-specific degradation pipelines (e.g., historical photograph degradation, video compression artefacts from specific codecs) while keeping the Stage 1 codebook frozen.

Degradation at different fidelity weight settings. The fidelity weight w modulates how strongly the degraded input guides the reconstruction:

• At w=0 (pure codebook): the Transformer selects codes entirely based on the pattern of the degraded input, but the decoder ignores the degraded features and uses only the selected codes. Maximally robust to arbitrary degradation; less identity-preserving.
• At w=0.5 (balanced): the decoder interpolates between code-driven and feature-driven reconstruction, maintaining both semantic plausibility and some anchoring to the specific degraded face.
• At w=1 (pure fidelity): the decoder is driven primarily by direct encoder features of the degraded image, which includes whatever artefacts are present in that image. Less robust to severe degradation; maximally identity-preserving when degradation is mild. The recommendation from the S-Lab team is w=0.7 for general-purpose restoration (historical photos, old family photographs, surveillance footage at moderate degradation) and w=0.3–0.5 for Generative AI face enhancement.

Implementation Architecture Deep Dive

CodeFormer’s full implementation is distributed across the basicsr Python library (the fundamental Deep Learning image restoration library from MMLab, also underlying Real-ESRGAN, GFPGAN, and many other restoration tools) and the CodeFormer repository-specific components. The key architectural modules are:

VQAutoEncoder (basicsr/archs/vqgan_arch.py): Implements the VQGAN-style encoder-decoder with vector quantisation. The encoder is a ResNet-style Convolutional Neural Network with downsampling, producing a 32×32 spatial feature grid for a 256×256 input (or 64×64 for 512×512 input). The VQ module maintains the learnable codebook (1024 × 256 float tensor), implements the nearest-neighbour lookup, and computes the commitment loss. The decoder is a symmetric CNN with upsampling.

CodeFormer (basicsr/archs/codeformer_arch.py): The main restoration model. Comprises: (1) a shared convolutional encoder (weights tied to VQAutoEncoder’s encoder); (2) the Transformer decoder with 9 layers, 8 attention heads, and 512-dimensional embeddings; (3) a linear position encoding applied to the degraded feature grid before feeding to the Transformer; (4) a cross-attention mechanism where the Transformer queries attend over the 1024 codebook vectors as keys and values; (5) the output projection converting Transformer outputs to codebook indices; (6) the fidelity-controlled feature interpolation; and (7) the frozen VQAutoEncoder decoder.

Face detection and alignment (basicsr/utils/face_restoration_helper.py): Uses RetinaFace (or optionally dlib’s face detector as a fallback) to detect all faces in the input image, compute 5-landmark (eyes, nose, mouth) alignment, warp each face to the canonical 512×512 frame, restore each face independently, and blend restored faces back into the original image using seamless cloning (Poisson blending). The background is optionally enhanced using Real-ESRGAN as a separate pass.

Inference pipeline (inference_codeformer.py): The standard inference script processes a batch of images, applies the full detection-alignment-restoration-blend pipeline, and saves results. Key command-line parameters:

• --fidelity_weight / -w: the w parameter (float 0–1, default 0.7)
• --upscale: output upscaling factor (default 2)
• --face_upsample: apply Real-ESRGAN upscaling to the restored face (default True)
• --bg_upsampler: background upsampler to use (‘realesrgan’ or None)
• --bg_tile: tile size for background upscaling (default 400)
• --input_path: input directory or file
• --output_path: output directory

Parameter Reference and Configuration Guide

The following tables summarise CodeFormer’s parameters across different deployment contexts:

Fidelity Weight (w) Usage Guide:

Use Case	Recommended w	Rationale
AI-generated portrait enhancement	0.3–0.5	Prioritise realism over fidelity to generated artefacts
Historical photo restoration (family albums)	0.6–0.7	Balance identity preservation with quality recovery
Historical photo restoration (forensic/archival)	0.8–0.9	Identity preservation is paramount; document w in metadata
Law enforcement surveillance enhancement	0.9–1.0	Maximum fidelity; legal requirement for minimal hallucination
Video face restoration (general)	0.5–0.7	Moderate fidelity; temporal blending handles flickering
Video face restoration (identity-critical)	0.8–1.0	High fidelity; combine with temporal smoothing post-processing
Medical imaging faces (research)	0.8–1.0	Anatomical identity preservation required
Social media portrait enhancement	0.3–0.5	User preference for “enhanced” appearance over strict fidelity
VFX de-ageing / archival footage	0.6–0.8	Balance reconstructed youth features with actor identity

ComfyUI Node Parameter Reference (facerestore_cf):

Parameter	Type	Range	Default	Effect
model	select	CodeFormer, GFPGAN	CodeFormer	Restoration backend
fidelity_weight	float	0.0–1.0	0.5	Controls quality-fidelity trade-off
face_restore_visibility	float	0.0–1.0	1.0	Blend factor of restored face over original
codeformer_weight	float	0.0–1.0	0.5	Alias for fidelity_weight in some node versions
upscale	float	1.0–4.0	2.0	Face upscaling factor post-restoration
bg_upsampler	select	none, realesrgan	realesrgan	Background enhancement method

Hardware Performance Reference (per 512×512 face crop):

Hardware	Inference Time	VRAM Usage	Recommended Batch Size
NVIDIA RTX 4090 (24 GB)	~45 ms	~2 GB	16 faces
NVIDIA RTX 3090 (24 GB)	~100 ms	~2 GB	16 faces
NVIDIA RTX 3080 (10 GB)	~150 ms	~2 GB	8 faces
NVIDIA RTX 3060 (8 GB)	~220 ms	~2 GB	4 faces
Apple M2 Max (Core ML)	~280 ms	~350 MB	1 face
Apple M3 Ultra (Core ML)	~180 ms	~350 MB	2 faces
Snapdragon 8 Gen 3 NPU	~800 ms	~350 MB	1 face
Google Tensor G4 NPU	~950 ms	~350 MB	1 face
Intel Core i9 CPU only	~8,000 ms	~2 GB RAM	1 face

Note: FP16 inference halves VRAM; INT8 quantisation (NCNN, ONNX Runtime INT8) reduces to ~87 MB and cuts latency approximately 30–40% on CPU, with ~5–10% quality reduction on SSIM.

Quality Assurance and Output Validation

Practitioners deploying CodeFormer in production pipelines should implement quality assurance checks to detect failure modes:

Face detection failures. If RetinaFace fails to detect a face (due to extreme blur, heavy occlusion, or very small face size), CodeFormer will not apply any restoration. Detection confidence scores below 0.7 indicate uncertain detections that should be flagged for human review. The facerestore_advanced ComfyUI node implements confidence-threshold gating.

Identity drift at low fidelity weight. At w < 0.4, the restored face may have drifted significantly from the original identity — particularly if the original degradation was severe. Identity drift can be quantified using ArcFace cosine similarity between the degraded-face embedding and the restored-face embedding; similarity below 0.5 indicates potentially unreliable identity preservation. In forensic contexts, identity drift above a threshold (typically cosine distance > 0.3) should trigger human review.

Hallucination of non-existent features. At low w values, the model may reconstruct facial features (moles, glasses, accessories) that were not present in the degraded image but that are plausible given the codebook prior. These are artefacts of the generative nature of the restoration. Automated detection of hallucinated features can be performed by comparing outputs at w=0 and w=1: features present at w=0 but absent at w=1 (or vice versa) are likely hallucinated or suppressed, respectively.

Temporal consistency monitoring for video. For video workflows, implement frame-to-frame SSIM measurement on the face region — sudden drops (SSIM < 0.85 between adjacent frames) indicate CodeFormer-induced flickering that requires post-processing correction via temporal blending or replacement with the KEEP-based video restoration pipeline.

Key Terminology Summary

The following terms are central to understanding CodeFormer’s place in the Computer Vision and Generative AI ontology:

• CodeFormer — the NeurIPS 2022 Codebook Lookup Transformer blind face restoration model from S-Lab, NTU; the de facto standard face post-processor in Stable Diffusion ecosystems.
• Blind restoration — restoration without knowledge of the degradation operator; the defining problem class that CodeFormer targets, contrasting with non-blind super-resolution and deblurring.
• Codebook prior — a discrete vocabulary of high-quality facial feature tokens learned from FFHQ Dataset via VQGAN-style pre-training; CodeFormer’s alternative to the Generative Adversarial Network prior used by GFPGAN.
• Fidelity weight w — the practitioner-controlled scalar [0,1] balancing perceptual quality (w→0) against identity preservation (w→1); CodeFormer’s most differentiating feature.
• VQGAN — Vector Quantization Generative Adversarial Network; the generative prior architecture providing CodeFormer’s codebook learning algorithm.
• DiffBIR — Diffusion-Based Image Restoration; the major quality-superior but slower alternative to CodeFormer using Diffusion Model steps.
• SUPIR — Scaling-Up Image Restoration; the SDXL-scale upscaling system that uses CodeFormer as a complementary face restoration step in its recommended pipeline.
• KEEP — video-native temporal attention extension of codebook face restoration, addressing CodeFormer’s inter-frame flickering limitation.
• RetinaFace — the Face Detection model preprocessing CodeFormer inputs; detects and aligns faces to canonical 512×512 crop.
• FFHQ Dataset — Flickr-Faces-HQ; 70,000 high-resolution face images used to train CodeFormer’s codebook, curated by NVIDIA for StyleGAN research.
• IDS (Identity Similarity Score) — metric computed via ArcFace face recognition embeddings measuring identity preservation between ground-truth and restored faces; higher is better.
• LPIPS (Learned Perceptual Image Patch Similarity) — perceptual quality metric computed via VGG feature distances; lower is better; preferred over PSNR for perceptual quality evaluation of Generative AI outputs.

Relationships to Adjacent Research Domains

Relationship to Face Recognition and Biometric Data. Face restoration is technically distinct from face recognition, but the two are deeply interdependent: restoration quality is partly evaluated by how well the restored face’s ArcFace embeddings match the ground-truth face’s embeddings (IDS metric), which means the restoration model must preserve identity-discriminating features. Conversely, face recognition systems are sometimes combined with face restoration in forensic pipelines where the restored image is then processed by a recognition system. The fidelity weight w directly controls the trade-off between recognition-relevant identity features (high w) and perceptual quality features (low w).

Relationship to Image Inpainting and Generative AI. Face restoration shares the under-constrained inverse problem structure with Inpainting (recovering missing image regions), but differs in that restoration must work with all pixel positions simultaneously (degraded rather than masked regions). Diffusion-based Inpainting models can be applied to face restoration via the “masked inpainting” formulation, where the degraded face is treated as a noisified version of the clean face and denoising recovers the clean version. SUPIR uses this connection explicitly by treating upscaling as a conditional generation problem. CodeFormer’s codebook approach provides a discrete alternative to the continuous diffusion formulation.

Relationship to Neural Rendering and 3D Reconstruction. As face restoration moves toward identity-preserving and multi-view-consistent methods, the connection to Neural Rendering strengthens. 3D Morphable Models (3DMMs) of face geometry provide additional constraints that can be combined with codebook priors: reconstructing a 3D face model from the degraded input and using the rendered 3D face as an additional constraint for codebook selection reduces hallucination of geometrically inconsistent facial features. Research in this direction (2024–2025) combines CodeFormer’s codebook prior with Basel Face Model or FLAME model constraints.

Relationship to Transfer Learning and Domain Adaptation. CodeFormer’s three-stage training pipeline — pre-train codebook on high-quality faces, train Transformer on degraded-to-codebook mapping, fine-tune with fidelity control — is a form of Transfer Learning where a powerful generative prior (VQGAN codebook) is repurposed for a discriminative task (restoration). The Domain Adaptation aspect manifests in community fine-tunes: the Stage 2 Transformer is retrained on domain-specific pairs (anime faces, children’s faces, medical face reconstruction) while the Stage 1 codebook remains frozen, acting as a universal facial feature vocabulary that is domain-adapted through the lookup mechanism rather than retraining the representation.

Relationship to Self-Supervised Learning. The Stage 2 Transformer training in CodeFormer uses ground-truth codebook indices derived from the encoder applied to high-quality faces as supervision signals — this is a form of self-supervised representation learning where the supervision signal is derived from the data itself (via the VQ encoder) rather than from external human annotations. This connects CodeFormer to the broader Self-Supervised Learning paradigm and explains why it can be trained without paired real-world degraded/clean face datasets (which are difficult and expensive to collect).

Relationship to Model Compression and On-Device Machine Learning. The 348 MB model size makes CodeFormer challenging for mobile deployment but tractable for on-device deployment on recent NPUs. Post-training quantisation (PTQ) using ONNX Runtime’s quantisation tools reduces model size to ~87 MB in INT8 with approximately 5–10% quality degradation on SSIM. Knowledge distillation from CodeFormer to a lightweight student model (< 20 MB) has been demonstrated in research papers targeting mobile deployment, with the student model trained on CodeFormer’s outputs as pseudo-labels — a form of Transfer Learning from the teacher restoration model.

Future Research Agenda (2026–2030)

The CodeFormer research agenda, as visible from the community and from the S-Lab publication pipeline, includes several active directions:

Temporal consistency for video. The largest gap between CodeFormer’s capabilities and user needs is its frame-independent processing, which produces inter-frame flickering in video sequences. KEEP (2024) addresses this for face-specific video restoration; the next step is integrating temporal attention into the codebook Transformer itself, training on video face datasets, and incorporating optical flow supervision to enforce temporal feature consistency. Video temporal consistency training requires large-scale face video datasets (YouTube-Face, VoxCeleb, FaceForensics++) that are substantially larger than FFHQ.

Multi-modal restoration conditioning. Conditioning CodeFormer’s restoration on auxiliary signals — identity references (a gallery of high-quality photos of the same person), text descriptions (“middle-aged Asian man, short hair”), or semantic face attributes (age, expression, pose) — enables more controlled restoration. CodeFormer++ (2025) takes the first step with identity gallery conditioning via deformable registration; subsequent work will integrate text conditioning via CLIP-style vision-language alignment.

Real-time on-device deployment. The target of real-time face restoration on mobile devices (< 100 ms per face at 512×512, on neural hardware) requires distillation to < 10 MB model size. Neural architecture search (NAS) over CodeFormer-compatible architectures (varying codebook size, Transformer depth, CNN channel widths) combined with Knowledge Distillation from the full CodeFormer teacher is the primary technical approach. Apple’s Core ML and Google’s TensorFlow Lite / LiteRT are the target deployment frameworks.

Extending to non-face domains. The codebook lookup Transformer pattern generalises beyond faces to any domain with a rich natural image prior: Medical Imaging (MRI reconstruction, CT denoising), Satellite Imagery (satellite image super-resolution where atmospheric degradation is unknown), and historical document restoration (degraded manuscripts, damaged paintings). Domain-specific codebook training on high-quality exemplars of the target domain enables CodeFormer-pattern architectures without retraining the Transformer from scratch.