ULMFiT (Universal Language Model Fine-tuning) is a transfer-learning method introduced by Howard and Ruder in 2018 that adapts a language model pre-trained on a large general corpus to downstream NLP tasks such as text classification. It popularised techniques including discriminative learning rates, slanted triangular learning rates, and gradual unfreezing, demonstrating that language-model pre-training transfers effectively to many tasks.
Content
- ULMFiT proceeds in three stages: general-domain language-model pre-training, target-task language-model fine-tuning, and target-task classifier fine-tuning. Its training tricks stabilise transfer to small datasets and presaged the transformer-based pre-training methods that now dominate natural language processing.