BLEU Score

BLEU Score (Bilingual Evaluation Understudy) is an automatic evaluation metric for machine translation and text generation quality that measures the overlap of n-gram sequences between a candidate output and one or more human reference translations, applying a brevity penalty to discourage pathologically short outputs. Scores range from 0 to 1 (or 0 to 100 in percentage form), with higher values indicating closer correspondence to the reference. BLEU correlates moderately with human judgement at the corpus level but is known to be unreliable for single-sentence evaluation and insufficient alone for capturing semantic adequacy.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:hasPart ai:ModifiedNGramPrecision))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:hasPart ai:BrevityPenalty))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:hasPart ai:GeometricMean))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:hasPart ai:NGramClipping))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:hasPart ai:Tokenisation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:hasPart ai:ReferenceTranslation))

Dependency Relationships

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:requires ai:ReferenceTranslation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:requires ai:Tokenisation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:requires ai:ParallelCorpus))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:requires ai:TestDataset))

Capability Relationships

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:enables ai:AutomatedEvaluation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:enables ai:ModelComparison))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:enables ai:RegressionTesting))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:enables ai:ReproducibleBenchmarking))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:enables ai:SystemRanking))

Implementation Relationships

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:implements ai:EvaluationMetric))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:implements ai:ModifiedPrecision))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:implements ai:NGramOverlapSimilarity))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:implements ai:CorpusLevelEvaluation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:implements ai:ReproducibleBenchmarking))

Support Relationships

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:supports ai:MachineTranslation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:supports ai:TextSummarisation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:supports ai:BenchmarkEvaluation))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:supports ai:NaturalLanguageProcessing))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:supports ai:RegressionTesting))

Contrast Relationships

SubClassOf(ai:BLEUScore
  ObjectAllValuesFrom(ai:contrastsWith ai:COMETMetric))
SubClassOf(ai:BLEUScore
  ObjectAllValuesFrom(ai:contrastsWith ai:BERTScore))
SubClassOf(ai:BLEUScore
  ObjectAllValuesFrom(ai:contrastsWith ai:HumanEvaluation))
SubClassOf(ai:BLEUScore
  ObjectAllValuesFrom(ai:contrastsWith ai:METEOR))
SubClassOf(ai:BLEUScore
  ObjectAllValuesFrom(ai:contrastsWith ai:EvaluationHarness))

Standardisation Relationships

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:standardizedBy ai:SacreBLEU))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:standardizedBy ai:WMTBenchmark))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:standardizedBy ai:ACLSharedTask))

Reduction Relationships

SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:reducesTo ai:EvaluationMetric))
SubClassOf(ai:BLEUScore
  ObjectSomeValuesFrom(ai:reducesTo ai:PrecisionMeasure))

About

BLEU was introduced by Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu of IBM T.J. Watson Research Centre in a landmark paper presented at the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002). The paper, titled “BLEU: a Method for Automatic Evaluation of Machine Translation,” demonstrated for the first time that an automated metric based on n-gram string overlap could achieve statistically significant correlation with human quality judgements of machine translation output when evaluated at the corpus level — across hundreds to thousands of translated sentences aggregated together.

The timing of BLEU’s introduction was pivotal. The early 2000s saw the rise of statistical machine translation (SMT), particularly phrase-based models, which required rapid evaluation of competing system configurations, training data conditions, and decoding strategies. Before BLEU, every evaluation required expensive human annotation, which throttled the research iteration cycle. BLEU changed this: given a set of candidate translations and human reference translations, BLEU could be computed automatically in seconds, enabling teams to run tens of experiments per day and select configurations that would plausibly improve human-evaluated quality. This enabled the rapid progress of statistical MT through the 2000s and established the norm of automated metric reporting that persists in NLP research to the present day.

The metric became so dominant that for over a decade, “improving BLEU” was essentially synonymous with “improving machine translation” in the academic literature. WMT (Workshop on Machine Translation) shared tasks adopted BLEU as their primary ranking metric from their inception, cementing it as the field’s lingua franca for system comparison. The flip side of this dominance was an increasing awareness of the metric’s failure modes, which culminated in Post (2018) introducing SacreBLEU to at least ensure that different papers were computing the same number when they reported a “BLEU score.”

By 2014–2016, as neural sequence-to-sequence models began to outperform phrase-based SMT, BLEU’s limitations became more visible: neural MT systems could produce fluent, semantically adequate translations that differed lexically from references in legitimate ways (synonyms, paraphrase, grammatical alternations), and BLEU penalised these equally with genuinely wrong translations. This mismatch between BLEU and human quality drove sustained research into alternative metrics — METEOR, TER, CHRF, BERTScore, BLEURT, COMET — each addressing different failure modes.

As of 2026, BLEU occupies a curious legacy position: reported in virtually all machine translation papers for backward compatibility and reproducibility, but rarely treated as a reliable primary quality signal by practitioners. WMT 2025 expanded evaluation using Error Span Annotations (ESA) and Multidimensional Quality Metrics (MQM) for high-resource language pairs, with COMET-22 as the primary neural metric. In LLM evaluation more broadly, BLEU has largely been displaced by task-specific accuracy metrics, embedding-based similarity, and preference judgements from stronger model judges.

Historical Development and Paradigm Shifts

The history of BLEU Score is inseparable from the history of machine translation evaluation, which in turn reflects the broader history of how the NLP community operationalises scientific progress. Before BLEU, the field had no agreed automatic metric: researchers evaluated MT systems by having human translators rank outputs or score fluency and adequacy on Likert scales. This approach was expensive, slow (weeks to obtain evaluation results), non-reproducible (different annotators produce different rankings), and non-comparable across labs (each group designed its own human evaluation protocol). Papineni et al. (2002) demonstrated that a simple n-gram precision measure, calibrated against human judgements across a variety of translation outputs, could provide a cheap, fast, reproducible proxy for human quality rankings at the corpus level. The paper showed correlations of Kendall’s τ = 0.82–0.98 with human rankings across four human translation quality levels evaluated on a Chinese-to-English test set, a result that was immediately compelling to the MT research community.

The paper is notable for what it explicitly acknowledged as limitations in its first section: BLEU is “not meant to predict human judgments of translation quality for individual sentences” and is reliable only at the corpus level. These caveats were largely ignored in subsequent practice, contributing to the pathologies that plagued BLEU-based research for two decades.

Statistical MT era (2002–2013): BLEU became the primary optimisation target for phrase-based SMT systems. Minimum Error Rate Training (MERT, Och 2003) directly optimised BLEU on a development set by adjusting feature weights via line search. This direct coupling of training signal and evaluation metric was effective for SMT because SMT systems generated hypotheses that were lexically close to references, making BLEU a reasonable proxy. The WMT shared tasks (beginning 2006) further institutionalised BLEU as the primary ranking metric.

Neural MT transition (2014–2019): The introduction of attention-based encoder-decoder models (Bahdanau et al., 2015) and then Transformer-based NMT (Vaswani et al., 2017) produced translations that were qualitatively different from SMT outputs — more fluent, more idiomatic, but sometimes less word-for-word accurate. BLEU scores for NMT systems were numerically similar to or only slightly above competitive SMT systems on standard benchmarks, despite human evaluators strongly preferring NMT outputs (a finding first documented in a WMT human evaluation). Callison-Burch et al. (2006) had already highlighted cases where BLEU could be gamed (high BLEU, low quality), and the NMT era made these limitations more systematic. Multiple papers (notably Reiter 2018, Mathur et al. 2020) provided rigorous empirical evidence of BLEU’s failures.

Post-2018 — learned metrics and the BERTScore era: The success of BERT and related contextual embedding models opened a new paradigm for evaluation: instead of counting surface string overlaps, metrics could measure semantic similarity using contextual embeddings. BERTScore (Zhang et al., 2020) and BLEURT (Sellam et al., 2020) demonstrated substantially better correlation with human judgements by leveraging BERT-derived representations. COMET (Rei et al., 2020) pushed further by training a regression model directly on human direct assessment scores from WMT. By WMT 2020–2022, COMET and BLEURT were the top-ranked metrics in the WMT metrics shared task, with BLEU ranked far below learned metrics in correlation with human segment-level judgements.

LLM era (2023–2026): The proliferation of large language models as translation systems introduced a fundamental challenge for BLEU: LLM translations are often preferred by human evaluators even when they score similarly on BLEU to specialised NMT systems, because LLMs produce more natural, contextually appropriate translations that may differ lexically from reference translations in ways BLEU penalises. This further eroded BLEU’s credibility as a primary quality signal and accelerated the transition to COMET, human preference evaluations, and LLM-as-judge frameworks.

Formal Algorithm and Computation

The BLEU computation proceeds in five steps:

Step 1 — Modified N-Gram Precision For n-gram order n ∈ {1, 2, 3, 4}, compute modified precision pₙ:

• For each n-gram in the candidate translation, count its occurrences (Count)

• Clip this count by the maximum count of that n-gram in any single reference (Count_clip)

• pₙ = Σ Count_clip(n-gram) / Σ Count(n-gram) summed over all candidate n-grams

  The clipping prevents credit for repeated n-grams beyond their reference frequency. Without clipping, a candidate that repeats a common unigram many times could achieve a high unigram precision score despite being a degenerate translation.

  Step 2 — Geometric Mean Compute the log-space weighted sum of modified precisions: log(BLEU) = Σ wₙ × log(pₙ) for n from 1 to N where wₙ = 1/N (equal weights by default, N=4). This is equivalent to taking the geometric mean of the four precision values.

  Step 3 — Brevity Penalty BP = 1 if c > r, else BP = exp(1 − r/c) where c is the total length of candidate translations and r is the effective reference length (closest reference length for each sentence, summed over the test corpus). The brevity penalty equals 1 (no penalty) when the candidate is longer than or equal to the reference, and declines exponentially when the candidate is shorter.

  Step 4 — BLEU Score BLEU = BP × exp(Σ wₙ × log(pₙ))

  Step 5 — Tokenisation The original Papineni et al. implementation used a specific tokenisation scheme (punctuation splitting, lowercasing). Variations in tokenisation produce different numerical scores for the same candidate and reference, making cross-paper comparisons unreliable. SacreBLEU (Post, 2018) standardises tokenisation and computes the score on detokenised output with a canonical internal tokeniser, reporting a version string (e.g., BLEU+case.mixed+lang.en-de+...) that fully specifies computation parameters.

  Corpus vs. Sentence Level BLEU is designed for corpus-level evaluation. Sentence-level BLEU suffers from sparsity: short sentences have very few n-grams, so precision values fluctuate wildly. SacreBLEU and related tools discourage sentence-level BLEU as a primary signal. Sentence-level alternatives include METEOR and BLEURT.

  Major Variants and Related Metrics

• SacreBLEU (Post, 2018): Standardised, reproducible BLEU implementation with canonical tokenisation and version strings. The de facto standard for all contemporary MT evaluation. Available via the sacrebleu Python package (PyPI) and integrated into Hugging Face evaluate.

• METEOR (Banerjee & Lavie, 2005; Denkowski & Lavie, 2014): Alignment-based metric incorporating unigram recall (not just precision), exact word matching, stemming, synonym matching via WordNet, and paraphrase matching. Higher correlation with human judgement at sentence level than BLEU. Commonly reported alongside BLEU for MT evaluation.

• TER (Translation Edit Rate, Snover et al., 2006): Measures the minimum number of word edits (insertions, deletions, substitutions, and phrase shifts) required to transform the hypothesis into the reference, normalised by reference length. Lower is better. Captures reordering errors that BLEU misses.

• chrF (Character n-gram F-score, Popovic, 2015): Computes F-score of character n-gram overlap between hypothesis and reference. Particularly effective for morphologically rich languages. SacreBLEU includes chrF and chrF++ computation.

• BERTScore (Zhang et al., 2020): Leverages contextual BERT embeddings to compute token-level similarity between hypothesis and reference via cosine similarity, yielding precision, recall, and F1 measures. Substantially better correlation with human judgements than BLEU, particularly for paraphrase-equivalent translations and diverse outputs. Sensitive to the BERT model variant and layer used.

• COMET Metric (Rei et al., 2020): A family of learned evaluation metrics trained on human direct assessment (DA) or MQM (Multidimensional Quality Metrics) annotations. COMET uses XLM-RoBERTa as backbone and produces scores that correlate far more strongly with human quality judgements than BLEU. COMET-22 and COMET-KIWI are the de facto standard at WMT 2023–2025. Reference-free COMET-KIWI enables quality estimation without human references.

• BLEURT (Sellam et al., Google Brain, 2020): Pre-trains a BERT-based model on a mixture of synthetic and human-annotated evaluation data to produce a regression score directly predicting human quality ratings. Strong performance on WMT metrics shared tasks.

• ROUGE (Lin, 2004): Recall-oriented n-gram overlap metric widely used for text summarisation. ROUGE-1, ROUGE-2 (bigrams), and ROUGE-L (longest common subsequence) are the standard variants. ROUGE has the same fundamental limitations as BLEU (surface-level matching, synonym insensitivity) but emphasises recall rather than precision.

• MQM / ESA (Multidimensional Quality Metrics / Error Span Annotations): Human annotation frameworks that decompose translation quality into error categories (accuracy, fluency, terminology, style) with severity weights. Used in WMT 2022–2025 to produce gold standard human scores that COMET and other learned metrics are trained to predict.

  Use Cases

• Machine Translation Research: BLEU is the standard metric reported in every MT paper since 2002. WMT shared tasks (WMT14–WMT25) provide standardised test sets with BLEU and COMET leaderboards, enabling direct comparison of all competing systems. BLEU on WMT14 English-German (BLEU-4, sacrebleu) is a de facto canonical benchmark: BLEU 30–35 represents strong neural MT; human translators score ~50.

• Regression Testing in MT Systems: Industrial MT providers (Google, DeepL, Microsoft) use BLEU on held-out test sets as a regression gate in continuous integration pipelines. A significant BLEU drop on a standard test set triggers investigation before deployment, even when BLEU is not the primary quality target.

• Low-Resource Language Evaluation: BLEU is particularly valuable when human evaluation resources are limited (no native speaker annotators for a language pair). For low-resource language pairs in FLORES-200 and related benchmarks, BLEU and chrF remain primary because learnable metrics like COMET require training data that does not exist for rare languages.

• Text Summarisation: While ROUGE is the dominant metric for summarisation, BLEU is reported alongside ROUGE in some summarisation papers for cross-task comparability, particularly for abstractive summarisation where n-gram overlap between output and reference remains a useful signal.

• Dialogue and Question Answering: BLEU was widely applied to dialogue response generation evaluation in the 2015–2020 period, before it was demonstrated (Liu et al., 2016) that BLEU has near-zero correlation with human appropriateness and interestingness in open-domain dialogue. This finding significantly reduced its use in dialogue research.

• Code Generation: BLEU-4 (and its variant CodeBLEU, Ren et al., 2020) is used to evaluate code generation quality by comparing generated code tokens to reference solutions. CodeBLEU extends BLEU with AST and data-flow graph matching to capture structural code similarity beyond token surface match.

• LLM Output Evaluation: In LLM evaluation frameworks (Hugging Face evaluate, EleutherAI LM Evaluation Harness), BLEU is available as a metric but is treated as a baseline footnote. Task-specific accuracy, perplexity, BERTScore, and COMET are preferred for translation; for other generation tasks, instruction-following accuracy or LLM-judge preference is standard.

• Iterative System Development: Even with its limitations, BLEU’s zero-cost computation and high reproducibility (via SacreBLEU) make it valuable for rapid iteration during development. Teams use BLEU as a quick signal to filter large numbers of experimental configurations before investing in human evaluation or COMET scoring of the top candidates.

  Academic Context

  BLEU was introduced into a landscape where human evaluation of MT was the norm, evaluations were expensive and slow, and no standard existed for comparing systems across labs. The Papineni et al. (2002) paper demonstrated that BLEU correlated with human rankings at the system level with correlation coefficients of 0.82–0.98 depending on language pair and test set, across a comparison of four human translation outputs evaluated at different quality levels. This correlation was measured using Kendall’s τ rank correlation, not Pearson, emphasising rank ordering over linear correspondence.

  The critical limitation identified even in the original paper was that BLEU is meaningful only at the corpus level. Sentence-level BLEU is unreliable because individual sentences have too few n-grams to produce stable estimates. The paper recommended using a test corpus of at least 300–400 sentences for reliable system comparison, a guideline widely ignored in subsequent practice.

  Post (2018) identified that the NLP community was computing BLEU differently in different papers — different tokenisation schemes, different handling of case and punctuation — making published scores non-comparable even when purportedly measuring the same test set. SacreBLEU addressed this directly by providing a single canonical implementation with a version string that encodes all computation parameters. Adoption of SacreBLEU as the standard has been strong since 2019, particularly after WMT began requiring it for official submissions.

  The metrics shared tasks at WMT (beginning with WMT 2008) created a systematic framework for evaluating evaluation metrics by collecting human quality judgements (direct assessment, relative ranking, MQM annotations) and measuring how well automatic metrics correlate with them. Results from WMT 2020–2025 metrics shared tasks consistently show COMET, BLEURT, and hybrid metrics outperforming BLEU by large margins in correlation with human judgement, particularly at the segment level and for high-resource language pairs. BLEU’s remaining value is its zero training-data requirement, interpretability, and the decades of published scores that provide historical comparison baselines.

  Key theoretical limitations identified in the literature (Callison-Burch et al., 2006; Reiter, 2018; Mathur et al., 2020):

• BLEU does not model synonymy or paraphrase equivalence; a valid translation using different but correct words scores identically to a wrong translation with the same word mismatch.

• BLEU is insensitive to word order beyond n-gram boundaries (n ≤ 4); long-distance reordering errors that produce ungrammatical or misleading outputs are invisible to BLEU.

• BLEU cannot assess factual accuracy, hallucination, or named entity correctness.

• BLEU correlates poorly with human judgement at the sentence level and when comparing systems of fundamentally different architectures (SMT vs. NMT, or NMT vs. LLM-based translation).

• Goodhart’s Law applies: when BLEU became the optimisation target, MT systems learned to game it — producing shorter, higher-precision outputs at the expense of recall and naturalness.

  Current Landscape (2026)

  BLEU occupies a stable but diminished role in the 2026 NLP evaluation ecosystem. Several trends characterise its current position:

• WMT 2025: The WMT 2025 shared task on automated metrics (Findings of WMT25 Shared Task on Automated Metrics, ACL Anthology 2025) used COMET-22 (specifically wmt22-comet-da from the Unbabel/COMET framework v2.0.2 with XLM-RoBERTa-large backbone) as the primary metric for system ranking in high-resource language pairs, and expanded Error Span Annotation (ESA) collection for gold human scores. BLEU was still computed and reported for all systems but was explicitly not used for ranking decisions.

• Machine Translation Leaderboards (2026): Benchmarks such as FLORES-200, WMT24, and the Machine Translation Benchmarks Leaderboard (awesomeagents.ai, 2026) rank systems primarily by COMET and human preference. BLEU appears as a secondary figure. IWSLT 2024–2025 cascaded systems achieved BLEU 24–28 on English-to-German, but these figures are contextualised with COMET and human preference data.

• LLM Translation: Large language models including GPT-4, Claude 3.5, and Gemini 1.5 have overtaken specialised MT engines on human evaluation for high-resource language pairs. On BLEU, LLM translations sometimes score lower than specialised MT systems despite being preferred by human evaluators — a concrete demonstration of BLEU’s failure to capture actual translation quality in the LLM era.

• LLM general evaluation: Frameworks such as EleutherAI LM Evaluation Harness, Hugging Face evaluate, and HELM (Holistic Evaluation of Language Models) include BLEU as one metric among many but emphasise task-specific accuracy, multiple-choice performance, and calibration as primary signals. For generation tasks, LLM-judge frameworks (using GPT-4 or a fine-tuned judge model to rate outputs) increasingly supplement or replace n-gram metrics.

• SacreBLEU adoption: SacreBLEU 2.x (available via PyPI) is the universal implementation; papers that still report BLEU universally use SacreBLEU with version strings. The sacrebleu package has been downloaded hundreds of millions of times, reflecting BLEU’s continued importance as a reproducibility baseline even where it is not the primary metric.

• Low-resource languages: BLEU and chrF remain the dominant metrics for rare and endangered language evaluation where COMET models have not been trained and human annotation is scarce. FLORES-200 (a benchmark for 200 languages) relies on BLEU and chrF as primary metrics for most language directions.

  UK Context

  The UK has contributed meaningfully to MT evaluation research and benefits practically from machine translation infrastructure:

• University of Edinburgh: Edinburgh’s MT group (including members of the EMNLP and WMT communities) has been active in both MT system development and evaluation methodology, contributing to WMT shared tasks and metrics research. The group has historically advocated for robust evaluation beyond BLEU, including work on human evaluation protocols and the limitations of automatic metrics.

• Alan Turing Institute: The ATI has hosted workshops on evaluation methodology in NLP and AI, including discussions of the limitations of BLEU and the transition to learned metrics for assessing large language model outputs. This intersects with the ATI’s broader agenda on AI transparency and responsible evaluation.

• NHS and Public Sector Translation: The UK’s National Health Service operates in a multilingual environment — particularly in Northern English NHS trusts (Manchester, Leeds, Bradford) serving large populations of patients whose first language is not English. MT systems evaluated and developed partly with BLEU-based methodology are deployed or evaluated for patient communication, discharge summary translation, and multilingual triage. The limitations of BLEU are practically salient here: semantic adequacy and factual accuracy (poorly captured by BLEU) matter more than lexical overlap for safety-critical medical translation.

• Unbabel (UK/Portugal): Unbabel, which develops the COMET metric family — the primary alternative to BLEU at WMT 2023–2025 — has significant presence in the UK and EU translation market, providing quality-assured machine-assisted translation for enterprises. Their deployment of reference-free COMET-KIWI for production quality estimation directly displaces BLEU in industrial MT quality assurance.

• BBC and Media: UK media organisations use MT for multilingual subtitle generation and news translation. Evaluation of these systems uses a combination of BLEU (for automated regression) and human post-editing effort metrics that capture quality beyond n-gram overlap.

• Sheffield NLP Group: Sheffield researchers have contributed to MT evaluation and low-resource language work, including evaluation of MT quality for minority languages (Welsh, Scottish Gaelic) where BLEU and chrF remain the primary automated metrics due to the absence of COMET training data for these language directions.

• UKRI-funded NLP research: EPSRC and UKRI programmes have funded MT and NLP evaluation research at multiple UK universities. The transition from BLEU to neural metrics is a recurring theme in grant applications and papers from the UK NLP community.

  Future Directions (2026–2030)

• Neural metric dominance: COMET, BLEURT, and successor learned metrics will continue to displace BLEU as the primary ranking signal in high-resource MT evaluation. WMT is expected to adopt COMET-based ranking officially for all language pairs with sufficient training data within the 2026–2028 timeframe.

• LLM-based evaluation: Using large language models (GPT-4, Claude, Gemini) as judges to rate translation quality — scoring fluency, adequacy, and terminology separately — will become a standard complement to automatic metrics. These “LLM-as-judge” approaches produce interpretable error analysis alongside a scalar score, addressing a major gap in BLEU.

• Reference-free evaluation at scale: COMET-KIWI and its successors enable quality estimation without human reference translations, opening evaluation to unseen language pairs and low-resource directions at scale. This dramatically reduces the dependence on parallel reference corpora that BLEU requires.

• Factuality and faithfulness metrics: For translation and summarisation in safety-critical domains, new metrics that specifically measure factual accuracy (whether the hypothesis preserves all facts from the source), hallucination rate, and named entity fidelity will emerge. BLEU is blind to all of these.

• Multimodal evaluation: As MT expands to speech and multimodal inputs (speech, images, video), new evaluation frameworks must capture quality across modalities. BLEU-like surface metrics are poorly suited to this; learned perceptual metrics calibrated on multimodal human judgements will be required.

• Standardisation and governance: EU AI Act and international AI standards (ISO/IEC 42001, forthcoming evaluation standards) will require organisations deploying high-risk AI systems (including MT in healthcare and legal contexts) to document evaluation methodology. This will drive formalisation of which metrics are acceptable for conformity assessment, likely requiring neural metrics alongside or instead of BLEU for high-stakes applications.

• BLEU’s persistence as a baseline: Despite its limitations, BLEU will remain reported in the NLP literature for the foreseeable future as a backward-compatibility baseline, enabling comparison with decades of published results. Its computational efficiency and complete interpretability (no neural backbone required) ensure it remains a useful sanity check even when not a primary quality signal.

• Automatic evaluation democratisation: For research groups, startups, and organisations in low-income countries or with limited access to commercial LLM APIs, BLEU remains a zero-cost, zero-API-call evaluation option that enables MT system development without dependency on paid evaluation infrastructure. This democratising role is likely to sustain BLEU’s use in resource-constrained research environments even as wealthier labs adopt COMET and LLM-judge evaluation as defaults.

• Regulatory metrics standardisation: As the EU AI Act and international AI governance frameworks develop standards for high-risk AI system evaluation in translation and NLP, there is likely to be pressure for standardised evaluation protocols. SacreBLEU with version strings may be adopted as a minimum required metric documentation standard, while learned metrics like COMET become recommended for high-stakes applications.

• Sentence-level evaluation recovery: Research into better sentence-level smoothing methods for BLEU (beyond the existing add-one and exponential smoothing methods in SacreBLEU) may rehabilitate sentence-level BLEU for certain applications, particularly in iterative post-editing workflows where sentence-level quality signals are needed without the computational cost of COMET.

• Hybrid metric packages: Future evaluation suites will likely bundle BLEU, chrF, COMET, and BERTScore together with statistical significance testing into single pip-installable packages, making multi-metric evaluation as simple as single-metric BLEU computation is today, further reducing BLEU’s role as the sole default metric.

  Benchmark Datasets and Evaluation Settings

  BLEU is computed over standard MT test sets that define the canonical benchmarks of the field:

• WMT14 English-German (En-De): Newstest2014 is the most widely cited MT benchmark, with 3,003 sentence pairs from news text. BLEU-4 (SacreBLEU, tokenised with the WMT tokeniser) is the standard reporting format. State-of-the-art 2020–2024 NMT systems score approximately BLEU 30–35; human translators score approximately BLEU 50 on average.

• WMT14 English-French (En-Fr): 3,003 sentence pairs; one of the highest-resource MT test sets. Used to compare SMT vs. NMT in early neural MT papers. State-of-the-art NMT achieves BLEU 40–45.

• WMT17–WMT25 Test Sets: The WMT shared tasks produce annual test sets for 10–30 language pairs from diverse domains including news, social media, biomedical, and legal text. Newstest is used for general domain; Flores and WMT Biomedical subsets for domain-specific evaluation.

• FLORES-200 (Meta, 2022): A benchmark for 200 languages, including low-resource and endangered languages, based on Wikipedia and Wikinews. BLEU-4 and chrF++ are the primary metrics because learned metrics like COMET have insufficient training data for most languages. NLLB-200 achieved BLEU scores ranging from below 5 (for very low-resource pairs) to above 40 (for high-resource European pairs).

• IWSLT (International Workshop on Spoken Language Translation): Annual test sets for spoken language translation from TED talks and similar sources. BLEU-4 is the standard metric; IWSLT 2024–2025 cascaded systems achieved BLEU 24–28 on En-De.

• SacreBLEU Test Set IDs: SacreBLEU maintains a registry of official WMT and IWSLT test sets that can be automatically downloaded and scored. Using SacreBLEU with the official test set ID ensures scores are exactly reproducible (e.g., sacrebleu -t wmt14 -l en-de < output.txt).

• Flores-101 and Flores-200: Multilingual evaluation sets widely used to assess BLEU on 100–200 language directions simultaneously. Facebook’s NLLB-200 and Google’s Gemini translation capabilities are compared on these sets, with BLEU serving as a secondary metric behind COMET and human preference.

  Key Terminology

• N-gram: A contiguous sequence of n tokens. BLEU uses n=1 (unigram), n=2 (bigram), n=3 (trigram), and n=4 (4-gram) with equal weighting.

• Modified precision: N-gram precision computed with clipping — each candidate n-gram is credited at most as many times as the maximum count of that n-gram in any single reference translation. Prevents gaming by repetition.

• Brevity Penalty (BP): Multiplicative penalty for candidates shorter than references: BP = min(1, exp(1 − r/c)). Equals 1.0 (no penalty) when candidate length c ≥ reference length r; declines exponentially for shorter candidates.

• Geometric mean: BLEU takes the geometric mean of the four modified precision values (equivalent to the exponential of the average log-precision), which ensures that any zero-valued precision (zero n-gram matches) collapses the entire score to zero, preventing meaningless scores on pathological outputs.

• Corpus-level vs. sentence-level BLEU: Corpus-level BLEU aggregates n-gram counts across all sentences before computing precision, producing stable estimates. Sentence-level BLEU (BLEU applied to individual sentences) is unreliable due to sparse n-gram counts and is generally discouraged.

• SacreBLEU: The standardised, canonical BLEU implementation (Post, 2018) that detokenises output, applies its own internal tokenisation, and reports a version string encoding all computation parameters. The de facto standard since 2019.

• chrF: Character-level F-score metric (Popovic, 2015) computed as the harmonic mean of character n-gram precision and recall. Better than BLEU for morphologically rich languages; reported alongside BLEU in most contemporary MT evaluations.

• COMET: Crosslingual Optimised Metric for Evaluation of Translation (Rei et al., 2020). A family of learned metrics fine-tuned on human quality annotations from WMT, based on XLM-RoBERTa. Substantially stronger correlation with human judgement than BLEU. The de facto primary metric at WMT 2023–2025.

• BERTScore: A metric using contextual BERT embeddings to compute token-level semantic similarity between hypothesis and reference via cosine similarity, yielding precision, recall, and F1 values. Addresses BLEU’s synonym insensitivity.

• TER (Translation Edit Rate): Minimum word edit distance between hypothesis and reference, normalised by reference length. Captures reordering and phrasal errors BLEU misses. Lower is better.

• MQM (Multidimensional Quality Metrics): A fine-grained human annotation framework that decomposes translation quality into typed error spans (accuracy, fluency, style, terminology) with severity weights (minor, major, critical). Used at WMT 2021–2025 for gold standard human scores.

• BLEURT: A learned metric (Sellam et al., 2020, Google Brain) trained on a mixture of synthetic and human-annotated evaluation data, producing a regression score directly predicting human quality ratings.

• Reference translation: One or more human translations of the source sentence against which the machine translation hypothesis is evaluated. BLEU supports multiple references via clipped precision — each n-gram credit is capped at the maximum frequency in any single reference.

• Goodhart’s Law in MT evaluation: The observation that when BLEU is used as an optimisation target (e.g., in minimum risk training or reinforcement learning from BLEU rewards), models learn to maximise n-gram precision without necessarily improving actual translation quality, often producing shorter or more repetitive outputs.

  Detailed Comparison: BLEU vs. Alternative Metrics

  Understanding BLEU requires comparing it systematically to the metrics that have been proposed to address its limitations:

Property	BLEU	METEOR	chrF	TER	BERTScore	COMET
Reference required	Yes (1+)	Yes (1+)	Yes (1+)	Yes (1+)	Yes (1+)	Optional (COMET-KIWI: no)
Synonym sensitivity	No	Yes (WordNet)	Partial	No	Yes	Yes
Character-level	No	Partial	Yes	No	No	No
Reordering	Partial (n≤4)	Yes (phrase-level)	Partial	Yes (shifts)	Yes	Yes
Sentence-level	Unreliable	Better	Better	Better	Better	Best
Human correlation	Low-medium	Medium	Medium	Low-medium	High	Highest
Training data required	No	No	No	No	Pretrained LM	Yes (QE model)
Inference speed	Very fast	Fast	Very fast	Fast	Moderate	Slow
Interpretability	High	Medium	Medium	High	Low	Low

This comparison highlights the core trade-off: BLEU’s speed, simplicity, and interpretability make it useful as a cheap proxy and historical baseline, but its low human correlation and insensitivity to semantic equivalence make it inadequate as a primary quality signal when richer metrics are computationally feasible.

When BLEU remains the right choice:

• Rapid development iteration where compute is limited

• Low-resource language evaluation where COMET models do not exist

• Historical comparison with published results that used BLEU

• Regression testing where consistent, reproducible results matter more than absolute quality

• Situations where human references are not available for COMET training but are available for BLEU computation

  When BLEU should be replaced or supplemented:

• Primary evaluation of production MT system quality claims

• Evaluation of paraphrastic, creative, or literary translation

• Evaluation of LLM-generated translation where output differs lexically from references

• Medical or legal translation where semantic accuracy and factual correctness are safety-critical

• Segment-level quality estimation in post-editing workflows

  Software Ecosystem and Tooling

  BLEU is implemented in virtually every NLP evaluation library:

• SacreBLEU (mjpost/sacrebleu on GitHub): The canonical implementation. Manages test set downloading, tokenisation, and scoring with version string reporting. 1 million+ PyPI downloads. Integrated into Hugging Face evaluate, LM Evaluation Harness, and most MT research frameworks. Actively maintained with support for 100+ WMT and IWSLT test sets.

• Hugging Face evaluate: Wraps SacreBLEU as the bleu metric. Compatible with all Hugging Face datasets and models. Provides consistent API alongside ROUGE, BERTScore, METEOR, and COMET.

• NLTK (Natural Language Toolkit): Includes an older, non-standard BLEU implementation (nltk.translate.bleu_score). Not recommended for publication-quality results due to non-standardised tokenisation; use SacreBLEU instead.

• torchmetrics: PyTorch-compatible BLEU implementation for in-training evaluation on GPU.

• TensorBLEU (arXiv:2510.05485, 2025): Vectorised GPU-based BLEU implementation enabling per-sentence evaluation during training at GPU-accelerated speed, making sentence-level BLEU feasible as a training monitoring signal.

• OpenNMT-py and OpenNMT-tf: Include BLEU evaluation as part of their MT training and evaluation pipelines.

• Fairseq (Facebook/Meta): MT training framework that computes BLEU during training using sacrebleu, providing per-checkpoint BLEU curves for model selection.

• EleutherAI LM Evaluation Harness: Includes BLEU as one metric in its comprehensive NLP evaluation suite, alongside exact match, F1, BERTScore, and task-specific metrics.

  Research and Literature

  1. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of ACL 2002, 311–318.
  2. Post, M. (2018). A Call for Clarity in Reporting BLEU Scores. Proceedings of WMT 2018, 186–191. [SacreBLEU]
  3. Rei, M., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A Neural Framework for MT Evaluation. Proceedings of EMNLP 2020, 2685–2702. arXiv:2009.09025.
  4. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR 2020. arXiv:1904.09675.
  5. Sellam, T., Das, D., & Parikh, A. P. (2020). BLEURT: Learning Robust Metrics for Text Generation. Proceedings of ACL 2020, 7881–7892. arXiv:2004.04696.
  6. Banerjee, S., & Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. Proceedings of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarisation, 65–72.
  7. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A Study of Translation Edit Rate with Targeted Human Annotation. Proceedings of AMTA 2006, 223–231.
  8. Popovic, M. (2015). chrF: Character n-gram F-score for Automatic MT Evaluation. Proceedings of WMT 2015, 392–395.
  9. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of ACL Workshop on Text Summarisation Branches Out, 74–81.
  10. Callison-Burch, C., Osborne, M., & Koehn, P. (2006). Re-evaluating the Role of BLEU in Machine Translation Research. Proceedings of EACL 2006, 249–256.
  11. Liu, C.-W., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016). How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Proceedings of EMNLP 2016. arXiv:1603.08023.
  12. Reiter, E. (2018). A Structured Review of the Validity of BLEU. Computational Linguistics, 44(3), 393–401.
  13. Mathur, N., Wei, J., Freitag, M., Ma, Q., & Bojar, O. (2020). Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. Proceedings of ACL 2020, 4984–4997. arXiv:2006.06264.
  14. Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., & Macherey, W. (2021). Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Transactions of the ACL, 9, 1460–1474. arXiv:2104.14478.
  15. Freitag, M., Bleu, D., Rei, M., Mathur, N., Lo, C.-k., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., & Bojar, O. (2022). Results of the WMT22 Metrics Shared Task. Proceedings of WMT 2022, 46–98.
  16. Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Herold, C., Junczys-Dowmunt, M., Koehn, P., Koppel, K., Lo, C.-k., Morishita, M., Nagata, M., Nakazawa, T., & Knowles, R. (2023). Findings of the 2023 Conference on Machine Translation (WMT23). Proceedings of WMT 2023.
  17. Kocmi, T., et al. (2025). Findings of the WMT25 Shared Task on Automated Metrics. Proceedings of WMT 2025. ACL Anthology 2025.wmt-1.24.
  18. Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., & Ma, S. (2020). CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297.
  19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. [Transformer architecture underlying modern MT systems evaluated by BLEU]
  20. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., & Tamchyna, A. (2014). Findings of the 2014 Workshop on Statistical Machine Translation. Proceedings of WMT 2014, 12–58. [Canonical WMT14 En-De BLEU benchmark]
  21. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised Cross-Lingual Representation Learning at Scale (XLM-RoBERTa). Proceedings of ACL 2020. [Backbone for COMET replacing BLEU at WMT]
  22. Bentivogli, L., Bisazza, A., Cettolo, M., & Federico, M. (2016). Neural versus Phrase-Based Machine Translation Quality: a Case Study. Proceedings of EMNLP 2016. [Early study on BLEU vs. NMT quality disconnect]
  23. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473. [Attention mechanism that drove NMT improvements measured by BLEU]
  24. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019. [Foundation for BERTScore]
  25. NLLB Team (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672. [200-language MT evaluated with BLEU, chrF, and human scores]
  26. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., & Herbst, E. (2007). Moses: Open Source Toolkit for Statistical Machine Translation. Proceedings of ACL 2007 Demo and Poster Sessions, 177–180. [Primary SMT system on which BLEU was developed and standardised]
  27. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014. arXiv:1409.3215. [Seq2seq foundation for NMT evaluated with BLEU]

  Critique, Failures, and Debates in the Literature

  BLEU has attracted sustained critical scrutiny from the NLP research community over more than two decades. The critique is not merely that BLEU is imperfect — all metrics are imperfect — but that BLEU’s specific failure modes are systematically misaligned with what practitioners actually care about when they evaluate NLP systems.

  The synonymy problem: The most fundamental issue with BLEU is that it treats word identity as the criterion for a match. Two sentences that express the same proposition using different but correct words score identically to two sentences where one is simply wrong. If a reference says “the vehicle was destroyed” and a hypothesis says “the car was demolished,” BLEU credits zero overlap for these words despite complete semantic equivalence. This problem is not marginal: it pervades every domain with rich vocabulary (legal text, medical reporting, literary translation, technical documentation) and is particularly severe when comparing translations by different professional translators, who naturally choose different but equally valid vocabulary.

  The word order problem: BLEU’s n-gram model captures local word order up to n=4 but is blind to reordering beyond four consecutive words. A hypothesis with systematically inverted phrase orders within clauses can score high on BLEU if local n-gram sequences match, despite producing structurally incorrect target-language syntax. For languages with very different word orders from English (Japanese, Turkish, Hindi), this is a serious limitation that BLEU shared with most unigram-heavy metrics.

  The sentence-level instability problem: BLEU’s corpus-level design means sentence-level scores are statistically unreliable. A sentence with ten words has at most seven 4-grams; if none match the reference, the 4-gram BLEU component is zero regardless of how close the translation otherwise is. Many papers report sentence-level BLEU anyway, producing scores that fluctuate randomly across similar quality outputs. SacreBLEU’s documentation explicitly warns against this use, but it persists in the literature.

  The multiple reference problem: BLEU’s handling of multiple reference translations is theoretically principled (each n-gram credited at most as many times as in the best reference) but practically problematic. The effective ceiling for BLEU rises with more references, making comparisons between papers that use different numbers of references invalid. WMT test sets for some language pairs provide four references while others provide one, making cross-test-set comparison treacherous even with SacreBLEU.

  Callison-Burch et al. (2006) — “Re-evaluating the Role of BLEU”: This ACL 2006 paper documented cases where BLEU rankings of MT systems diverged from human rankings, showing that system modifications that improve BLEU do not necessarily improve human-evaluated quality and vice versa. The paper demonstrated specific failure cases involving synonyms, word choice, and phrase boundary detection, and called for metric diversity rather than reliance on BLEU alone.

  Reiter (2018) — “A Structured Review of the Validity of BLEU”: A systematic review of the psychometric validity of BLEU as a metric for natural language generation quality. Reiter applied test theory (face validity, content validity, criterion validity, construct validity) to BLEU and found it lacking on every criterion, concluding that “BLEU is not a valid metric for text quality” in the general sense. The paper called for domain-specific metrics and human evaluation as supplements to BLEU.

  The dialogue catastrophe (Liu et al., 2016): Perhaps the most dramatic BLEU failure was documented for open-domain dialogue. Liu et al. demonstrated that BLEU has near-zero correlation with human judgements of dialogue response appropriateness across multiple dialogue datasets. This finding effectively ended BLEU’s use in dialogue evaluation, but the broader lesson — that BLEU’s validity is task-specific and degrades as task diversity increases — was not always generalised.

  The Goodhart’s Law trap: When minimum risk training (MRT) began optimising directly against BLEU scores using reinforcement learning (Shen et al., 2016), the gap between BLEU and human quality became acute. MRT-optimised systems achieved higher BLEU but produced translations that were shorter, more repetitive, and less natural by human standards — a textbook example of a metric being gamed. The lesson reinforced the principle that BLEU should never be used as a direct training objective without careful regularisation and complementary human evaluation.

  Contemporary consensus (2025): The field has reached a rough consensus that BLEU is a necessary legacy metric but an insufficient standalone quality signal. Current best practice: report SacreBLEU with version string for backward compatibility, report COMET-22 as primary quality signal for MT, report BERTScore or BLEURT for generation tasks where COMET models are unavailable, and supplement all automatic metrics with human evaluation or LLM-judge preference data for high-stakes claims.

  BLEU Score Across Languages: Performance Variation

  BLEU scores vary dramatically across language pairs, reflecting differences in linguistic typology, resource availability, and reference diversity:

  High-resource European pairs (BLEU-4 SacreBLEU, WMT24 newstest):

• English → French: 35–45 (high vocabulary overlap with English, mature NMT)

• English → German: 28–35 (morphologically richer, complex compounding)

• English → Spanish: 38–48 (related vocabulary, well-resourced)

• English → Czech: 18–25 (morphologically complex, Slavic syntax)

  Distant/morphologically rich pairs:

• English → Japanese: 12–20 (different scripts, complex morphology, SOV word order)

• English → Turkish: 10–18 (agglutinative morphology, very different word order)

• English → Finnish: 14–22 (agglutinative, 15 grammatical cases)

• English → Arabic: 15–23 (right-to-left, rich morphology, orthographic variation)

  Low-resource pairs (FLORES-200):

• English → Lao: 2–8 (very limited parallel data)

• English → Yoruba: 4–10 (few resources, tonal language)

• English → Swahili: 20–30 (better resourced for African languages)

  The variation in BLEU across pairs means that a BLEU-4 score of 25 is excellent for English→Japanese but mediocre for English→French. Cross-pair BLEU comparisons are therefore meaningless — a common mistake in non-specialist reporting of MT results.

  Script and tokenisation effects: Tokenisation choices can change BLEU scores by 2–5 points for the same hypothesis and reference. Chinese and Japanese require character-level or word-boundary segmentation; Arabic benefits from morphological normalisation before scoring; languages with extensive orthographic variation (dialectal Arabic, medieval texts) produce unreliable BLEU scores regardless of tokenisation. SacreBLEU’s language-specific tokenisation options (13a, flores101, char, spm) partially address these issues, but cross-system comparison remains hazardous without identical tokenisation.

  WMT Shared Task History and BLEU’s Role

  The Workshop on Machine Translation (WMT) has been the primary annual venue for MT system comparison since 2006, and its history traces the rise and partial decline of BLEU as a primary ranking metric:

• WMT 2006–2010: Phrase-based SMT systems dominate; BLEU is primary ranking metric; Moses (Koehn et al., 2007) is the standard open-source system; BLEU improvements of 1–2 points per year are typical.

• WMT 2011–2013: Human evaluation (relative ranking) is introduced alongside BLEU; first evidence of systematic disagreement between BLEU ranking and human ranking for specific language pairs.

• WMT 2014: Neural MT (attention-based seq2seq) appears; WMT 2014 En-De newstest2014 becomes the canonical NMT benchmark; human evaluators prefer NMT at lower BLEU scores than competitive SMT.

• WMT 2016: WMT introduces human direct assessment (DA) as primary human evaluation protocol; corpus-level Pearson correlation with DA becomes the standard for metric evaluation.

• WMT 2017–2019: Neural MT systems dominate; BLEU scores plateau on high-resource pairs (optimal n-gram overlap reached given reference translation variability); COMET and BLEURT introduced.

• WMT 2020–2022: Metrics shared task shows COMET and BLEURT substantially outperforming BLEU in segment-level correlation with human judgements; MQM (Multidimensional Quality Metrics) introduced for high-resource pairs as more reliable human ground truth.

• WMT 2023–2025: COMET-22 is the de facto primary metric; ESA (Error Span Annotation) expands MQM collection; BLEU is reported as a secondary metric; LLM-based systems (GPT-4, Claude) enter and perform well by human judgement but not always by BLEU.

  Domain-Specific BLEU Variants and Extensions

  The n-gram precision framework that defines BLEU has been extended and adapted to specific domains and tasks:

  CodeBLEU (Ren et al., 2020): Extends BLEU for code generation evaluation by adding:

• Abstract Syntax Tree (AST) match: n-gram overlap over AST node sequences, capturing structural code similarity beyond token surface match

• Data-flow graph match: n-gram overlap over data-flow graph sequences, capturing semantic code behaviour The combined CodeBLEU score is a weighted sum of token BLEU, keyword BLEU, AST match, and data-flow match. Used in code generation benchmarks (HumanEval, MBPP, CodeXGLUE).

  Character-level BLEU: Using character n-grams rather than word tokens. Particularly effective for morphologically rich languages where word-level tokenisation produces sparse vocabulary. Character BLEU is often computed alongside word BLEU for language pairs involving Arabic, Finnish, Turkish, or agglutinative languages.

  CIDEr (Consensus-Based Image Description Evaluation, Vedantam et al., 2015): A BLEU-derived metric for image captioning that weights n-grams by their TF-IDF scores across the reference corpus, down-weighting common n-grams and up-weighting rare, informative n-grams. Standard metric for COCO image captioning benchmark alongside BLEU-4 and METEOR.

  BLEU for speech: Applied to automatic speech recognition (ASR) output evaluation as a complement to Word Error Rate (WER). Used in spoken language translation (SLT) to evaluate end-to-end systems that produce text output from speech input.

  Multilingual BLEU: Applied to multilingual MT evaluation across 100–200 language pairs simultaneously. FLORES-200 and similar benchmarks compute BLEU for all language pairs to produce aggregate coverage statistics. The variation in BLEU across language pairs (from below 5 for rare pairs to above 40 for high-resource pairs) reflects differences in resource availability, language pair distance, and model coverage.

  Relationship to Statistical MT and the BLEU Optimisation Era

  BLEU’s introduction coincided precisely with the rise of statistical phrase-based MT (Koehn et al., 2003), and the two co-evolved in ways that shaped a decade of MT research:

  The central architecture of phrase-based SMT is the log-linear combination of feature functions (phrase translation probabilities, language model probability, distortion penalty, word count penalty) weighted by parameters λ. Minimum Error Rate Training (MERT, Och 2003) optimised these weights directly against corpus-level BLEU score via coordinate ascent, iteratively adjusting each weight while holding others fixed to maximise BLEU on a development set. This direct coupling of optimisation objective and evaluation metric was unprecedented in NLP and produced dramatic improvements in phrase-based SMT quality through 2010.

  The problem with this coupling, which became clear retrospectively, is that MERT produces parameters that are optimal for BLEU but may not generalise to other quality dimensions. Systems trained with MERT produce translations that match reference n-gram sequences closely but may sacrifice naturalness, coverage of source content, and adequacy of specific phrases in ways that human evaluators notice but BLEU does not. When neural MT arrived, it did not use MERT; instead, neural MT systems were trained with cross-entropy loss against reference translations, optimising a fundamentally different objective that produced more fluent translations at similar or lower BLEU scores.

  The transition from MERT to neural training thus involved a deliberate decoupling of training objective and evaluation metric — training against cross-entropy and evaluating with BLEU. This decoupling, while pragmatic, highlighted that BLEU was never a truly reliable training signal and should be treated as an approximate proxy for human quality rather than a ground truth objective.

  Minimum Risk Training (MRT, Shen et al., 2016) attempted to reintroduce BLEU as a training signal for neural MT using REINFORCE-style gradient estimation, producing systems with higher BLEU at the cost of reduced diversity and naturalness. The lesson drawn by the field was that BLEU is better used for evaluation than optimisation — a principle now widely accepted.

  Connections to Other Ontology Concepts

  BLEU Score’s position in this knowledge graph situates it within a network of evaluation, NLP, and AI concepts:

• Machine Translation: BLEU was designed for and is primarily used to evaluate machine translation output. Every significant MT research paper since 2002 reports BLEU; WMT shared tasks have used BLEU as primary ranking metric for over a decade. Understanding BLEU requires understanding the MT task and the nature of reference translations.

• COMET Metric: The primary learned alternative to BLEU in MT evaluation. COMET uses XLM-RoBERTa to regress human quality annotations, producing scores that better predict actual translation quality. BLEU and COMET frequently disagree in their ranking of systems, with COMET typically more reliable. This relationship — BLEU as baseline, COMET as primary metric — defines the current state of the MT evaluation field.

• BERTScore: The BERT-based alternative to BLEU for text generation evaluation more broadly. BERTScore computes pairwise cosine similarities between contextual BERT embeddings of hypothesis and reference tokens, capturing semantic equivalence that BLEU misses. BERTScore (Zhang et al., 2020) demonstrated substantially better correlation with human judgements than BLEU across MT, summarisation, and image captioning tasks.

• Evaluation Metric: BLEU is the canonical exemplar of an n-gram overlap evaluation metric — the class of metrics that measure string-level similarity between hypothesis and reference without semantic modelling. This positions it in the broader taxonomy of evaluation metrics alongside ROUGE (for summarisation), METEOR (word alignment-based), chrF (character-level), and CIDEr (image captioning).

• Natural Language Processing: BLEU’s domain is NLP evaluation. It was developed for MT but applied to dialogue, summarisation, image captioning, and code generation, revealing both the utility and the limits of n-gram precision as a quality measure across NLP generation tasks.

• BERT: The relationship between BLEU Score and BERT is bidirectional: BERT (the language model) enabled BERTScore (the metric), which was specifically designed to address BLEU’s limitations. BERT embeddings capture semantic similarity that BLEU cannot, directly addressing BLEU’s synonym insensitivity.

• Evaluation Harness: Modern NLP evaluation frameworks such as EleutherAI LM Evaluation Harness and Hugging Face evaluate include BLEU as one metric within a broader suite. The relationship is one of containment: BLEU is a component of evaluation harnesses rather than a standalone evaluation system.

• Benchmarks: BLEU enables benchmark comparison: without an automated metric, comparing MT systems across labs and over time is impractical. BLEU defined what “benchmark performance” meant for MT from 2002 until approximately 2018, and continues to provide the historical baseline against which all subsequent progress is measured.

• Human Evaluation: The relationship between BLEU and human evaluation is complex. BLEU was designed to approximate human evaluation cheaply; its correlation with human judgements motivated its adoption. But its failures in the neural MT era (NMT systems preferred by humans scoring similarly on BLEU to SMT systems) drove the community back towards human evaluation and learned metrics that predict human scores. Human evaluation remains the gold standard; BLEU is a fallible proxy.

• Text Summarisation: BLEU is applied to summarisation as a secondary metric (ROUGE is primary for summarisation). The n-gram precision formulation transfers directly, but BLEU’s poor recall emphasis makes ROUGE-L (longest common subsequence recall) more informative for summarisation tasks.

• Large Language Models: The LLM era has been the final stress test of BLEU’s adequacy. LLM translations and summaries are qualitatively better by human standards but do not consistently score higher on BLEU, because LLMs produce paraphrastic outputs that differ lexically from references. This disconnect has accelerated the transition to semantic evaluation metrics and LLM-as-judge evaluation paradigms.

• Goodhart’s Law: BLEU is a textbook example of Goodhart’s Law in AI: when a measure (BLEU) becomes the target (for model training, researcher incentives, and benchmark ranking), it ceases to be a good measure (of actual translation quality). Minimum risk training with BLEU reward (Shen et al., 2016) produces systems with high BLEU that are not preferred by human evaluators.

  Practical Computation: Step-by-Step Example

  To ground the abstract formulation in concrete terms, consider a simple example:

  Source (Chinese): “猫坐在垫子上” Reference translation: “The cat sat on the mat” Candidate A: “The cat is on the mat” Candidate B: “On the mat sat the cat” Candidate C: “The cat sat on the mat” (identical to reference)

  Candidate A (“The cat is on the mat”) — 6 tokens Unigram matches (clipped): The(1), cat(1), is(0 — not in reference), on(1), the(already counted, 0), mat(1) = 4/6 Bigram matches: “The cat”(1), “cat is”(0), “is on”(0), “on the”(1), “the mat”(1) = 3/5 Trigram matches: “The cat is”(0), “cat is on”(0), “is on the”(0), “on the mat”(1) = 1/4 4-gram matches: “The cat is on”(0), “cat is on the”(0), “is on the mat”(0) = 0/3 Geometric mean: (4/6 × 3/5 × 1/4 × 0/3)^{1/4} — since 4-gram precision is zero, BLEU = 0 (Note: in practice, smoothing methods add ε to zero precision counts to prevent collapse to zero for sentence-level BLEU; NLTK uses add-one smoothing; corpus-level BLEU typically has non-zero 4-gram precision because rare 4-gram matches appear across many sentences)

  Candidate B (“On the mat sat the cat”) — 6 tokens, same words different order Unigram matches: On(0 — lowercase mismatch aside: “on”(1), “the”(1), “mat”(1), “sat”(1), “the”(already counted: 1), “cat”(1)) = 5/6 Bigram matches: “On the”(1), “the mat”(1), “mat sat”(0), “sat the”(0), “the cat”(1) = 3/5 Trigram matches: “On the mat”(0), “the mat sat”(0), “mat sat the”(0), “sat the cat”(0) = 0/4 4-gram: 0/3 BLEU = 0 (collapsed by zero 4-gram precision at sentence level)

  This example illustrates BLEU’s insensitivity to correct content words in wrong order (Candidate B has all correct words but lower BLEU than Candidate A which has one wrong word in the right order), and the sentence-level collapse problem.

  Corpus-level correction: At corpus level across many sentences, 4-gram precision rarely collapses to exactly zero, because occasional long matching sequences appear across the test set. Corpus-level BLEU-4 is dominated by the geometric mean of all four n-gram precisions, making unigram and bigram precision less decisive relative to 3-gram and 4-gram precision. The brevity penalty rarely applies if the system generates approximately reference-length outputs.

  Computing BLEU in Practice (SacreBLEU)

  The standard command-line usage of SacreBLEU for MT evaluation follows this pattern:

# Install
pip install sacrebleu
 
# Compute BLEU on WMT14 En-De (Newstest2014), system output in output.txt
sacrebleu -t wmt14 -l en-de < output.txt
 
# Output includes version string:
# BLEU+case.mixed+lang.en-de+numrefs.1+smooth.exp+test.wmt14/full+tok.13a+version.2.3.1 = 27.3 ...

The version string encodes: case handling (mixed = case-sensitive), language pair, number of references, smoothing method, test set, tokenisation, and sacrebleu version. This string uniquely identifies all computation parameters, enabling exact reproduction by any researcher who runs the same command.

Key SacreBLEU parameters:

• --tokenize 13a: Standard WMT tokenisation (default for European languages)

• --tokenize spm: SentencePiece tokenisation (default for morphologically rich languages)

• --tokenize char: Character-level tokenisation

• --smooth exp: Exponential smoothing for sentence-level scores

• --width 1: Reports BLEU to 1 decimal place (standard for WMT papers)

  BLEU and Low-Resource Language AI in the UK Context

  Within the specific context of the United Kingdom and Northern England, BLEU Score evaluation has practical implications that extend beyond academic benchmarking:

• Welsh language MT (Cymraeg): The Welsh Government has invested in MT for Welsh-English bilingual public services. Systems are evaluated partly using BLEU on test sets derived from official bilingual documents (Senedd Cymru proceedings, official Welsh Government publications). BLEU is one of the available metrics but is supplemented by Welsh-language expert evaluation because no COMET model trained on Welsh DA scores exists. The University of Wales Trinity Saint David and Bangor University contribute to Welsh NLP evaluation research.

• Scottish Gaelic: Even lower-resourced than Welsh, with very limited parallel corpus for MT. BLEU scores for Gaelic translation are typically in the 8–18 range even for the best systems. The University of Edinburgh’s Celtic language NLP work has produced evaluation sets and baselines for Gaelic MT.

• South Asian language pairs for NHS: NHS trusts in Bradford, Leicester, Birmingham, and Tower Hamlets serve populations with high proportions of Urdu, Punjabi, Gujarati, and Bengali speakers. MT systems for these language pairs are evaluated partly with BLEU, but clinical accuracy requirements mandate human expert evaluation as the primary quality gate — a context where BLEU’s semantic adequacy failures are directly safety-relevant.

• Edinburgh’s WMT contributions: The University of Edinburgh MT group has been one of the most active participants in WMT shared tasks for over a decade, submitting top-ranked systems for multiple language pairs and contributing to the evaluation infrastructure that assigns BLEU as a baseline metric.

• Unbabel UK: Unbabel, which develops the COMET metric family and provides quality-assured MT services, operates in the UK market and maintains a significant engineering presence that bridges academic BLEU-based evaluation and production COMET-based quality assurance.

  BLEU in the Context of Broader AI Evaluation Methodology

  The history and critique of BLEU Score provides a case study for broader methodological questions in AI evaluation that extend well beyond machine translation:

  What does it mean for a metric to be valid? The psychometric literature distinguishes face validity (does the metric appear to measure what it claims?), content validity (does it cover all aspects of the construct?), criterion validity (does it correlate with external measures of the construct?), and construct validity (does it behave as a theoretically coherent construct?). BLEU has reasonable face validity (n-gram overlap is intuitively related to translation quality) but poor construct validity (it does not distinguish semantically equivalent translations from genuinely wrong ones) and declining criterion validity in the neural MT era.

  Metric saturation and benchmark inflation: BLEU scores on standard benchmarks have increased monotonically as MT systems improved, but this increase reflects a mix of genuine quality improvement and BLEU gaming. When WMT introduced human direct assessment as the gold standard, some systems ranked highest by human judges scored lower on BLEU than systems ranked below them, revealing systematic saturation of what BLEU can detect in competitive system comparisons.

  The Goodhart cascade in AI research: BLEU’s dominance in MT evaluation created institutional incentives to optimise for BLEU at the expense of other quality dimensions. Research groups tuned hyperparameters for BLEU on development sets; funding agencies assessed progress by BLEU improvements; papers were more likely to be accepted if they showed BLEU gains. This created a feedback loop where the community’s research agenda was implicitly shaped by what BLEU rewards, potentially neglecting quality dimensions BLEU ignores (adequacy, terminology accuracy, factual fidelity).

  Lessons for LLM evaluation: The BLEU experience informs the design of LLM evaluation metrics. The field has broadly accepted that single-number automated metrics are insufficient for complex generation tasks; that human evaluation remains the gold standard; that learned metrics (trained to predict human scores) outperform surface metrics; and that evaluation suites covering multiple task types, quality dimensions, and demographic groups are needed to detect capability and failure modes that single metrics miss.

  Standards and Governance

  BLEU intersects with AI evaluation standards in several ways:

• SacreBLEU (Post, 2018; GitHub: mjpost/sacrebleu): The standardised implementation is now the only acceptable form of BLEU in serious research. The SacreBLEU PyPI package (over 1 million downloads) and its version string reporting mechanism have become de facto standards for MT evaluation reproducibility. Any paper reporting BLEU without using SacreBLEU is now considered non-reproducible.

• WMT Shared Tasks: The Workshop on Machine Translation (annual, 2006–present) has been the primary standardisation body for MT evaluation protocols, including BLEU computation methodology. WMT test sets and reference translations are managed by the WMT organisers and distributed via SacreBLEU’s test set registry, ensuring consistent evaluation conditions across labs.

• ISO/IEC 22989:2022 (AI Concepts and Terminology): While BLEU itself is not defined in this standard, the evaluation metric category and the requirements for documented evaluation methodology it prescribes apply to BLEU-based evaluation in AI system documentation.

• EU AI Act (2024): For high-risk AI systems that include MT components (e.g., systems processing medical or legal documents across language boundaries), the EU AI Act requires documented evidence of performance evaluation under defined conditions. BLEU scores on standard benchmarks are a form of such evidence, though the Act does not prescribe specific metrics and learned metrics like COMET are increasingly preferred for their stronger human correlation.

• NIST Machine Translation Evaluations: NIST (National Institute of Standards and Technology) has organised MT evaluation campaigns (NIST MT) since the early 2000s, using BLEU as the primary metric. These evaluations established much of the infrastructure for reproducible MT benchmarking that later influenced WMT and SacreBLEU.

• ACL/EMNLP/NAACL publication standards: Since approximately 2020, major NLP venues have encouraged or required reporting SacreBLEU with version strings. ACL 2021 introduced a reproducibility checklist that includes metric specification as a mandatory item.

  Relationship to the Ontology

  Within this knowledge graph, BLEU Score serves as a connecting node between the evaluation methodology cluster and the machine translation and NLP application clusters:

• As a subclass of Evaluation Metric, it inherits the general properties of quantitative AI evaluation including ground truth requirements, test set dependencies, and model comparison capabilities.

• As a metric for Machine Translation, it is closely linked to the broader NLP task cluster including Natural Language Processing, Sequence-to-Sequence Model, Attention Mechanism, and Transformer Architecture — the systems BLEU was designed to evaluate.

• As a contrasting concept to COMET Metric and BERTScore, it anchors a comparison cluster of MT evaluation approaches across different generations and methodological paradigms (n-gram overlap vs. learned neural metrics).

• As a historically significant tool that shaped the field, it connects to Benchmarks and Evaluation benchmarks and leaderboards within the broader benchmark and evaluation cluster.

• Its relationship to Goodhart’s Law and Benchmark Standard positions it as a concept that illustrates meta-level epistemological challenges in AI evaluation — relevant to AI Benchmark Epistemological Critique and AI Governance nodes in the graph.

• Its connection to BERT is bidirectional: BERT embeddings underpin BERTScore which was explicitly designed to replace BLEU, while BLEU’s limitations motivated the development of BERT-based learned metrics as the successor evaluation paradigm.

Provenance