Evaluation Metric

An evaluation metric is a quantitative or qualitative measure used to assess the performance, quality, or behaviour of a machine learning model, algorithm, or system against a defined objective. Metrics are computed over held-out test data or through human judgement protocols and provide the empirical basis for model comparison, selection, and deployment decisions. The choice of metric directly shapes what properties a model optimises for during training and what trade-offs are made between competing objectives such as accuracy, fairness, and calibration. Metric selection is therefore a first-class design decision whose consequences cascade from training dynamics through to safety, governance, and societal impact.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:Accuracy))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:Precision))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:Recall))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:F1Score))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:AUC_ROC))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:FairnessMetric))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:CalibrationMeasure))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:hasPart ai:SemanticSimilarityMetric))

Dependency Relationships

SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:requires ai:GroundTruthLabel))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:requires ai:TestDataset))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:requires ai:EvaluationProtocol))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:dependsOn ai:SupervisedLearning))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:dependsOn ai:StatisticalSignificanceTesting))

Capability Relationships

SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:enables ai:ModelComparison))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:enables ai:HyperparameterTuning))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:enables ai:ModelSelection))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:enables ai:ProductionMonitoring))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:enables ai:RegulatoryCompliance))

Implementation Relationships

SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:supports ai:AISafety))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:supports ai:AIGovernance))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:supports ai:ResponsibleAI))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:implements ai:ScientificReproducibility))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:standardizedBy ai:ISO_IEC_22989))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:standardizedBy ai:NIST_AI_RMF))

Reduction Relationships

SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:ScalarPerformanceMeasure))
SubClassOf(ai:ClassificationMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:ConfusionMatrixEntry))
SubClassOf(ai:RegressionMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:PredictionError))
SubClassOf(ai:TextGenerationMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:NgramOverlapScore))
SubClassOf(ai:FairnessMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:DemographicParityDifference))
SubClassOf(ai:CalibrationMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:ExpectedCalibrationError))
SubClassOf(ai:RankingMetric
  ObjectSomeValuesFrom(ai:reducesTo ai:DiscountedCumulativeGain))

Formal Properties

AnnotationAssertion(rdfs:comment ai:EvaluationMetric "EvaluationMetric instances are not required to be differentiable; contrast with LossFunction which must be differentiable for gradient descent.")
AnnotationAssertion(rdfs:comment ai:EvaluationMetric "The relationship log p(x) = ELBO + KL(q||p) implies that EvaluationMetric can diagnose LossFunctionMisalignment when metric and loss diverge.")
DisjointClasses(ai:EvaluationMetric ai:LossFunction)
SubClassOf(ai:BenchmarkSuite
  ObjectAllValuesFrom(ai:hasComponent ai:EvaluationMetric))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:assessedVia ai:StatisticalSignificanceTest))
SubClassOf(ai:HighRiskAISystem
  ObjectSomeValuesFrom(ai:mustReportOn ai:EvaluationMetric))
SubClassOf(ai:EvaluationMetric
  ObjectSomeValuesFrom(ai:contrastsWith ai:LossFunction))

About

Evaluation metrics are the empirical currency of machine learning: without them, model comparisons are unfalsifiable and scientific progress cannot be tracked. The concept has deep roots in classical statistics — precision and recall originate in information retrieval research of the 1950s and 1960s (Cleverdon, 1960; van Rijsbergen, 1979) — and was operationalised into the machine learning mainstream through the development of standardised benchmark datasets and associated scoring protocols in the 1990s and 2000s. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC), launched in 2010, demonstrated how a single well-designed metric suite combined with an open dataset could catalyse a decade of dramatic progress in Computer Vision; the 2012 entry of AlexNet, which halved the top-5 error rate, is conventionally marked as the beginning of the modern deep learning era.

The relationship between evaluation metrics and training objectives is crucial and widely misunderstood. A loss function must be differentiable and is evaluated on every training example; an evaluation metric is computed post-hoc on discrete predictions from a held-out set and need not be differentiable. The two encode complementary information: cross-entropy loss guides gradient flow through the model’s parameter space, while F1 Score or AUC-ROC summarise the deployment-relevant consequences of those parameters. When the two are well-aligned — as in models trained with surrogate losses that closely approximate the evaluation metric — training converges toward the desired behaviour. When they diverge, models learn to game the proxy rather than improving genuine quality, a manifestation of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” This dynamic drives demand for continuously refreshed benchmark datasets, adversarial probing, and contamination detection, particularly urgent in large language model evaluation where enormous training corpora may include test data from publicly available benchmarks.

The metric landscape has fragmented along task lines. For classification tasks, the confusion-matrix family — Accuracy, Precision, Recall, F1 Score, Matthews Correlation Coefficient, AUC-ROC — remains canonical. Regression tasks rely on mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), and R². For Natural Language Processing, n-gram overlap metrics (BLEU, ROUGE, METEOR) dominated for two decades before the advent of model-based semantic similarity metrics (BERTScore, MoverScore) that better correlate with human judgement. For generative image models, Fréchet Inception Distance (FID) and Inception Score (IS) measure distributional quality in the embedding space of an Inception Network. For ranking and Information Retrieval, Mean Reciprocal Rank (MRR), Normalised Discounted Cumulative Gain (NDCG), and precision-at-K are standard. Each family reflects a different notion of what it means for a model to be “correct,” and the choice between them encodes consequential assumptions about task priorities and acceptable trade-offs.

Metric Families and Key Instances

Classification Metrics

The confusion matrix provides the foundational data structure for binary and multiclass classification evaluation, recording true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for each class.

• Accuracy — (TP + TN) / (TP + TN + FP + FN). Intuitive but misleading under class imbalance: a model predicting the majority class always achieves high accuracy on a dataset where one class comprises 95% of examples.

• Precision — TP / (TP + FP). Fraction of positive predictions that are genuinely positive; relevant when false positives are costly (e.g., spam filtering, medical screening where unnecessary treatment carries risk).

• Recall (Sensitivity, True Positive Rate) — TP / (TP + FN). Fraction of actual positives correctly retrieved; critical when false negatives are costly (e.g., cancer detection, fraud detection).

• Specificity — TN / (TN + FP). Complement of false positive rate; important in medical testing where healthy patients must not be incorrectly flagged.

• F1 Score — 2 × (Precision × Recall) / (Precision + Recall). Harmonic mean that balances precision and recall; preferred summary statistic for imbalanced classification. The F-beta generalisation weights recall beta times more than precision.

• Matthews Correlation Coefficient (MCC) — balanced measure treating all four confusion matrix cells symmetrically; outperforms F1 on highly imbalanced binary classification.

• AUC-ROC — Area under the Receiver Operating Characteristic curve, which plots true positive rate against false positive rate across all decision thresholds. Threshold-independent; equals 0.5 for a random classifier and 1.0 for a perfect one. Widely used for ranking and binary classification.

• PR-AUC (Precision-Recall AUC) — area under the precision-recall curve; preferred over AUC-ROC when positive class prevalence is low.

  Regression Metrics

• Mean Absolute Error (MAE) — average absolute deviation between predictions and targets; robust to outliers.

• Mean Squared Error (MSE) — average squared deviation; penalises large errors quadratically and is differentiable, making it useful as a Loss Function.

• Root Mean Squared Error (RMSE) — square root of MSE; expressed in the same units as the target variable.

• Mean Absolute Percentage Error (MAPE) — percentage-scale MAE; useful for heteroscedastic targets but undefined when targets contain zeros.

• R-squared (R²) — proportion of variance in the target explained by the model; scale-independent, though can be negative for poorly fitted models.

• Spearman Rank Correlation — rank-based correlation; useful when the ordering of predictions matters more than their magnitude.

  Natural Language Generation Metrics

• BLEU (Bilingual Evaluation Understudy) — n-gram precision against one or more reference translations with a brevity penalty. Standard for Machine Translation since Papineni et al. (2002). Criticised for insensitivity to meaning, paraphrase, and word order beyond n-gram windows.

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation) — recall-oriented n-gram overlap; standard for Text Summarisation. Variants: ROUGE-N (n-gram), ROUGE-L (longest common subsequence), ROUGE-W (weighted LCS).

• METEOR — alignment-based metric incorporating stemming, synonymy, and paraphrase matching; higher correlation with human judgement than BLEU.

• ChrF — character n-gram F-score; more robust to morphologically rich languages than word-level metrics.

• BERTScore — token-level semantic similarity computed by cosine similarity between contextual embeddings from a pre-trained BERT model. Superior correlation with human judgements of text quality and increasingly adopted in generation evaluation. Computationally expensive; sensitivity to embedding model choice creates fairness concerns across demographic subgroups.

• Perplexity — exponentiated average negative log-likelihood per token under a Language Model; measures how well the model predicts held-out text. Lower perplexity indicates better predictive compression. Standard for language modelling but not directly comparable across different tokenisations.

• BLEURT — learned evaluation metric fine-tuned to predict human quality ratings; trained on synthetic data from mT5 and BLEURT-20 checkpoints.

  Generative Image and Multimodal Metrics

• Fréchet Inception Distance (FID) — compares the distribution of real and generated images in the feature space of an Inception-v3 network using the Fréchet distance between fitted Gaussians. Lower FID indicates better perceptual quality and diversity. Standard for generative image evaluation.

• Inception Score (IS) — measures quality (high confidence predictions) and diversity (high marginal entropy) of generated images using an Inception classifier. Largely superseded by FID.

• CLIP Score — cosine similarity between CLIP image and text embeddings; evaluates text-image alignment in text-to-image generation. Standard for diffusion model evaluation.

• LPIPS (Learned Perceptual Image Patch Similarity) — perceptual similarity metric based on deep feature differences; correlates with human judgements of image similarity better than pixel-level MSE.

  Fairness and Safety Metrics

• Fairness Metrics — demographic parity difference (equal positive prediction rates across groups), equalised odds (equal TPR and FPR across groups), disparate impact ratio, and counterfactual fairness. Essential for bias audits under EU AI Act high-risk system requirements.

• Calibration — alignment between predicted probabilities and empirical frequencies. Measured by Expected Calibration Error (ECE) and Maximum Calibration Error (MCE); visualised via reliability diagrams. Well-calibrated models are essential for decision support and uncertainty-aware applications.

• Adversarial Robustness Score — degradation in performance metric under adversarial perturbations (FGSM, PGD attacks); quantifies vulnerability to distribution shift and adversarial manipulation.

• Worst-Group Accuracy — minimum accuracy across demographically or contextually defined subgroups; emphasises performance on under-represented populations.

  Holistic Benchmark Suites

  Benchmark suites aggregate multiple tasks and metrics into unified evaluation frameworks:

• GLUE / SuperGLUE — Natural language understanding benchmarks that served as canonical NLP evaluation through 2019-2021; largely saturated by frontier models (GPT-4 class) by 2022.

• MMLU / MMLU-Pro — Massive Multitask Language Understanding; covers 57 academic subjects. Frontier models including Gemini 3 Pro and Claude Opus 4.5 approach 90% on MMLU-Pro (2026), indicating saturation for the hardest modern models.

• HELM (Holistic Evaluation of Language Models) — Stanford multi-metric evaluation framework covering accuracy, calibration, robustness, fairness, efficiency, and disinformation potential across 42 scenarios.

• BIG-Bench and BIG-Bench Hard — community-contributed tasks targeting model capabilities that prior benchmarks under-measured; focus on reasoning under uncertainty, social reasoning, and out-of-distribution generalisation.

• Humanity’s Last Exam — crowdsourced benchmark of expert-level questions in STEM, humanities, and law; designed to maintain headroom after MMLU saturation.

• MLPerf — industry benchmark for training and inference throughput across hardware and model types; provides reproducible performance metrics for systems evaluation.

  Metric Computation Pipeline

  The standard evaluation workflow involves five stages:

  1. Dataset preparation: Hold out a representative test split from training and validation data. For Large Language Model evaluation, verify the test set is absent from pretraining corpora using n-gram deduplication tools or membership inference attacks.
  2. Prediction generation: Run the model on each test example to produce outputs — predicted class labels, probability distributions, generated text, or bounding boxes depending on the task.
  3. Metric computation: Apply the scoring function(s) to compare predictions against ground truth. For aggregate metrics (e.g., macro-averaged F1), compute per-class values then combine according to the averaging scheme.
  4. Statistical analysis: Report confidence intervals (bootstrap or t-test) and significance tests comparing models. A performance difference is scientifically meaningful only if it is statistically significant and practically relevant in magnitude.
  5. Disaggregated analysis: Compute metrics separately across demographic subgroups, domains, and difficulty levels to surface performance disparities that aggregate metrics conceal.

  Human Evaluation Protocols

  For tasks without a unique correct output — dialogue, creative writing, visual question answering, code generation — automatic metrics are supplemented or replaced by structured human rating:

• Likert scale ratings: Raters assess quality dimensions (fluency, coherence, factual accuracy, helpfulness) on a 1-5 or 1-7 scale.

• Pairwise preference: A and B outputs are compared head-to-head; win rate against a reference model is reported. Used in RLHF reward model training and model releases.

• RLHF human preference data: Large-scale human preference annotation underlies the reward models used in Reinforcement Learning from Human Feedback, making human evaluation an indirect training signal as well as an evaluation tool.

• Inter-annotator agreement (IAA): Reported alongside human scores as Cohen’s Kappa, Fleiss’s Kappa, or Krippendorff’s Alpha; values below 0.6 suggest the task specification is insufficiently unambiguous to support reliable evaluation.

  Goodhart’s Law and Benchmark Contamination

  Once a metric becomes a public leaderboard target, two pathologies emerge. First, Goodhart’s Law: models are fine-tuned or trained explicitly on benchmark tasks, optimising the proxy at the expense of genuine generalisation. Second, benchmark contamination: test examples may appear in the internet-crawled pretraining corpora of large language models. A 2024 study found that removing contaminated GSM8K examples reduced measured accuracy by 13 percentage points, implying that reported progress was partially artefactual. These dynamics drove the retirement of MMLU, HumanEval, HellaSwag, and original GSM8K as primary evaluation benchmarks and motivated rolling benchmark strategies, private held-out sets maintained by evaluation organisations, and cryptographic commitment schemes that allow model developers to evaluate privately against tests they cannot see.

  Use Cases

  Research Benchmarking

  Academic leaderboards (Papers With Code, Open LLM Leaderboard, Stanford HELM) use standardised metric suites to rank models; the metric choice defines what the community operationally treats as “progress.” Benchmark saturation — frontier models exceeding human performance on a metric — signals the need for harder benchmarks, not that AI capabilities have been fully characterised. The rapid progression from GLUE to SuperGLUE to MMLU to MMLU-Pro to Humanity’s Last Exam illustrates this treadmill.

  Model Selection and Hyperparameter Optimisation

  Validation-set metrics guide Hyperparameter Tuning and architecture search. Bayesian optimisation frameworks (Optuna, Hyperopt), neural architecture search (NAS) algorithms, and automated ML (AutoML) pipelines optimise directly against evaluation metrics, requiring the metric to faithfully represent deployment requirements. Choosing the wrong validation metric during model selection propagates the error through the entire development pipeline.

  Production Monitoring and Data Drift Detection

  After deployment, evaluation metrics are computed continuously on labelled production samples to detect Distributional Shift, model degradation, and data drift. Automated alerting triggers retraining or rollback when metrics drop below operational thresholds. A/B testing frameworks compare champion and challenger model metrics on live traffic before full rollout.

  Regulatory and Governance Compliance

  High-risk AI systems under the EU AI Act (Regulation (EU) 2024/1689) must document accuracy metrics, robustness scores, and fairness gap measurements in technical documentation submitted to conformity assessment bodies. The UK’s sector-led approach mandates sector-specific evaluation requirements through regulators such as the FCA (financial services), CQC (healthcare), and ICO (data protection). Model cards (Mitchell et al., 2019) institutionalised the practice of reporting evaluation metrics alongside dataset conditions and disaggregated results as the foundational responsible AI documentation artefact.

  Academic Context

  Evaluation methodology has its own research sub-field, distinct from the primary task research it supports. Key contributions include:

• Van Rijsbergen (1979): formalised precision and recall in information retrieval; introduced the F-measure as their harmonic mean.

• Papineni et al. (2002): introduced BLEU for machine translation evaluation; established the model of n-gram overlap against human references.

• Lin (2004): introduced ROUGE for summarisation evaluation.

• Liang et al. (2022): introduced HELM as a multi-dimensional evaluation framework emphasising transparency and fairness.

• Zhang et al. (2020): introduced BERTScore, demonstrating that pre-trained language model embeddings yield evaluation metrics that outperform n-gram methods in correlation with human judgement.

• Mitchell et al. (2019): proposed Model Cards for model reporting, formalising the documentation of evaluation conditions, metrics, and disaggregated results.

• Bowman & Dahl (2021): argued that current NLP benchmarks are too narrow and saturate before capturing the full breadth of language understanding.

  The statistical validity of common evaluation practices has come under scrutiny: Dror et al. (2018) showed that many published NLP evaluation comparisons were under-powered and that widely used significance tests were misapplied.

  Current Landscape (2026)

  The 2025-2026 evaluation landscape is characterised by four dominant trends:

  1. LLM-as-judge: Using large language models as automated evaluators of other models’ outputs, replacing or supplementing human raters in tasks like instruction following, reasoning quality, and factual accuracy. GPT-4 and Claude 3 class judges achieve inter-annotator agreement with human raters comparable to agreement between human rater pairs on many text quality tasks, enabling cost-effective large-scale evaluation. Concerns remain about self-preference bias (models rating their own outputs higher) and sensitivity to prompt formatting.

  2. Agentic and multi-step evaluation: As AI systems shift from single-turn generation to multi-step agentic workflows, traditional per-output metrics fail to capture cumulative task completion quality. Frameworks such as AgentBench, SWE-bench, and GAIA evaluate agents over trajectories — sequences of observations, reasoning steps, and actions — with metrics for task completion rate, step efficiency, and error recovery.

  3. Benchmark rotation and contamination control: The AI Security Institute (formerly AI Safety Institute, renamed February 2025) conducts private evaluations of frontier models against non-public test sets as part of the UK government’s pre-deployment safety evaluation regime. The US NIST operates AISIC (AI Safety Institute Consortium) with parallel evaluation programmes. These institutional evaluators maintain rolling private benchmarks inaccessible to model developers to prevent overfitting.

  4. Standardisation initiatives: The EU AI Act’s implementation via ISO/IEC standardisation (ISO/IEC 42001 AI Management Systems; ISO/IEC 22989 AI Concepts and Terminology) is creating harmonised evaluation requirements across EU member states. MLCommons’ MLPerf inference benchmarks define reproducible performance metrics for inference hardware and model efficiency across vision, language, and recommender tasks.

  The Alan Turing Institute’s Fairness, Transparency and Privacy interest group and the UCL UKRI Generative AI hub are central UK academic nodes in evaluation methodology research, with particular focus on fairness metrics, calibration, and the governance dimensions of evaluation choices. DeepMind’s Ethics and Society team (William Isaac and collaborators) has produced foundational work on bias measurement and fairness metric selection for deployed AI systems.

  UK Context

  UK academic contributions to evaluation metric research span several institutions. The Alan Turing Institute, operating as the UK’s national data science and AI institute, has published extensively on AI fairness measurement — its Fairness, Transparency and Privacy interest group brings together researchers from Cambridge, Edinburgh, Oxford, and UCL to develop rigorous metrics for evaluating AI systems against ethical criteria. UCL leads the UKRI generative AI hub and maintains a Google DeepMind academic partnership, situating it at the centre of evaluation methodology for foundation models.

  Imperial College London’s AI research groups have contributed to robustness evaluation and adversarial metrics. The University of Edinburgh’s School of Informatics, through its Alan Turing Institute programme and the Edinburgh Centre for Robotics, works on evaluation of perception and planning systems in autonomous settings. The University of Manchester and the University of Leeds contribute through industrial partnerships in manufacturing quality control, healthcare AI (particularly cancer screening metrics in the Northern Health Science Alliance), and autonomous systems.

  Northern England’s AI cluster, centred on Manchester’s role in the Northern Powerhouse, applies evaluation metrics in precision medicine, industrial inspection, and financial fraud detection contexts. NHS Digital and the NHS AI Lab have driven demand for clinically meaningful evaluation metrics — sensitivity and specificity at clinically relevant operating points, calibration curves for risk stratification tools — that differ substantially from standard ML benchmark metrics.

  Future Directions (2026-2030)

  Several research and governance trajectories are reshaping evaluation metric practice:

• Multi-objective Pareto evaluation: Moving beyond single-metric leaderboards to Pareto-front analysis across accuracy, fairness, robustness, efficiency, and safety simultaneously. Tools like the HELM “metric radar” visualisation point the way toward richer evaluation summaries.

• Dynamic benchmark curation: Adversarial human-AI collaborative benchmark construction (ARC-Challenge, HellaSwag evolved from adversarial filtering) and rolling private benchmark maintenance by institutional evaluators will become standard practice to combat contamination.

• Neuroimaging-inspired metrics: Cognitive science-informed evaluation protocols that test for systematic reasoning abilities, causal inference, and compositional generalisation rather than narrow pattern matching.

• Automated metric learning: Meta-learning approaches that learn evaluation metrics from human preference data without hand-crafting scoring functions; potentially enabling task-adaptive metrics that capture novel quality dimensions.

• Regulatory metric standardisation: As the EU AI Act conformity assessment regime matures post-2026, harmonised metric thresholds for high-risk AI systems across sectors (healthcare, financial services, biometrics) will become mandatory specifications rather than voluntary guidelines.

• Embodied and physical AI evaluation: Evaluation of robotics and physical AI systems operating in complex, partially observable environments requires metrics that capture temporal robustness, safety constraint satisfaction, and generalisation to unseen physical configurations — an open research problem as of 2026.

  Key Terminology

Term	Definition
True Positive (TP)	A case where the model correctly predicts the positive class.
False Positive (FP)	A case where the model incorrectly predicts positive when the true class is negative; a Type I error.
False Negative (FN)	A case where the model misses a true positive; a Type II error.
Precision	TP / (TP + FP); fraction of positive predictions that are correct.
Recall	TP / (TP + FN); fraction of actual positives correctly identified.
F1 Score	Harmonic mean of Precision and Recall; 2·P·R / (P + R).
AUC-ROC	Area under the Receiver Operating Characteristic curve; threshold-independent discriminative power.
Calibration	Agreement between predicted probabilities and empirical frequencies of outcomes.
Benchmark contamination	Test set data present in training corpora, inflating measured performance.
Goodhart’s Law	When a measure becomes a target, it ceases to be a good measure.
Macro-averaging	Compute metric per class, then average equally across classes; treats all classes equally regardless of size.
Micro-averaging	Aggregate TP/FP/FN counts across classes before computing the metric; dominated by frequent classes.
BLEU	Bilingual Evaluation Understudy; n-gram precision metric for machine translation.
FID	Fréchet Inception Distance; distributional similarity for generative image model evaluation.
HELM	Holistic Evaluation of Language Models; multi-scenario, multi-metric LLM benchmark from Stanford.
LLM-as-judge	Using a large language model to evaluate the outputs of another model, replacing or supplementing human raters.
Pareto-front analysis	Multi-objective optimisation framing that identifies the set of configurations where no metric can be improved without degrading another.
Demographic parity	Fairness criterion requiring equal positive prediction rates across demographic groups.
Equalised odds	Fairness criterion requiring equal true positive rate and false positive rate across groups.
Expected Calibration Error (ECE)	Weighted average absolute difference between predicted confidence and empirical accuracy across confidence bins.

Research and Literature

1. Van Rijsbergen, C. J. (1979). Information Retrieval (2nd ed.). Butterworths. — Foundational formulation of precision, recall, and F-measure.
2. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). BLEU: a method for automatic evaluation of machine translation. ACL 2002. https://aclanthology.org/P02-1040
3. Lin, C. Y. (2004). ROUGE: A package for automatic evaluation of summaries. ACL Workshop on Text Summarization Branches Out.
4. Davis, J., & Goadrich, M. (2006). The relationship between Precision-Recall and ROC curves. ICML 2006. https://doi.org/10.1145/1143844.1143874
5. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., & Gebru, T. (2019). Model Cards for Model Reporting. FAccT 2019. https://doi.org/10.1145/3287560.3287596
6. Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in NLP. ACL 2018. https://aclanthology.org/P18-1128
7. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. ICLR 2019. https://openreview.net/forum?id=rJ4km2R5t7
8. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR 2020. https://openreview.net/forum?id=SkeHuCVFDr
9. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS 2017. — Introduced FID. https://papers.nips.cc/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html
10. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., … & Hashimoto, T. (2022). Holistic Evaluation of Language Models. TNLP 2022. https://arxiv.org/abs/2211.09110
11. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., … & Team, B. B. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. https://arxiv.org/abs/2206.04615
12. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. https://arxiv.org/abs/2009.03300
13. Bowman, S. R., & Dahl, G. E. (2021). What will it take to fix benchmarking in natural language understanding? NAACL 2021. https://aclanthology.org/2021.naacl-main.316
14. Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Machine Learning: Limitations and Opportunities. MIT Press. https://fairmlbook.org
15. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017. https://arxiv.org/abs/1706.04599
16. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012). Fairness through awareness. ITCS 2012. https://doi.org/10.1145/2090236.2090255
17. Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. NeurIPS 2016. https://papers.nips.cc/paper/2016/hash/9d2682367c3935defcb1f9e247a97c0d-Abstract.html
18. Corbett-Davies, S., & Goel, S. (2018). The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv:1808.00023. https://arxiv.org/abs/1808.00023
19. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2018). Datasheets for Datasets. Communications of the ACM, 2021. https://doi.org/10.1145/3458723
20. OpenAI (2023). GPT-4 Technical Report. https://arxiv.org/abs/2303.08774
21. Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023). Capabilities of GPT-4 on Medical Challenge Problems. arXiv:2303.13375. https://arxiv.org/abs/2303.13375
22. Theis, L., van den Oord, A., & Bethge, M. (2016). A note on the evaluation of generative models. ICLR 2016. — Seminal analysis of generative model metric pathologies. https://arxiv.org/abs/1511.01844
23. Rajpurkar, P., Jia, R., & Liang, P. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD. ACL 2018. https://aclanthology.org/P18-2124
24. Sellam, T., Das, D., & Parikh, A. P. (2020). BLEURT: Learning Robust Metrics for Text Generation. ACL 2020. https://aclanthology.org/2020.acl-main.704
25. Zheng, L., Chiang, W-L., Sheng, Y., … & Gonzalez, J. E. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://arxiv.org/abs/2306.05685
26. NIST (2023). AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1
27. ISO/IEC 22989:2022. Artificial Intelligence — Concepts and Terminology. International Organization for Standardization.
28. EU Regulation (EU) 2024/1689 (AI Act). Official Journal of the European Union, 12 July 2024.

Provenance