Statistical testing is the practice of using sample data to assess evidence for or against a hypothesis about a population or process. It formalises a null and alternative hypothesis, computes a test statistic and an associated significance level, and decides whether observed effects are likely to be genuine rather than due to chance. It underpins rigorous evaluation of models, experiments and measurements.
Overview
- A test posits a null hypothesis representing no effect, then asks how surprising the observed data would be if the null were true.
- The significance level controls the rate of false positives, while statistical power governs the chance of detecting a true effect.
- Frequentist testing complements Bayesian approaches, which instead update beliefs as probabilities.
Key aspects
- Hypothesis formulation and choice of an appropriate test statistic.
- Significance levels, p-values and confidence intervals.
- Power analysis and sample-size planning under Experimental Design.
- Corrections for multiple comparisons to control false discoveries.
Applications
- Validating model improvements during Model Evaluation and benchmarking.
- Online experimentation and A-B Testing of product changes.
- Quality control and scientific measurement under Uncertainty Quantification.