Post Hoc Explanation

Interpretability techniques applied after a machine learning model has been trained, providing explanations for model behaviour and predictions without modifying the model’s architecture or requiring retraining.

Semantic Classification

Content

• Interpretability techniques applied after a machine learning model has been trained, providing explanations for model behaviour and predictions without modifying the model’s architecture or requiring retraining.

  Related Terms

• Broader: Explainable AI, Model Interpretability

• Narrower: SHAP, LIME, Grad-CAM, Saliency Map

• Related: Intrinsic Interpretability, Model-Agnostic Explanations, Feature Attribution

• Contrasts: Intrinsic Interpretability

  Formal Specification

  Core Characteristics

  1. Temporal Independence: Applied after model training completion
  2. Architecture Agnosticism: Typically model-agnostic or adaptable
  3. Non-invasive: Does not alter original model parameters
  4. Explanatory Focus: Describes rather than redesigns decision-making

  Formal Framework

  Given a trained model f: X → Y, a post-hoc explanation method E produces:

E(f, x) → explanation

Where:

• f is the black-box model

• x is the input instance

• explanation is a human-interpretable representation

  Categories

  By Scope

  Local Post-Hoc Methods

  Explain individual predictions:

  LIME (Local Interpretable Model-agnostic Explanations):

  ξ(x) = argmin L(f, g, πₓ) + Ω(g)
     g∈G
  

• Fits interpretable model locally around instance

• Uses perturbation sampling

• Model-agnostic applicability

  SHAP (SHapley Additive exPlanations):

  φᵢ(f, x) = Σ |S|!(|N|-|S|-1)! / |N|! [f(S∪{i}) - f(S)]
         S⊆N\{i}
  

• Game-theoretic feature attribution

• Satisfies desirable properties (local accuracy, missingness, consistency)

• Computationally expensive

  Counterfactual Explanations:

  CF(x) = argmin d(x, x') subject to f(x') ≠ f(x)
      x'∈X
  

• Minimal input changes to alter prediction

• Actionable insights for users

• Causal reasoning support

  Global Post-Hoc Methods

  Explain overall model behaviour:

  Partial Dependence Plots (PDP):

  PDₛ(xₛ) = 𝔼ₓₓ[f(xₛ, Xₓ)]
  

• Marginal effect of features

• Averages over other features

• Assumes feature independence

  Permutation Feature Importance:

  FI(j) = Score(original) - 𝔼[Score(permuted feature j)]
  

• Global feature relevance

• Model-agnostic

• Handles feature interactions

  Surrogate Models:

  g* = argmin L(f(X), g(X))
   g∈G_interpretable
  

• Train interpretable model to mimic black-box

• Global approximation

• Inherent interpretability of surrogate

  By Method Type

  Perturbation-Based

• LIME: Local perturbations with interpretable model fitting

• Permutation Importance: Feature shuffling impact assessment

• Occlusion Sensitivity: Masking input regions (computer vision)

  Gradient-Based

• Integrated Gradients: Path integral of gradients

• Saliency Maps: First-order gradient visualisation

• Grad-CAM: Class activation mapping with gradients

  Decomposition-Based

• Layer-wise Relevance Propagation (LRP): Backpropagation of relevance scores

• DeepLIFT: Difference from reference activation

• SHAP: Shapley value decomposition

  Key Properties

  Desirable Characteristics

  Fidelity:

• Accurate representation of model behaviour

• High correlation with actual model reasoning

• Minimal approximation error

  Stability:

• Consistent explanations for similar inputs

• Robustness to minor perturbations

• Reproducibility across runs

  Comprehensibility:

• Human-understandable format

• Appropriate for target audience

• Actionable insights

  Efficiency:

• Computational feasibility

• Scalability to production systems

• Real-time explanation capability (where needed)

  Trade-offs

  Accuracy vs. Interpretability:

• Complex models require more sophisticated explanations

• Simplified explanations may sacrifice fidelity

• Context-dependent balance required

  Local vs. Global:

• Local methods: High fidelity, limited generalisability

• Global methods: Broader insights, potential inaccuracies

• Complementary use recommended

  Implementation Approaches

  SHAP Implementations

  Kernel SHAP:

• Model-agnostic approximation

• Weighted linear regression

• Slower but universally applicable

  Tree SHAP:

• Optimised for tree ensembles

• Polynomial time complexity

• Exact Shapley values

  Deep SHAP:

• DeepLIFT-based approximation for neural networks

• Faster than kernel SHAP

• Leverages network structure

  LIME Implementation

  Algorithm:

  1. Generate perturbed samples around instance
  2. Weight samples by proximity to original
  3. Train interpretable model (e.g., linear regression)
  4. Extract coefficients as feature importance

  Parameters:

• Kernel width (locality)

• Number of samples

• Feature selection threshold

• Interpretable model type

  Gradient-Based Methods

  Integrated Gradients:

  IG(x)ᵢ = (xᵢ - x'ᵢ) × ∫₀¹ ∂f(x' + α(x - x'))/∂xᵢ dα
  

• Baseline-dependent

• Path integral accumulates gradients

• Satisfies sensitivity and implementation invariance

  Grad-CAM:

  L^c_Grad-CAM = ReLU(Σₖ αₖ^c Aᵏ)
  

• Weighted combination of activation maps

• Visual explanations for CNNs

• Class-discriminative localisation

  Application Domains

  Computer Vision

  Methods:

• Grad-CAM, Grad-CAM++, Score-CAM

• Saliency maps

• Occlusion sensitivity

  Use Cases:

• Medical image diagnosis explanation

• Autonomous vehicle perception

• Security and surveillance analysis

  Natural Language Processing

  Methods:

• Attention visualisation

• LIME for text

• Integrated gradients

  Applications:

• Sentiment analysis justification

• Machine translation quality assessment

• Hate speech detection transparency

  Tabular Data

  Methods:

• SHAP (Kernel, Tree variants)

• LIME

• Partial dependence plots

  Domains:

• Credit scoring (finance)

• Healthcare risk prediction

• Fraud detection

  Standards & Compliance

  IEEE Standards

  IEEE P2976 (XAI):

• Post-hoc methods as XAI implementation

• Partially explainable AI classification

• Optional explainability constraints

  IEEE P2863 (Governance):

• Post-hoc explanations for transparency

• Accountability through interpretability

• Bias detection via explanation analysis

  Regulatory Context

  GDPR Article 22:

• Right to explanation for automated decisions

• Post-hoc methods as compliance mechanism

• Meaningful information requirement

  IEEE P2802 (Medical AI):

• Clinical validation of explanations

• Post-hoc analysis for device transparency

• Safety-critical explanation standards

  Evaluation Metrics

  Quantitative Measures

  Fidelity Metrics:

• Local Accuracy: Agreement with model on explained instance

• Global Fidelity: R² between explanation model and black-box

• Consistency: Stability across similar inputs

  Efficiency Metrics:

• Computation time

• Number of model queries

• Memory requirements

  Robustness Metrics:

• Sensitivity to input perturbations

• Explanation variance across runs

• Adversarial robustness

  Qualitative Assessment

  User Studies:

• Task performance with explanations

• Trust calibration

• Decision-making improvement

  Expert Validation:

• Domain specialist agreement

• Alignment with ground truth (where available)

• Utility for debugging and model improvement

  Challenges & Limitations

  Methodological Challenges

  Approximation Errors:

• LIME may not faithfully represent model in complex regions

• Surrogate models sacrifice accuracy for interpretability

• Sampling-based methods have stochastic variance

  Computational Cost:

• SHAP requires exponential evaluations (exact)

• Real-time constraints in production

• Scalability to high-dimensional inputs

  Instability:

• LIME sensitive to perturbation sampling

• Gradient-based methods affected by saturation

• Explanation variance across runs

  Conceptual Limitations

  Rashomon Effect:

• Multiple valid explanations possible

• No ground truth for explanation correctness

• Context-dependent interpretation

  Oversimplification:

• Complex models may not reduce to simple explanations

• Feature interactions lost in additive models

• Non-linear relationships linearised

  Audience Dependence:

• Technical vs. lay explanations differ

• Cultural interpretation variations

• Expertise-appropriate granularity

  Research Directions

  Emerging Areas

  Causal Post-Hoc Explanations:

• Interventional reasoning

• Counterfactual causal models

• Beyond correlational attribution

  Multi-modal Explanations:

• Cross-modal consistency

• Vision-language explanation alignment

• Unified multimodal frameworks

  Interactive Explanations:

• Dialogue-based refinement

• User-guided exploration

• Personalised explanation generation

  Adversarial Robustness:

• Explanations under adversarial attack

• Robust feature attribution

• Manipulation-resistant methods

  Industry Innovation

  Microsoft InterpretML:

• Unified API for multiple methods

• Performance optimisation

• Production-ready implementations

  Google Cloud Explainable AI:

• Integrated with Vertex AI

• Feature attributions at scale

• What-If Tool for counterfactuals

  SHAP Library (Lundberg):

• Comprehensive implementations

• GPU acceleration

• Interactive visualisations

  Best Practices

  Method Selection

  Decision Tree:

  1. Black-box model type: Neural network → gradient methods; Trees → SHAP TreeExplainer
  2. Explanation scope: Local → LIME/SHAP; Global → PDP/surrogate
  3. Computational budget: Limited → sampling-based; Ample → exact methods
  4. Data modality: Images → CAM variants; Text → attention; Tabular → SHAP/LIME

  Implementation Guidelines

  Pre-deployment:

• Validate explanation fidelity on test set

• Benchmark computational performance

• Test stability across input distribution

• Verify regulatory compliance

  Production:

• Cache explanations where appropriate

• Monitor explanation quality metrics

• Track user engagement with explanations

• A/B test explanation formats

  Post-deployment:

• Collect user feedback

• Refine methods based on utility

• Update as model evolves

• Audit explanation accuracy

  Documentation

  Model Cards should include:

• Post-hoc methods employed

• Fidelity metrics

• Computational requirements

• Known limitations

Provenance