Roadmaps for risk mitigation
  • Risk mitigation roadmaps
  • Mitigation Roadmaps
    • Improving generalization through model validation
      • Step 1: Estimating generalization
      • Step 2: Model validation for hyperparameters tuning
      • Step 3: Performing algorithmic selection
      • Additional Material
    • Hyperparameter Optimisation
      • Step 1: Validation
      • Step 2: Hyperparameter Search
      • Additional Considerations
    • Handling dataset shift
      • Step 1: Understanding dataset shifts
      • Step 2: Detecting dataset shifts
      • Step 3: Handling dataset shifts
      • Additional Material
    • Adversarial training for robustness
      • Step 1: Understanding adversarial examples
      • Step 2: Finding adversarial examples
      • Step 3: Defending against adversarial examples
      • Additional Material
    • Data Minimization techniques
      • Step 1: Understanding the data minimization principle
      • Step 2: Data minimization techniques for Supervised Learning
        • Option 1: Reducing features
        • Option 2: Reducing data points
      • Step 3: Other privacy-preserving techniques
      • Additional Material
    • Measuring Bias and Discrimination
      • Step 1: Understanding bias
      • Step 2A: Measuring Bias for Classification tasks
        • Equality of Outcome metrics
        • Equality of Opportunity metrics
      • Step 2B: Measuring Bias in Regression tasks
        • Equality of Outcome metrics
        • Equality of Opportunity metrics
      • Additional Material
    • Mitigating Bias and Discrimination
      • Step 1: Understanding bias
      • Step 2: Mitigating Bias
        • Option 1: Pre-processing
        • Option 2: In-processing
        • Option 3: Post-Processing
      • Additional Material
    • Documentation for improved explainability of Machine Learning models
      • Step 1: Datasheets for Datasets
      • Step 2: Model Cards for Model Reporting
      • Additional Material
    • Extracting Explanations from Machine Learning Models
      • Step 1: Understanding algorithmic explainability
      • Step 2: In-processing methodologies for Explainability
      • Step 3: Post-processing methodologies for Explainability
      • Additional Material
Powered by GitBook
On this page
  1. Mitigation Roadmaps
  2. Handling dataset shift

Step 2: Detecting dataset shifts

PreviousStep 1: Understanding dataset shiftsNextStep 3: Handling dataset shifts

Last updated 3 years ago

In order to make sure that your algorithm does not break or fail suddenly once in production, it is essential to monitor the behaviour of the system and detect dataset shifts.

There is a number of statistical properties that you may be interested in calculating and monitor over time, including:

  • The performance of the model (e.g. accuracy). A decay in performance would indicate model decay. It is advisable to have alerts in case this happens.

  • Performance-related metrics like classification confidence. If these vary a lot it may indicate concept drift.

  • Specific features and their relationship to the target variable. If you suspect that these features are particularly vulnerable to drift, it may be relevant to see how these change over time.

  • Statistical tests can help you measure how much the distribution of the training data differs from the current data. Some of these tests are:

    • the population stability index (PSI)

    • kolmogorov-smirnov statistic (KS)

    • histogram intersection

  • is suitable for more complex applications, such as computer vision. You can test the likelihood that a new data point comes from the original distribution.

  • Data points of adjacent time windows can also be used to detect concept drift. The package offers various types of algorithms, including the ADWIN algorithm implemented in our example notebook below.

Monitoring could include dashboards, ad-hoc analysis and investigations, or calculations implemented directly in your system and paired with alerts for unexpected behaviors. You can find an implementation of various techniques for monitoring dataset shift in our notebook, which can be accessed or downloaded as a file:

Novelty Detection
scikit-multiflow
here
173KB
Dataset_shift.ipynb