Roadmaps for risk mitigation
  • Risk mitigation roadmaps
  • Mitigation Roadmaps
    • Improving generalization through model validation
      • Step 1: Estimating generalization
      • Step 2: Model validation for hyperparameters tuning
      • Step 3: Performing algorithmic selection
      • Additional Material
    • Hyperparameter Optimisation
      • Step 1: Validation
      • Step 2: Hyperparameter Search
      • Additional Considerations
    • Handling dataset shift
      • Step 1: Understanding dataset shifts
      • Step 2: Detecting dataset shifts
      • Step 3: Handling dataset shifts
      • Additional Material
    • Adversarial training for robustness
      • Step 1: Understanding adversarial examples
      • Step 2: Finding adversarial examples
      • Step 3: Defending against adversarial examples
      • Additional Material
    • Data Minimization techniques
      • Step 1: Understanding the data minimization principle
      • Step 2: Data minimization techniques for Supervised Learning
        • Option 1: Reducing features
        • Option 2: Reducing data points
      • Step 3: Other privacy-preserving techniques
      • Additional Material
    • Measuring Bias and Discrimination
      • Step 1: Understanding bias
      • Step 2A: Measuring Bias for Classification tasks
        • Equality of Outcome metrics
        • Equality of Opportunity metrics
      • Step 2B: Measuring Bias in Regression tasks
        • Equality of Outcome metrics
        • Equality of Opportunity metrics
      • Additional Material
    • Mitigating Bias and Discrimination
      • Step 1: Understanding bias
      • Step 2: Mitigating Bias
        • Option 1: Pre-processing
        • Option 2: In-processing
        • Option 3: Post-Processing
      • Additional Material
    • Documentation for improved explainability of Machine Learning models
      • Step 1: Datasheets for Datasets
      • Step 2: Model Cards for Model Reporting
      • Additional Material
    • Extracting Explanations from Machine Learning Models
      • Step 1: Understanding algorithmic explainability
      • Step 2: In-processing methodologies for Explainability
      • Step 3: Post-processing methodologies for Explainability
      • Additional Material
Powered by GitBook
On this page
  1. Mitigation Roadmaps

Hyperparameter Optimisation

Maximising Model Performance Via Hyperparameter Selection

PreviousAdditional MaterialNextStep 1: Validation

Last updated 3 years ago

Machine learning models comprise two parameter types:

  • Model parameters - These are typically simply referred to as ‘parameters’ and are the parameters that are learned from data via the model training process;

  • Hyperparameters - These describe all the parameters that can be explicitly set by the model owner prior to training. This might comprise elements of the model architecture (e.g. number of hidden layers in a feed-forward neural network) or the learning process (e.g. whether to use Adam optimisation, the strength of the ridge regression penalty term, the learning rate).

The performance of a trained machine learning model can be very sensitive to the hyperparameter settings and thus it is important for the model owner to choose settings carefully in order to maximise the model’s efficacy (‘hyperparameter optimisation’). In particular, it is often practicable to optimise for hyperparameters for relatively low marginal compute cost when compared to competing methodologies for increasing model performance (for example, the use of larger training datasets or deeper models or even ).

This Roadmap

This guide will help you to improve model performance (that is, the model’s ‘efficacy’) via the process of hyperparameter optimisation. Moreover, to fully explain the implementation of hyperparameter optimisation, this guide will additionally also discuss validation methodologies which, in particular, seek to ensure that models perform well on new, unseen data.

We present the roadmap in two steps:

  • Step 1: Validation

  • Step 2: Hyperparameter search

before concluding with topics for further consideration.

choice of model