Revisit active learning workflow #893
Description
From discussion with @radka-j, following #872, some aspects that would be good to revisit:
- When is it good to reinitialize an emulator following adding new training points and when is it not? For example, with Gaussian Process emulators it seems that any included observation noise is typically driven to very small values for increasing numbers of epochs; if not reinitialized this effectively becomes the case
- Relatedly, for Gaussian Process emulators, should uncertainty with observation noise be included in order to attain optimal AL performance at a given number of epochs?
- How does the AL performance change when we using a different UQ emulator (e.g.
EnsembleMLP) - Is there a way of using probabilistic metrics (e.g. introduced in Add metrics for probabilistic outputs #884) to help motivate/support any of the choices above regarding emulators, their parameters and choice of re-initialization?
- Should reinitialization be done only upon a certain schedule? Or when a batch of new inputs are fitted? Relates to Add batch active learning #849 too.
An outcome from the issue could be a notebook/tutorial demonstrating and comparing the impact of the choices above in a given AL application.