Efficient multi-criteria optimization on noisy machine learning problems

Abstract

Recent research revealed that model-assisted parameter tuning can improve the quality of supervised 
machine learning (ML) models. The tuned models were especially found to generalize better and to be 
more robust compared to other optimization approaches. However, the advantages of the tuning often came 
along with high computation times, meaning a real burden for employing tuning algorithms. While the 
training with a reduced number of patterns can be a solution to this, it is often connected with decreasing 
model accuracies and increasing instabilities and noise. Hence, we propose a novel approach defined by a 
two criteria optimization task, where both the runtime and the quality of ML models are optimized. Because 
the budgets for this optimization task are usually very restricted in ML, the surrogate-assisted Efficient 
Global Optimization (EGO) algorithm is adapted. In order to cope with noisy experiments, we apply two hypervolume 
indicator based EGO algorithms with smoothing and reinterpolation of the surrogate models. The techniques do 
not need replicates. We find that these EGO techniques can outperform traditional approaches such as latin hypercube 
sampling (LHS), as well as EGO variants with replicates. (C) 2015 Elsevier B.V. All rights reserved.