SPRINT Multi-Objective Model Racing


Abstract

Multi-objective model selection, which is an important aspect of Machine Learning, refers to the problem of identifying a set of Pareto optimal models from a given ensemble of models. This paper proposes SPRINT-Race, a multi-objective racing algorithm based on the Sequential Probability Ratio Test with an Indifference Zone. In SPRINT-Race, a non-parametric ternary-decision sequential analogue of the sign test is adopted to identify pair-wise dominance and non-dominance relationship. In addition, a Bonferroni approach is employed to control the overall probability of any erroneous decisions. In the fixed confidence setting, SPRINT-Race tries to minimize the computational effort needed to achieve a predefined confidence about the quality of the returned models. The efficiency of SPRINT-Race is analyzed on artificially-constructed multi-objective model selection problems with known ground-truth. Moreover, SPRINT-Race is applied to identifying the Pareto optimal parameter settings of Ant Colony Optimization algorithms in the context of solving Traveling Salesman Problems. The experimental results confirm the advantages of SPRINT-Race for multi-objective model selection.