Using algorithm configuration tools to optimize genetic programming parameters: a case study

We use Sequential Model-based Algorithm Configuration (SMAC) to optimize a group of parameters for PushGP, a stack-based genetic programming system, for several software synthesis problems. Applying SMAC to one particular problem leads to marked improvements in the success rate and the speed with which a solution was found for that problem. Applying these "tuned" parameters to four additional problems, however, only improved performance on one, and substantially reduced performance on another. This suggests that SMAC is "overfitting", tuning the parameters in ways that are highly problem specific, and raises doubts about the value of using these "tuned" parameters on previously unsolved problems. Efforts to use SMAC to optimize PushGP parameters on other problems have been less successful due to a combination of long PushGP run times and low success rates, which make it hard for SMAC to acquire enough information in a reasonable amount of time.