How to fool a black box machine learning based side-channel security evaluation

Machine learning and deep learning algorithms are increasingly considered as potential candidates to perform black box side-channel security evaluations. Inspired by the literature on machine learning security, we put forward that it is easy to conceive implementations for which such black box security evaluations will incorrectly conclude that recovering the key is difficult, while an informed evaluator / adversary will reach the opposite conclusion (i.e., that the device is insecure given the amount of measurements available).