Binary ensemble machine learning for confident decisions

Folk wisdom on the subject of human knowledge holds that it is “better to know nothing than to know what ain’t so1”. In some circumstances that precept may be particularly important. If the negative consequences of false knowledge are sufficiently severe, we may be willing to forgo the benefits of knowing some facts to avoid the dangers of believing “facts” that are incorrect. This is the foundation of the “innocent until proven guilty” system of justice. According to English jurist William Blackstone, “it is better that ten guilty persons escape than that one innocent suffer2”. Similar principles apply wherever it is especially harmful to act upon false beliefs. If we wish to employ machine learning as an aid to human judgment, it may in some cases be advisable to insist upon near-certainty from the machine’s reported results. One way of working towards this goal is to use an ensemble of different agents. If their results are consistent with each other, we can have greater confidence in their overall reliability. We can also set a threshold value for the average confidence of the agents themselves. This paper explores the decision-making process for developing an ensemble of classifiers, and evaluates the results in the context of an example set. This set is appropriately categorized by a hierarchical structure, which permits less-specific judgments to be made if confidence falls below our predetermined threshold. We examine the tradeoffs to be made when setting parameters, and discuss aligning them with overarching requirements.