An ‘Ethical Black Box’, Learning From Disagreement in Shared Control Systems

Shared control, where a human user cooperates with an algorithm to operate a device, has the potential to greatly expand access to powered mobility, but also raises unique ethical challenges. A shared-control wheelchair may perform actions that do not reflect its user’s intent in order to protect their safety, causing frustration or distrust in the process. Unlike physical accidents there is currently no framework for investigating or adjudicating these events, leading to a reduced capability to improve the shared control algorithm’s user experience. In this paper we suggest a system based on the idea of an ‘ethical black box’ that records the sensor context of sub-critical disagreements and collision risks in order to allow human investigators to examine them in retrospect and assess whether the algorithm has taken control from the user without justification.