Towards a natural language programming interface for smart homes

Recently several major industry players have introduced smart home personal assistants to the Internet of Things market. However, these products are often little more than voice interfaces to device control apps. More focus is needed on supporting the high-level applications and goals that end users express in natural language. In this preliminary work, we collect and analyze over 1,600 utterances from nontechnical end users encompassing a variety of smart home interactions. We find that given the right context, end users voluntarily use computational language in response to open-ended prompts. From these commands and queries, we extract the computational structures and the entities, actions, and conditions that end users employ when interacting with the home. Our preliminary results suggest a path forward for end-to-end architectures that automatically map natural language utterances to executable smart home programs. We hope to help bring natural language interfaces for the home into alignment with end users' mental models.