Evaluation of visual analytics environments: The road to the Visual Analytics Science and Technology challenge evaluation methodology

Evaluation of software can take many forms ranging from algorithm correctness and performance to evaluations that focus on the value to the end user. This article presents a discussion of the development of an evaluation methodology for visual analytics environments. The Visual Analytics Science and Technology Challenge was created as a community evaluation resource. This resource is available to researchers and developers of visual analytics environments and allows them to test out their designs and visualization and compare the results with the solution and the entries prepared by others. Sharing results allows the community to learn from each other and to hopefully advance more quickly. In this article, we discuss the original challenge and its evolution during the 7 years since its inception. While the Visual Analytics Science and Technology Challenge is the focus of this article, there are lessons for many involved in setting up a community evaluation program, including the need to understand the pur...