Measuring User Performance during Interactions with Digital Video Collections.

With more and more digital videos found online, video retrieval researchers have begun to create various representations or surrogates for digital videos, such as poster frames, storyboards, video skims and fast forwards. How to evaluate the effectiveness of these video surrogates has become an issue for researchers. This paper proposes two general classes of user tasks—recognition tasks and tasks requiring inference—for which performance measures were developed. The measures include graphical object recognition, textual object recognition, action recognition, free-text gist determination, multiple-choice gist determination and visual gist determination. The preliminary results from two user studies applying these six measures are also discussed in this paper.