Annotation of video by alignment to reference imagery

Video as an entertainment or information source in consumer, military and broadcast television applications is widespread. Typically however, the video is simply presented to the viewer, with only minimal manipulation. Examples include chroma-keying (often used in news and weather broadcasts) where specific color components are detected and used to control the video source. In the past few years, the advent of digital video and increases in computational power has meant that more complex manipulation can be performed. We present some highlights of our work in annotating video by aligning features extracted from the video to a reference set of features. Video insertion and annotation require manipulation of the video stream to composite synthetic imagery and information with real video imagery. The manipulation may involve only the 2D image space or the 3D scene space. The key problems to be solved are: indexing and matching to determine the location of insertion; stable and jitter-free tracking to compute the time variation of the camera; and seamlessly blended insertion for an authentic viewing experience. We highlight our approach to these problems by showing three example scenarios: 2D synthetic pattern insertion in live video; annotation of aerial imagery through geo-registration with stored reference imagery and annotations; and 3D object insertion in a video for a 3D scene.