Seamless 3D Interaction of Virtual and Real Objects in Professional Virtual Studios

This paper presents a novel pipeline for adding 3D geometry information to conventional green screen assets in virtual broadcast studios. Today's green screen studios are limited to 2D object interaction. Information about depth is not available, and hence the 3D relation to non-virtual objects is unknown. We overcome this situation by using an integrated RGB+depth camera and a specialized processing pipeline, which delivers an additional synchronized depth channel in realtime. This technology enables interaction of virtual and real objects, including shadow casting and occluding virtual objects. We implemented and tested several of these effects within a professional virtual studio renderer.