Data-driven Local Coordinate Systems for Image-Based Rendering

Image-based representations of an object profit from known g eometry. The more accurate this geometry is known, the better corresponding pixels in the different images canbe aligned, which leads to less artifacts and better compression performance. For opaque objects the per-pixeldata can then be interpreted as a sampling of the BRDF at the respective surface point. In order to parameteri ze this sampled data a coordinate frame has to be defined. In previous work this coordinate frame was either th e global frame or a local frame derived from the base geometry. Both approaches lead to misalignments betwe en sample vectors: Features of basically very similar BRDFs will be shifted to different regions in the sample vect or leading to poor compression performance. In order to improve alignment between the sampled BRDFs in imag e-based rendering, we propose an optimization algorithm which determines consistent coordinate frames f or every sample point on the object surface. This way we efficiently align the features even of anisotropic reflection functions and reconstruct approximate local coordinate frames without performing an explicit 3D-reconstruction.The optimization is calculated efficiently by exploiting the Fourier-shift theorem for spherical harmonics. In orde r to deal with different materials in a scene, the technique is combined with a clustering algorithm. We demonstrate the utility of our method by applying it to BTFs and 6D surface reflectance fields.