Free viewpoint video (FVV) from multiview data

Multi-view video representation based on fast Monte Carlo surface reconstruction

This work, published in Salvador J, Casas J. Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction. IEEE Transactions on Image Processing. 2013 ;22(9):3342 - 3352., provides an alternative solution for the costly representation of multi-view video data, both for rendering and for scene analysis. It consists of two main steps:

1. Efficient 3D reconstruction: Foreground objects with static background are reconstructed via an efficient Monte Carlo discrete surface reconstruction method, which is suitable for GPU and exploits multi-resolution and temporal correlation, outperforming pre-existing volumetric techniques.
2. Fast meshing: A fast meshing algorithm interpolaties a continuous surface from the discrete 3D points reconstructed in the previous step.

The proposed reconstruction and meshing steps compare favourably to the state-of-the-art. Experimental results shows high accuracy in the reconstruction, evaluated by projecting the reconstructed objects onto the original viewpoints. The reconstructed scene can be easily projected onto any desired virtual viewpoint, allowing for Free-Viewpoint Video applications. We also introduce a rule-of-thumb for effective application as a convenient representation of 3D multi-view data fusion with a good quality vs. representation cost trade-off.

The Free viewpoint video (FVV) demo below shows the performance of the method when applied to multiview data with a virtual camera rotating around the reconstructed scene.

Datasets: multi-view sequences from two different sources: dancer, children and martial from 4D Repository, INRIA, 2011, kung-fu girl sequence from MPI Informatik, 2005

Illustration of the method along the FVV sequence:

1. Point reconstruction: green points during the first 2 seconds of each clip
2. Point coloring: colored points during the next 2-3 seconds
3. Meshing: continuous surfaces during the rest of the clips

Reproducible research: code available at the bottom of the page

Results: FVV videos obtained from fast Monte Carlo surface reconstruction results.
Each 3D reconstruction uses 100000 surface points rendered at 4fps (approx).

dancer: reconstruction from 8 views, 201 frames at a resolution of 780x582

children: reconstruction from 16 views, 339 frames at a resolution of 1624x1224

martial: reconstruction from 16 views, 210 frames at a resolution of 1624x1224

kung-fu girl: reconstruction from 25 views, 200 frames at a resolution of 320x240

Code to generate the sequences above:

Download compiled linux binaries (Debian squeeze) including a working example for the single core version of the algorithm. Please check the README file in the folder resulting when extracting the contents of the provided file.

Real-time versions using GPGPU

Two real-time versions of 3D reconstruction algorithms have been also developed to be run in the smart-room of the UPC. The first one was done by Enrique Oriol in 2010 and was based on Shape from Silhouette and Space Carving algorithms, using CUDA:

The second one was developed by Marc Maceira and was based on a Monte Carlo Surface Reconstruction  in 2011 using OpenCL: