Interactive registration for 3D data fusion

Resource Type Date
Results 2016-07-12


The results presented in this page have been developed in the context of the Bachelor thesis of Arantxa Casanova "Fusion of 3D data for interactive applications" (TelecomBCN/UPC July 2016) and later presented at A. Casanova, A. Pujol-Miró, J. Ruiz-Hidalgo, and J. Casas, Interactive Registration Method for 3D data Fusion, in IC3D, Liège (Belgium) 2016.

One way to overcome some of the inherent limitations of commercial depth sensors is by capturing the same scene with several depth sensors and merging their data, i.e. by performing 3D data fusion. This allows enlarging the interaction area or improving the precision of the detection, but requires the registration of point clouds from different sensors.

Arantxa Casanova won the Award to the Best Final Year Project in Spain in Information and Communications Technologies (2nd edition, UPF, award video) her Bachelor thesis. In her work, she implemented registration algorithms aimed to enlarge the detection area in interactive applications by the fusion of 3D data from two sensors. In particular, camera calibration from a geometrical pattern and auto-registration using the detection of a human finger as a reference. The finger detector method is more user-friendly and does not require external objects, while the elapsed time and the registration error are similar to those obtained through the classical camera calibration, as shown in the procedure in Figure 1.

Finger calibration procedure (step 1)          Finger calibration procedure (step 2)

Figure 1. Finger calibration procedure: First, the user describes a geometrical shape (a 3D cube) with the finger.
The finger is tracked in the two sensors (in orange and red in the left images from both sensors).
Then the orange points are registered (rotated/translated) and shown in green over the red points of the second sensor.

In the IC3D paper that followed, the application to an interactive scenario is extended, proposing a new interactive, fast and user-friendly method for depth sensor registration. A a detector provided by Exipple Studio (gestoos) tracks a hand through the detection area of a single depth sensor. The registration proposed allows to extend the detection area by 78.5% at 120cm from the sensors. The video in Figure 2 shows one possible application for hand drawing in an extended detection area of two kinect sensors.

Figure 2. Hand tracking application using two registered sensors. Points in green are detected by sensor 1, points in green by sensor 2.

This work has been one of the foundations of the 3D Telepresence project at GPI, which exploits the registration of multiple depth sensors to generate a virtual 3D view of the correspondant in a 3D Teleconference application with Virtual Reality Glasses. The image below shows a snapshot of the registered pointclouds of 4 kinect sensors.

Snapshot of the 3D Telepresence capture

Figure 3. Snapshot of the 3D Telepresence application, with 4 registered depth sensors

People involved

Alba Pujol PhD Candidate
Javier Ruiz Hidalgo Associate Professor
Josep R. Casas Associate Professor