A Spatio-Angular Binary Descriptor for Fast Light Field Inter View Matching

7th February 2020
A Spatio-Angular Binary Descriptor for Fast Light Field Inter View Matching

Abstract

Light fields are able to capture light rays from a scene arriving at different angles, effectively creating multiple perspective views of the same scene. Thus, one of the flagship applications of light fields is to estimate the captured scene geometry, which can notably be achieved by establishing correspondences between the perspective views, usually in the form of a disparity map. Such correspondence estimation has been a long standing research topic in computer vision, with application to stereo vision or optical flow. Research in this area has shown the importance of well designed descriptors to enable fast and accurate matching. We propose in this paper a binary descriptor exploiting the light field gradient over both the spatial and the angular dimensions in order to improve inter view matching. We demonstrate in a disparity estimation application that it can achieve comparable accuracy compared to existing descriptors while being faster to compute.

This paper has now been accepted to ICIP 2020!

Related publications

5 entries « 1 of 5 »

2020

Alain, Martin; Smolic, Aljosa

A Spatio-Angular Binary Descriptor for Fast Light Field Inter View Matching Inproceedings Forthcoming

2020 The International Conference on Image Processing (IEEE ICIP 2020), Forthcoming.

Abstract | Links | BibTeX

5 entries « 1 of 5 »

Implementation

The C++ code will soon be available on github.

Additional results

The proposed Spatio-Angular Binarised Orientation Maps (SABOM) descriptor is compared to SIFT, DAISY, and the original BOOM descriptor.

ROC performance

HCI dataset INRIA Dense dataset INRIA Sparse dataset

Disparity estimation

CPM – HCI CPM – INRIA
CPM+PF – HCI CPM+PF – INRIA

Visual results

We show here comparisons between the disparity maps obtained with the SIFT descriptor and the proposed SABOM descriptor for the HCI benchmark, the INRIA synthetic dataset, and the Stanford Lego Gantry dataset. As mentioned in the paper, the disparity estimation is used to compare existing descriptors but would require further improvement to compete with the best state-of-the-art method for disparity estimation. The disparity maps are color-coded with the Turbo colormap.
Overall, similar quality is obtained for both descriptors for while SABOM is much faster to compute.

HCI dataset

Antinous Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Boardgames Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Dishes Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Greek Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Kitchen Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Medieval2 Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Museum Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Pens Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Pillows Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Platonic Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Rosemary Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Table Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Tomb Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Tower Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Town Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Vinyl Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF

INRIA Dense dataset

Antiques Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Big_clock Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Blue_room Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Bottles Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Camera_brush Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Dinosaur Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Dishes Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Electro_devices Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Flowers_clock Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Flying_dice Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Flying_furniture Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Flying_toys Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Furniture Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Green_balloon Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Kiwi_bike Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Lonely_man Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Microphone Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Microphone_rooster Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Origami Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Pinenuts_blue Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Robots Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Rooster_clock Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Roses_bed Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Roses_table Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Smiling_crowd Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Smiling_crowd_roses Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Three_pillows Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Toy_bricks Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Toy_friends Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Two_vases Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
White_lamp Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
White_roses Center View Ground truth disparity map
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF

Stanford dataset

Amethyst Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
Chess Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
EucalyptusFlowers Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
JellyBeans Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
LegoBulldozer Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
LegoKnights Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
TarotCardsandCrystalBall_LargeAngle Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
TarotCardsandCrystalBall_SmallAngle Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
TheStanfordBunny Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF
TreasureChest Center View Ground truth disparity map not available
SIFT CPM SIFT CPM+PF
SABOM CPM SABOM CPM+PF