PAPER

Neural Disparity Refinement for Arbitrary Resolution Stereo

Filippo Aleotti*, Fabio Tosi*, Pierluigi Zama Ramirez*, Matteo Poggi, Samuele Salti, Stefano Mattoccia, Luigi Di Stefano
*Equal Contribution

We introduce a novel architecture for neural disparity refinement aimed at facilitating deployment of 3D computer vision on cheap and widespread consumer devices, such as mobile phones. Our approach relies on a continuous formulation that enables to estimate a refined disparity map at any arbitrary output resolution. Thereby, it can handle effectively the unbalanced camera setup typical of nowadays mobile phones, which feature both high and low resolution RGB sensors within the same device. Moreover, our neural network can process seamlessly the output of a variety of stereo methods and, by refining the disparity maps computed by a traditional matching algorithm like SGM, it can achieve unpaired zero-shot generalization performance compared to state-of-the-art end-to-end stereo models.

@inproceedings{aleotti2021neural,
    title={Neural Disparity Refinement for Arbitrary Resolution Stereo},
    author={Aleotti, Filippo and Tosi, Fabio and Zama Ramirez, Pierluigi and Poggi, Matteo and Salti, Samuele and Di Stefano, Luigi and Mattoccia, Stefano},
    booktitle={International Conference on 3D Vision},
    note={3DV},
    year={2021},
}

RESULTS

SCENEFLOW

Results on the SceneFlow testing set. From left to right, the RGB input image, the noisy input disparity map computed by SGM (rows 1-2), AD-Census (rows 3-4), C-CNN (rows 5-6) and the corresponding refined disparity estimated by our network.

MIDDLEBURY

Generalization results on Middlebury 2014 of our network (pre-trained on SceneFlow). From left to right, the RGB input image, the noisy input disparity map computed by SGM and the refined disparity estimated by our network.

KITTI

Generalization results on KITTI 2015 of our network (pre-trained on SceneFlow). From left to right, the RGB input image, the noisy input disparity map computed by SGM and the refined disparity estimated by our network.

UNBALANCE SETUP

Our network is able to handle also inputs with different resolutions. The top row depicts the input image, at 3840 × 2160 and the disparity maps, D, computed by SGM when the right image, is 480 × 270 and 320 × 180 (k = 8 and 12). The bottom row shows ground-truth and estimated disparity at 3840×2160.

CONTINOUS UPSAMPLING

Thanks to our continuous formulation we can estimate a disparity map at an arbitrary resolution. In the figure we see the comparison against a standard Nearest Neighbor interpolation.