Monocular Depth from Images of Specular and Transparent Surfaces


TRICKY 2024 Challenge - ECCV 2024


We are delighted to inform you that the Booster dataset will be employed in the Monocular Depth from Images of Specular and Transparent Surfaces Challenge as a part of the TRICKY 2024 workshop in conjunction with ECCV 2024!

CHALLENGES DESCRIPTION

This challenge aims at fostering the community towards developing next-generation monocular depth networks capable of reasoning at a higher level, and thus yield accurate, 3D reconstructions for challenging objects, yet of common use.

The challenges will be divided into two phases:
  1. Development: During this period, the participants will have to construct a model. The model can be trained using the Booster training split and any additional data. The approach can be evaluated on the official validation set of each track.
  2. Test: During this period, the participants can submit the predictions of their model on the official test set. The depth maps will be evaluated by the organizers with the quantitative metrics.


DATASETS

TRAINING DATA [DOWNLOAD]


The training set is composed of 38 different indoor scenes, containing transparent or reflective objects. Each scene was acquired with several illumination conditions, for a total of 228 training images at 4112x3008 resolution. For images belonging to the training split, we release high-resolution stereo images, material segmentation, left and right disparity ground-truth, occlusion mask, and calibration parameters.

Notes on training data:
  • We do not restrict submitted methods from using additional training data. If it is used, it is necessary to indicate the source and amount.
  • We do not restrict submitted methods from using pretrained networks. If it is used, it is necessary to provide details.

VALIDATION DATA [DOWNLOAD]


The validation set is composed of 6 different indoor scenes, containing transparent or reflective objects. To assess method robustness, each scene is validated on five different illumination conditions, for a total of 30 validation images at 1028x752 resolution. For images belonging to the validation split, only monocular rgb images, and the calibration parameters are released.

TEST DATA [DOWNLOAD]


To rank the submitted models, we test them on a separate test set. Also for the case of the test set, only rgb images and the calibration parameters are released. The participants are required to apply their models to the released images and submit their results to the server. It should be noted that the images in the test set cannot be used for training.

DEV KIT [DOWNLOAD]


We provide some useful scripts to read and visualize Booster training data.

SUBMISSION AND EVALUATION

EVALUATION SERVER [CODALAB SERVER]


We use CodaLab servers for online submission in the development and test phases, testing the results on the validation set and test set respectively. After the test phase, the final results and the source codes (both training and test) need to be submitted via emails (boosterbenchmark@gmail.com).

EVALUATION METRICS


IMPORTANT: We evaluate predictions in the depth domain, e.g., closer objects have smaller values. As monocular networks estimate depth up to a scale factor, we first compute a shift and scale to match predictions and ground truth ranges.
Then, we compute the absolute error relative to the ground value (ABS Rel.), and the percentage of pixels having the maximum between the prediction/ground-truth and ground-truth/prediction ratios lower than a threshold (δi, with i being 1.05, 1.15, and 1.25). We also estimate the mean absolute error (MAE), and Root Mean Squared Error (RMSE). All the metrics introduced so far are computed on any valid pixel (All), on pixels belonging to Transparent or Mirror surfaces (Class ToM), or on Other type of materials (Class Other). MAE, RMSE, and ABS Rel. are lower the better. δi instead is higher the better. To rank submissions, we use only the δ1.05 averaged on pixels belonging to ToM surfaces. Other metrics might be used to declare the final winner of the competition.

SUBMISSION - PREDICTIONS FORMAT


The npy should contain 32bits depth values for the 1028x752 images, matching its resolution. Note: We evaluate depth maps up to a scale and shift factors.

SUBMISSION - DEVELOPMENT PHASE


During the development phase, the participants submit their results on the validation set to get feedback from the CodaLab server.
The validation set should only be used for evaluation and analysis purposes but NOT for training.
The submitted zip files have a structure similar to this: $scene/$img_basepath.npy. E.g., Mirror3/0000.npy. Please do not include any root folder when creating the zip file. An example of submissions on the validation set can be found [ HERE ].

SUBMISSION - TEST PHASE


During the test phase, the participants submit their results on the test set on the CodaLab server.
The test results will not be visible to other participants during this phase.
The test set CANNOT be used for training.
The submitted zip files have a structure similar to this: $scene/$img_basepath.npy. E.g., Mirror3/0000.npy. Please do not include any root folder when creating the zip file.

FINAL SUBMISSION


After the test phase, the participants will submit a zip file (containing fact sheet, source code, and final results) to the official submission account (boosterbenchmark@gmail.com) by email.
The final submission should be made by the following rules:
The submitted results must be from the same method that generated the last submission to the CodaLab. We will check the consistency. Otherwise, the submission is invalid.
Both the testing source code (or executable) and the model weights must be submitted. We will run the test code to reproduce the results. Reproducibility is a necessary condition. Training code doesn't necessarily have to be included. The code and the model might be posted on the TRICKY 2024 website.
Factsheet describing the method must be submitted. The factsheet format is provided here. Participants must submit a compiled pdf file and the tex source of the factsheet. Participants must provide enough method details and include an overview method figure. This helps writing the challenge summary report.

EMAIL FORMAT
Please use the following format to submit your final results, fact sheet, code, model (with trained parameters).
to: boosterbenchmark@gmail.com;
cc: your_team_members
title: [TRICKY 2024: Monocular Depth from Images of Specular and Transparent Surfaces] - [Team_name]
body should include:
1) team name
2) team leader's name, affiliation, and email address
3) team members' names, affiliations, and email addresses
4) user name on the TRICKY 2024 CodaLab leaderboard (if any)
5) executable/source code attached or download links.
6) fact sheet attached (template available here: https://it.overleaf.com/read/mjxdpdtrdjcs#1b7e4c)
7) download link to the results


IMPORTANT CHALLENGES DATES

  • 2024-05-21: Release of training and validation data;
  • 2024-05-01: Validation server online;
  • 2024-06-16: Final test data release, validation server closed;
  • 2024-06-30: Test result submission deadline;
  • 2024-07-01: Fact sheet / code / model submission deadline;
  • 2024-07-07: Final leaderboard release to the participants;


NEWS AND UPDATES

  • 2024-07-05: Final Leaderboard released! See it [HERE]!
  • 2024-06-19: Test data have been released.
  • 2024-05-01: Training and validation data have been released.
  • 2024-04: Workshop proposal has been accepted.