About the challenge

Institute for Information Transmission Problems of the Russian Academy of Sciences (Kharkevich Institute or IITP RAS), in cooperation with University of Zagreb, invites research groups and individual enthusiasts to participate in the 2nd International Illumination Estimation Challenge (IEC).

The main goal of the challenge in this year is to develop novel algorithms for estimation of multiple light sources scene illumination and demonstrate its effectiveness using large and diverse image dataset.

This year IEC event will supplement the regular program of the 13th International Conference on Machine Vision (ICMV 2020, November 02-06, 2020, Rome, Italy), which bring together leading experts in computer vision and image processing. First IEC competition was held during 11th International Symposium on Image and Signal Processing and Analysis (ISPA 2019, September 23-25, 2019, Dubrovnik, Croatia) https://www.isispa.org/illumination-estimation-challenge.

We’d appreciate your help in attracting research teams to the participation in the challenge by placing an announce of the IEC on your site.

Significance of the Challenge

The importance of scene illumination estimation has increased significantly, i.e. most mobile phones, tablets and laptops began to be equipped with cameras. Each of these devices performs color correction based on the estimated scene illumination when shooting an image or video. This is the essential part of the color image formation pipeline in modern mobile devices.

Human color perception system has a color constancy feature providing object coloration recognition regardless of the scene illumination. Auto white balance (AWB) is an analogous feature in the world of digital cameras. Important step here is the illumination estimation problem. To solve the problem one needs a good algorithm and high quality dataset of images to evaluate the algorithm results. To date, thousands of scientific papers have been written on this problem, but we still can’t say it is solved. Why?

First, because the volumes of open scientific datasets are not large enough and do not fully cover various cases of illumination.

Second, because images in existing datasets are labeled indicating ground-truth scene illumination, but lack information about the scene content. This is important for a deeper study of the problem and dividing it into subtasks. Perhaps, scene illumination color estimation during the day, at night, in nature, in the city should be performed in various ways, which requires study.

Third, because up to now the problem of illumination parameters estimation was solved in an oversimplified formulation: the evaluation of a single dominant light source (and, accordingly, there are very few publicly available datasets with multiple light sources). On the one hand, single light source assumption is true for images taken on a cloudy day, or in a room with a single lamp, on the other hand - this assumption is not valid on a sunny day when there are two light sources: the sun and the sky. And at night on the street, or in a closed room, this assumption is far from reality.

The goal of IEC is to take a step towards overcoming these three obstacles. To do this, we collected the world's largest dataset (about 5,000 images) captured on cameras with the same sensor (Canon 600D and Canon 550D), and labeled each image with metadata. This metadata contains estimates of the two light sources in the scene (ground-truth) and additional information on the scene content.

Leaderboard

General track

Algorithm Mean (worst 25%) Mean Median Trimean Team Pastebin link Prize
CAUnet 4.084077 1.605 0.966 1.084 Zhihao Li
(Nanjing University)
kHb4GCeH 1st
CAUnet 4.321251 1.725 1.084 1.207 Zhihao Li
(Nanjing University)
m0SfbmKw
sde-awb 4.979334 1.914 1.164 1.269 Yanlin Qian
(Huawei MultiMedia Team)
CNV6MLmi 2nd
sde-awb 5.112188 1.952 1.149 1.292 Yanlin Qian
(Huawei MultiMedia Team)
Rq1CnueD
sde-awb 5.377026 2.034 1.150 1.282 Yanlin Qian
(Huawei MultiMedia Team)
6By3tqFL
illumGAN* 9.999407 4.643 3.588 3.841 Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) UDKTdpGe
GreyWorld baseline 10.419472 4.500 3.319 3.611
Const baseline 17.023748 7.081 4.020 5.275
illGAN1.0 25.954709 19.325 18.278 18.499 Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) izLazHJN 3rd
illGAN1.0 26.058358 19.360 18.328 18.498 Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) vppsZq0V
illGAN1.0 26.459121 19.408 18.468 18.710 Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) QB20MZLm

Indoor track

Algorithm Mean Median Trimean Team Pastebin link Prize
sde-awb 2.541370 1.763 1.943 Yanlin Qian
(Huawei MultiMedia Team)
z7L4ccwY 1st
illumGAN* 3.191023 2.596 2.674 Jianhui Qiu, Shaobing Gao,
Rui Yang, Qinyi Jiang
(College of Computer Science, SCU)
kkK1kgdL
PCGAN, MCGAN 3.301088 2.312 2.298 Riccardo Riva, Marco Buzzelli, Simone Bianco, Raimondo Schettini (Imaging and Vision Laboratory, University of Milan - Bicocca) XMy5NHzY 2nd
PCGAN, MCGAN 3.376422 2.312 2.337 Riccardo Riva, Marco Buzzelli, Simone Bianco, Raimondo Schettini (Imaging and Vision Laboratory, University of Milan - Bicocca) fYAyFsam
GreyWorld baseline 4.105811 3.673 3.545
Const baseline 15.269933 14.802 15.332

Two-illuminant track

Algorithm Mean squared Mean Median Trimean Team Pastebin link Prize
sde-awb 31.026217 2.751 2.262 2.290 Yanlin Qian
(Huawei MultiMedia Team)
5sbE4xMF 1st
sde-awb 31.542930 2.737 2.171 2.309 Yanlin Qian
(Huawei MultiMedia Team)
FK19nsTb
3du-awb* 37.305135 2.863 2.503 2.497 Yang Liu, Masoumeh Bakhtiariziabari, Gaurav Kudva, Sezer Karaoglu
(3DUniversum)
Theo Gevers
(3DUniversum/University of Amsterdam)
YrMDaGBx
GreyWorld baseline 81.840743 4.127 3.538 3.715
Const baseline 144.745182 5.264 3.475 3.815
* -- out of the competition (after the July 31).

Rules for participation

UPDATE. The train dataset will be published soon

Registration is open till 13:00 July 31 (GMT+3), 2020. To register, please send an email to chroma@iitp.ru indicating the team members and their affiliations.

Challenge consist of three tracks:

For each track we have prepared corresponding dataset and metric for final ranking. The metrics code is available on the github. Details are provided in the following section. The key from the test part of the dataset is published below.

Submission

The submission form is available HERE. You will be able to submit your solution till 13:00 July 31 (GMT+3). In the form you are required to:

You are expected to use Pastebin (with the Paste Exposure parameter set to the Unlisted value) to store your illumination estimations and provide links to them in the submission form. The predictions for the test data should be in a CSV format, like const or GreyWorld baselines. They will be checked with a code similar to this. Submissions under serious suspicion of dishonorable conduct will not be taken into account.

Each team can submit up to 3 solutions for each track. They can be sent in a single Google Form submission or split to up to 9 submissions with various algorithm names and participant lists. The results will be published on August 2nd. As the alternative, you may send email to chroma@iitp.ru with answers till the same deadline, but we highly recommend using the submission form.

Detailed tracks description

For the challenge we have prepared new dataset, which is essentially extention of the CubePlus dataset. It will be published soon.

General track

To date, many algorithms for evaluating lighting in the scene have been developed and successfully used. In solving this problem, the key role is played by the representativeness and variability of the dataset. These requirements impose restrictions on their size; it must be large. This will allow you to create stable solutions by analyzing system errors in various categories of images.

At first glance, a good solution is to combine the existing datasets into a single large dataset. However, in this case, it is difficult to distinguish which particular factor influences the behavior of the algorithm: the difficulty of the scene or the difference in camera sensitivity.

Another way to overcome this limitation is to collect a large dataset using the same sensor. To conduct this competition, we went exactly the second way and prepared for the participants of the competition a dataset assembled using one sensor. To analyze the operation of the algorithm, we supplied the images with markup files, which contain not only information about the lighting and shooting parameters, but also information about the contents of the scene.

In the competition of last year, the median was used as the ranking metric. This is a good metric that allows you to ignore dataset layout errors and periodic malfunctions in the algorithm. This year, however, the main goal of this track is to create a reliable solution, the behavior of which, even in extreme cases, is not bad.

Therefore, to assess the quality of the proposed solutions, we will use the average reproduction error for 25% of the worst images (see code).

Indoor track

One of the stand-alone photography categories is indoor photography. Very often, under these conditions, the lighting in the scene is quite complicated: there are many sources of lighting (light from the window, incandescent lamps, LEDs, and so on) in different places of the scene. Under these conditions, the determination of the dominant source in the scene may turn out to be a rather difficult task.

Since this task in itself is difficult for the final quality assessment in this track, we will use the average reproduction error (see code).

Two-illuminant track

In everyday life, there are rarely situations where there is really only one source of lighting in the scene. Even during the day, it is customary to divide the lighting into two sources - the sun and sky. The main question of this track is whether it is possible to reliably extract more information about lighting using a single image. For these purposes, a dataset was assembled using a volumetric color target (SpyderCube) whose faces are illuminated by different sources. This dataset includes images for which the angle between GT is greater than or equal to two degrees.

For this track, the ranking metric is squared sum of two angular reproduction errors. There are two variants to establish the correspondence between answers and ground truths, so we will choose one with minimal error (see code).

Structure of the datasets

All images were taken with the same camera sensor model, with the three cameras Canon 550D x1 and Canon 600D x2.

The image ordering with respect to their creation time. In the lower right corner of each image, the SpyderCube calibration object is placed. Its two neutral 18% gray faces were used to determine the ground-truth illumination for each image. Due to the angle between these two faces, for images with two illuminations, e.g. one in the shadow and one under the direct sunlight, it was possible to simultaneously recover both of them and they are provided for each image.

In all dataset images with two distinct illuminations, one of them is always dominant so that the uniform illumination assumption effectively remains valid. For Indoor Dataset we’ve chosen only images with an angular difference less than 2 degrees between left and right edges of the SpyderCube. For dominant light source was chosen the brighter one. The black level, i.e. the intensity that has to be subtracted from all images in order to use them properly, equals 2048.

All datasets contain files of three types:

Provided JSON files have the following field and structure

Python script for dataset images reading and visualization is provided here.

Please note, that final solutions for General and Indoor track should be provided in the following form and for Two-illuminant track in the following form.

Bonus data

CubePlus dataset was captured with exactly the same Canon 550D and can be downloaded here. We prepared the contest-like markup for it. The jsons contain extra 'contest_like' field, which is 'false' for images that could be probably filtered out for the challenge.

Interested parties could find it potentially useful to use artificially created data. What they need in that case is the Croatian Paper (CroP) dataset generator described in detail in the paper "CroP: Color Constancy Benchmark Dataset Generator" that is currently available at arXiv. The resources for the generator are available at following link.

Test datasets

The test data format differs slightly from the train one. The SpyderCube rectangle is cut out. JSON files lack ground truth answers and manually annotated properties. Image ids are shuffled.

For each scene, the test datasets have the two files (name.PNG, name.JPG.JSON). name.JPG.JSON contain all information available during the shooting procedure:

The ds_version field of all the test files is 2.0.

Prizes and awards

Winners will receive money prizes:

Also, winners will receive a winner certificate, invitations to the ICMV conference with a speech on their decision, and an opportunity to publish the paper about challenge results.

To receive a money prize, you need to take part in the ICMV conference.

Organizers

Egor Ershov
Kharkevich Institute for Information Transmission Problems
Alex Savchik
Kharkevich Institute for Information Transmission Problems
Ilya Semenkov
Kharkevich Institute for Information Transmission Problems
Arseniy Terekhin
Kharkevich Institute for Information Transmission Problems
Daria Senshina
Kharkevich Institute for Information Transmission Problems
Dmitry Nikolaev
Kharkevich Institute for Information Transmission Problems
Nikola Banic
Gideon Brothers, Croatia
Marko Subasic
University of Zagreb
Sven Loncaric
University of Zagreb

Sponsorship

Papers

If you use these datasets in your research, please refer to our papers from the list