About the challenge
Institute for Information Transmission Problems of the Russian Academy of Sciences (Kharkevich Institute or IITP RAS), in cooperation with University of Zagreb, invites research groups and individual enthusiasts to participate in the 2nd International Illumination Estimation Challenge (IEC).
The main goal of the challenge in this year is to develop novel algorithms for estimation of multiple light sources scene illumination and demonstrate its effectiveness using large and diverse image dataset.
This year IEC event will supplement the regular program of the 13th International Conference on Machine Vision (ICMV 2020, November 02-06, 2020, Rome, Italy), which bring together leading experts in computer vision and image processing. First IEC competition was held during 11th International Symposium on Image and Signal Processing and Analysis (ISPA 2019, September 23-25, 2019, Dubrovnik, Croatia) https://www.isispa.org/illumination-estimation-challenge.
We’d appreciate your help in attracting research teams to the participation in the challenge by placing an announce of the IEC on your site.
Significance of the Challenge
The importance of scene illumination estimation has increased significantly, i.e. most mobile phones, tablets and laptops began to be equipped with cameras. Each of these devices performs color correction based on the estimated scene illumination when shooting an image or video. This is the essential part of the color image formation pipeline in modern mobile devices.
Human color perception system has a color constancy feature providing object coloration recognition regardless of the scene illumination. Auto white balance (AWB) is an analogous feature in the world of digital cameras. Important step here is the illumination estimation problem. To solve the problem one needs a good algorithm and high quality dataset of images to evaluate the algorithm results. To date, thousands of scientific papers have been written on this problem, but we still can’t say it is solved. Why?
First, because the volumes of open scientific datasets are not large enough and do not fully cover various cases of illumination.
Second, because images in existing datasets are labeled indicating ground-truth scene illumination, but lack information about the scene content. This is important for a deeper study of the problem and dividing it into subtasks. Perhaps, scene illumination color estimation during the day, at night, in nature, in the city should be performed in various ways, which requires study.
Third, because up to now the problem of illumination parameters estimation was solved in an oversimplified formulation: the evaluation of a single dominant light source (and, accordingly, there are very few publicly available datasets with multiple light sources). On the one hand, single light source assumption is true for images taken on a cloudy day, or in a room with a single lamp, on the other hand - this assumption is not valid on a sunny day when there are two light sources: the sun and the sky. And at night on the street, or in a closed room, this assumption is far from reality.
The goal of IEC is to take a step towards overcoming these three obstacles. To do this, we collected the world's largest dataset (about 5,000 images) captured on cameras with the same sensor (Canon 600D and Canon 550D), and labeled each image with metadata. This metadata contains estimates of the two light sources in the scene (ground-truth) and additional information on the scene content.
Leaderboard
General track
| Algorithm | Mean (worst 25%) | Mean | Median | Trimean | Team | Pastebin link | Prize |
|---|---|---|---|---|---|---|---|
| CAUnet | 4.084077 | 1.605 | 0.966 | 1.084 | Zhihao Li (Nanjing University) |
kHb4GCeH | 1st |
| CAUnet | 4.321251 | 1.725 | 1.084 | 1.207 | Zhihao Li (Nanjing University) |
m0SfbmKw | |
| AL-AWB* | 4.419051 | 1.822 | 1.197 | 1.317 | Xiaoyan Xing (Huawei Media Technology Institute, Tsinghua University), Sibo Feng(Huawei Media Technology Institute), Yanlin Qian (Huawei MultiMedia Team) | G1VQ3QDr | |
| AL-AWB* | 4.656795 | 1.891 | 1.230 | 1.352 | Xiaoyan Xing (Huawei Media Technology Institute, Tsinghua University), Sibo Feng (Huawei Media Technology Institute), Yanlin Qian (Huawei MultiMedia Team) | 7j0xGHLz | |
| sde-awb | 4.979334 | 1.914 | 1.164 | 1.269 | Yanlin Qian (Huawei MultiMedia Team) |
CNV6MLmi | 2nd |
| sde-awb | 5.112188 | 1.952 | 1.149 | 1.292 | Yanlin Qian (Huawei MultiMedia Team) |
Rq1CnueD | |
| sde-awb | 5.377026 | 2.034 | 1.150 | 1.282 | Yanlin Qian (Huawei MultiMedia Team) |
6By3tqFL | |
| illumGAN* | 9.999407 | 4.643 | 3.588 | 3.841 | Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) | UDKTdpGe | |
| GreyWorld baseline | 10.419472 | 4.500 | 3.319 | 3.611 | |||
| Const baseline | 17.023748 | 7.081 | 4.020 | 5.275 | |||
| illGAN1.0 | 25.954709 | 19.325 | 18.278 | 18.499 | Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) | izLazHJN | 3rd |
| illGAN1.0 | 26.058358 | 19.360 | 18.328 | 18.498 | Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) | vppsZq0V | |
| illGAN1.0 | 26.459121 | 19.408 | 18.468 | 18.710 | Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) | QB20MZLm |
Indoor track
| Algorithm | Mean | Median | Trimean | Team | Pastebin link | Prize |
|---|---|---|---|---|---|---|
| AL-AWB* | 2.500120 | 2.293 | 2.201 | Xiaoyan Xing (Huawei Media Technology Institute, Tsinghua University), Sibo Feng(Huawei Media Technology Institute), Yanlin Qian (Huawei MultiMedia Team) | FZa90iH2 | |
| sde-awb | 2.541370 | 1.763 | 1.943 | Yanlin Qian (Huawei MultiMedia Team) |
z7L4ccwY | 1st |
| AL-AWB* | 2.855412 | 2.293 | 2.407 | Xiaoyan Xing (Huawei Media Technology Institute, Tsinghua University), Sibo Feng (Huawei Media Technology Institute), Yanlin Qian (Huawei MultiMedia Team) | 5UT3YUs3 | |
| illumGAN* | 3.191023 | 2.596 | 2.674 | Jianhui Qiu, Shaobing Gao, Rui Yang, Qinyi Jiang (College of Computer Science, SCU) |
kkK1kgdL | |
| PCGAN, MCGAN | 3.301088 | 2.312 | 2.298 | Riccardo Riva, Marco Buzzelli, Simone Bianco, Raimondo Schettini (Imaging and Vision Laboratory, University of Milan - Bicocca) | XMy5NHzY | 2nd |
| PCGAN, MCGAN | 3.376422 | 2.312 | 2.337 | Riccardo Riva, Marco Buzzelli, Simone Bianco, Raimondo Schettini (Imaging and Vision Laboratory, University of Milan - Bicocca) | fYAyFsam | |
| GreyWorld baseline | 4.105811 | 3.673 | 3.545 | |||
| Const baseline | 15.269933 | 14.802 | 15.332 |
Two-illuminant track
| Algorithm | Mean squared | Mean | Median | Trimean | Team | Pastebin link | Prize |
|---|---|---|---|---|---|---|---|
| sde-awb | 31.026217 | 2.751 | 2.262 | 2.290 | Yanlin Qian (Huawei MultiMedia Team) |
5sbE4xMF | 1st |
| sde-awb | 31.542930 | 2.737 | 2.171 | 2.309 | Yanlin Qian (Huawei MultiMedia Team) |
FK19nsTb | |
| AL-AWB* | 33.079119 | 2.657 | 1.844 | 2.082 | Xiaoyan Xing (Huawei Media Technology Institute, Tsinghua University), Sibo Feng(Huawei Media Technology Institute), Yanlin Qian (Huawei MultiMedia Team) | ZLVkhPUp | |
| 3du-awb* | 37.305135 | 2.863 | 2.503 | 2.497 | Yang Liu, Masoumeh Bakhtiariziabari, Gaurav Kudva, Sezer Karaoglu (3DUniversum) Theo Gevers (3DUniversum/University of Amsterdam) |
YrMDaGBx | |
| AL-AWB* | 41.883269 | 2.920 | 2.107 | 2.316 | Xiaoyan Xing (Huawei Media Technology Institute, Tsinghua University), Sibo Feng (Huawei Media Technology Institute), Yanlin Qian (Huawei MultiMedia Team) | AU2dBhZP | |
| GreyWorld baseline | 81.840743 | 4.127 | 3.538 | 3.715 | |||
| Const baseline | 144.745182 | 5.264 | 3.475 | 3.815 |
Rules for participation
UPDATE. The train dataset will be published soon
Registration is open till 13:00 July 31 (GMT+3), 2020. To register, please send an email to chroma@iitp.ru indicating the team members and their affiliations.
Challenge consist of three tracks:
- General case
- Two-illuminant case
- Indoor case
Submission
The submission form is available HERE. You will be able to submit your solution till 13:00 July 31 (GMT+3). In the form you are required to:
- provide contact email address
- list all authors and their affiliations in a free form
- specify a nickname and a brief description of your method
- provide the Pastebin URLs where you have pasted your results
Each team can submit up to 3 solutions for each track. They can be sent in a single Google Form submission or split to up to 9 submissions with various algorithm names and participant lists. The results will be published on August 2nd. As the alternative, you may send email to chroma@iitp.ru with answers till the same deadline, but we highly recommend using the submission form.
Detailed tracks description
For the challenge we have prepared new dataset, which is essentially extention of the CubePlus dataset. It will be published soon.General track
To date, many algorithms for evaluating lighting in the scene have been developed and successfully used. In solving this problem, the key role is played by the representativeness and variability of the dataset. These requirements impose restrictions on their size; it must be large. This will allow you to create stable solutions by analyzing system errors in various categories of images.
At first glance, a good solution is to combine the existing datasets into a single large dataset. However, in this case, it is difficult to distinguish which particular factor influences the behavior of the algorithm: the difficulty of the scene or the difference in camera sensitivity.
Another way to overcome this limitation is to collect a large dataset using the same sensor. To conduct this competition, we went exactly the second way and prepared for the participants of the competition a dataset assembled using one sensor. To analyze the operation of the algorithm, we supplied the images with markup files, which contain not only information about the lighting and shooting parameters, but also information about the contents of the scene.
In the competition of last year, the median was used as the ranking metric. This is a good metric that allows you to ignore dataset layout errors and periodic malfunctions in the algorithm. This year, however, the main goal of this track is to create a reliable solution, the behavior of which, even in extreme cases, is not bad.
Therefore, to assess the quality of the proposed solutions, we will use the average reproduction error for 25% of the worst images (see code).
Indoor track
One of the stand-alone photography categories is indoor photography. Very often, under these conditions, the lighting in the scene is quite complicated: there are many sources of lighting (light from the window, incandescent lamps, LEDs, and so on) in different places of the scene. Under these conditions, the determination of the dominant source in the scene may turn out to be a rather difficult task.
Since this task in itself is difficult for the final quality assessment in this track, we will use the average reproduction error (see code).
Two-illuminant track
In everyday life, there are rarely situations where there is really only one source of lighting in the scene. Even during the day, it is customary to divide the lighting into two sources - the sun and sky. The main question of this track is whether it is possible to reliably extract more information about lighting using a single image. For these purposes, a dataset was assembled using a volumetric color target (SpyderCube) whose faces are illuminated by different sources. This dataset includes images for which the angle between GT is greater than or equal to two degrees.
For this track, the ranking metric is squared sum of two angular reproduction errors. There are two variants to establish the correspondence between answers and ground truths, so we will choose one with minimal error (see code).
Structure of the datasets
All images were taken with the same camera sensor model, with the three cameras Canon 550D x1 and Canon 600D x2.
The image ordering with respect to their creation time. In the lower right corner of each image, the SpyderCube calibration object is placed. Its two neutral 18% gray faces were used to determine the ground-truth illumination for each image. Due to the angle between these two faces, for images with two illuminations, e.g. one in the shadow and one under the direct sunlight, it was possible to simultaneously recover both of them and they are provided for each image.
In all dataset images with two distinct illuminations, one of them is always dominant so that the uniform illumination assumption effectively remains valid. For Indoor Dataset we’ve chosen only images with an angular difference less than 2 degrees between left and right edges of the SpyderCube. For dominant light source was chosen the brighter one. The black level, i.e. the intensity that has to be subtracted from all images in order to use them properly, equals 2048.
All datasets contain files of three types:
- Preview files: name.jpg
- Raw files: name.png. To generate these files we’ve used dcraw library with options -D -4 -T and then applying only simple subsampling for debayering.
- Files with answers and meta-information: name.json. These files contain information about capturing process retrieved from .cr2 files and also our additional markup about scene content.
Provided JSON files have the following field and structure
- EXIF:ApertureValue
- EXIF:ExposureTime
- EXIF:ISO
- EXIF:Model
- EXIF:Orientation
- MakerNotes:InternalSerialNumber
- MakerNotes:LensModel:
- ds_version
- gt - ground-truth chromaticity
- left_gt - left edge of SpyderCube chromaticity
- right_gt - right edge of SpyderCube chromaticity
- properties
- daytime
- has_known_objects. Such as white paper, road signs, etc.
- illumination. Type of dominant illumination in the scene.
- is_sharp - indicating whenever image is sharp or blured.
- light_objects. List of light source in the scene.>
- place. List of three values: indoor, outdoor, unknown.
- richness - Indicating whenever the scene contains only almost single object color or not.
- shadows - Indicating whenever the scene contains shadows or not.
Python script for dataset images reading and visualization is provided here.
Please note, that final solutions for General and Indoor track should be provided in the following form and for Two-illuminant track in the following form.
Bonus data
CubePlus dataset was captured with exactly the same Canon 550D and can be downloaded here. We prepared the contest-like markup for it. The jsons contain extra 'contest_like' field, which is 'false' for images that could be probably filtered out for the challenge.
Interested parties could find it potentially useful to use artificially created data. What they need in that case is the Croatian Paper (CroP) dataset generator described in detail in the paper "CroP: Color Constancy Benchmark Dataset Generator" that is currently available at arXiv. The resources for the generator are available at following link.
Test datasets
The test data format differs slightly from the train one. The SpyderCube rectangle is cut out. JSON files lack ground truth answers and manually annotated properties. Image ids are shuffled.
For each scene, the test datasets have the two files (name.PNG, name.JPG.JSON). name.JPG.JSON contain all information available during the shooting procedure:
- EXIF:ApertureValue
- EXIF:ExposureTime
- EXIF:ISO
- EXIF:Model
- EXIF:Orientation
- MakerNotes:InternalSerialNumber
- MakerNotes:LensModel
Prizes and awards
Winners will receive money prizes:
- First place in any track - $1000
- Second place in any track - $700
- Third place in any track - $300
Also, winners will receive a winner certificate, invitations to the ICMV conference with a speech on their decision, and an opportunity to publish the paper about challenge results.
To receive a money prize, you need to take part in the ICMV conference.
Organizers
Kharkevich Institute for Information Transmission Problems
Kharkevich Institute for Information Transmission Problems
Kharkevich Institute for Information Transmission Problems
Kharkevich Institute for Information Transmission Problems
Kharkevich Institute for Information Transmission Problems
Kharkevich Institute for Information Transmission Problems
Kharkevich Institute for Information Transmission Problems
Gideon Brothers, Croatia
University of Zagreb
University of Zagreb
Image Processing Systems Institute
Sponsorship
Papers
If you use these datasets in your research, please refer to our papers from the list- Ershov, Egor I., A. V. Belokopytov, and A. V. Savchik. "Problems of dataset creation for light source estimation." arXiv preprint arXiv:2006.02692.