ImageNet bounding-box auto-annotations with self-assessment =========================================================== A. Vezhnevets, V. Ferrari Introduction ~~~~~~~~~~~~ The large, hierarchical ImageNet [2] dataset presents new challenges and opportunities for computer vision. Although the dataset contains over 14 million images, only a fraction of them has bounding-box annotations (10%). In this package we provide object localization for 0.5 million images from ImageNet together with the self-assessment score, i.e. estimation of their quality. This allows us to return only accurate boxes, automatically discarding the rest. For example, we can automatically return bounding-boxes for 30% of all images with high localization accuracy (i.e.~73% average overlap with ground-truth). Here we provide boxes obtained by AE-GP method trained from "family" source set (solid red curve in figure 7). We plan to release boxes produced by the more advanced AE-GP+ (solid black curve in figure 8) soon. Contents ~~~~~~~~ This package contains: - auto-annotated bounding-boxes for a subset of ImageNet with self-assessment scores - Matlab code to read-in and visualize bounding-boxes Let «dir_root» be the directory where this package was uncompressed. The directory contains: «dir_root»/AEGP_family_boxes_web.mat - a Matlab data file containing the annotations «dir_root»/Demo.m - Matlab code that loads a sample synset image and displays the bounding-box «dir_root»/ImageNet_web_download_demo.m - demo that loads synset from ImageNet and displays auto-annotated bounding-boxes; requires installed ImageNet toolbox. «dir_root»/ImNet_images - directory with a sample synset (just one image) Quick start ~~~~~~~~~~~ You can follow the next steps to check that everything is properly set: 1) start matlab 2) navigate to «dir_root»/ (e.g. by using cd command) 3) execute command: Demo This will load the sample image (provided in this package) and display it with an overlayed bounding-box obtained by [1]. Description of the annotation data structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ AEGP_family_boxes_web.mat contains two cell arrays: synset_boxes - each element corresponds to a synset. It contains an array of structures, corresponding to images of the sysnet. Each structure contains the following fields: box - the predicted bounding-box of the object imid - id of an image in ImageNet format (e.g. n01322604_16) predicted_overlap - the predictive mean of overlap with ground-truth [1] sigma - a standard deviation of the prediction (sec.5.1 in [1]) synset_names - contains a name of the synset, corresponding to the element of synset_boxes Using self-assessment score ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each box has a corresponding predictive mean (.predicted_overlap) and standard diviation (.sigma). The self-assessment score is produced as a combination of both (see [1] sec.5, formula (10)): \eta(w,\lambda) = \mu(w) + \beta(\lambda) \sigma (w) Notice the coefficient beta in front of \sigma. It regulates the degree of desired confidence from self-assessment. It can be derived from \lambda in formula (1) (see sec.3). There are two special cases, when \beta = -1 (\lambda=0.8) and \beta = 0 (\lambda=0.5). If \beta = -1, the self-assessment score \eta can be computed as: >> synset_boxes{1}(1).predicted_overlap - sqrt(synset_boxes{1}(1).sigma); This score equals to the largest overlap a box (from the first image from the first synset) has with at least ~80% probability. If you would like to use boxes that have, greater than 0.5 overlap with at least 80% probability, discard all boxes that have the score above less than 0.5. If \beta = 0, the self-assessment score \eta can be computed as: >> synset_boxes{1}(1).predicted_overlap; This score equals to the largest overlap a box (from the first image from the first synset) has with at least 50% probability. By varying \beta you can regulate how conservative the self-assessment score will be. Downloading images from ImageNet ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, you have to install the toolbox http://image-net.org/download-toolbox and register http://image-net.org/signup. You will obtain your user name and access key, which you should type in ImageNet_web_download_demo.m in lines 6-7. You can then run the script and it will automatically download the images of a synset and display our bounding-boxes for it. Support ~~~~~~~ For any query/suggestion/complaint or simply to say you like/use the annotation and software just drop us an email avezhnev@staffmail.ed.ac.uk vferrari@staffmail.ed.ac.uk References ~~~~~~~~~~ [1] Associative embeddings for large-scale knowledge transfer with self-assessment Alexander Vezhnevets and Vittorio Ferrari Proceedings of IEEE Conference in Computer Vision and Pattern Recognition, June 2014. [2] ImageNet: A large-scale hierarchical image database J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-fei, Proceedings of IEEE Conference in Computer Vision and Pattern Recognition, June 2009. Version History ~~~~~~~~~~~~~~~ Version 1.1 ----------- - updated readme - now in synset_boxes sigma is standard deviation Version 1.0 ----------- - code for loading boxes and downloading images from imagenet - changed data format - added readme Version 0.9 ----------- - initial release, just boxes.