Mohedano E, McGuinness K, Giró-i-Nieto X, O'Connor N. Fine-tuning of CNN models for Instance Search with Pseudo-Relevance Feedback. Long Beach, CA, USA: NIPS 2017 Women in Machine Learning Workshop; In Press.  (341.96 KB)


CNN classification models trained on millions of labeled images have been proven to encode “general purpose” descriptors in their intermediate layers. These descriptors are useful for a diverse range of computer vision problems~\cite{1}. However, the target task of these models is substantially different to the instance search task. While classification is concerned with distinguishing between different classes, instance search is concerned with identifying concrete instances of a particular class. 


In this work we propose an unsupervised approach to finetune a model for similarity learning~\cite{2}. For that, we combine two different search engines: one based on off-the-shelf CNN features, and another one on the popular SIFT features. As shown in the figure below, we observe that the information of pre-trained CNN representations and SIFT is in most of the cases complementary, which allows the generation of high quality rank lists. The fusion of the two rankings is used to generate training data for a particular dataset. A pseudo-relevance feedback strategy~\cite{3} is used for sampling images from rankings, considering the top images as positive examples of a particular instance and middle-low ranked images as negative examples.