Abstract: Standardized datasets are of vital importance in multimedia research, as they form the basis for reproducible experiments and evaluations. In the area of video retrieval, widely used datasets such as the IACC [5], which has formed the basis for the TRECVID Ad-Hoc Video Search Task and other retrieval-related challenges, have started to show their age. For example, IACC is no longer representative of video content as it is found in the wild [7]. This is illustrated by the figures below, showing the distribution of video age and duration across various datasets in comparison with a sample drawn from Vimeo and Youtube.

Abstract: In this tutorial we discuss interactive video search tools and methods, review their need in the age of deep learning, and explore video and multimedia search challenges and their role as evaluation benchmarks in the field of multimedia information retrieval. We cover three different campaigns (TRECVID, Video Browser Showdown, and the Lifelog Search Challenge), discuss their goals and rules, and present their achieved findings over the last half-decade. Moreover, we talk about datasets, tasks, evaluation procedures, and examples of interactive video search tools, as well as how they evolved over the years. Participants of this tutorial will be able to gain collective insights from all three challenges and use them for focusing their research efforts on outstanding problems that still remain unsolved in this area.

2019
[3]	Luca Rossetto, Fabian Berns, Klaus Schöffmann, George M. Awad, Christian Beecks, The V3C1 Dataset: Advancing the State of the Art in Video Retrieval, In ACM SIGMM Records, vol. 11, no. 2, 2019. [bib][url] [abstract] Abstract: Standardized datasets are of vital importance in multimedia research, as they form the basis for reproducible experiments and evaluations. In the area of video retrieval, widely used datasets such as the IACC [5], which has formed the basis for the TRECVID Ad-Hoc Video Search Task and other retrieval-related challenges, have started to show their age. For example, IACC is no longer representative of video content as it is found in the wild [7]. This is illustrated by the figures below, showing the distribution of video age and duration across various datasets in comparison with a sample drawn from Vimeo and Youtube.
[2]	Fabian Berns, Luca Rossetto, Klaus Schöffmann, Christian Beecks, George M. Awad, V3C1 Dataset: An Evaluation of Content Characteristics, In Proceedings of the ACM International Conference on Multimedia Retrieval, ACM - New York, New York, NY, pp. 334-338, 2019. [bib][url] [doi]
2018
[1]	Klaus Schöffmann, Werner Bailer, Cathal Gurrin, George M. Awad, Jakub Lokoč, Interactive Video Search: Where is the User in the Age of Deep Learning?, In MM '18 Proceedings of the 26th ACM international conference on Multimedia, ACM Press, New York (NY), pp. 2101-2103, 2018. [bib][url] [doi] [abstract] Abstract: In this tutorial we discuss interactive video search tools and methods, review their need in the age of deep learning, and explore video and multimedia search challenges and their role as evaluation benchmarks in the field of multimedia information retrieval. We cover three different campaigns (TRECVID, Video Browser Showdown, and the Lifelog Search Challenge), discuss their goals and rules, and present their achieved findings over the last half-decade. Moreover, we talk about datasets, tasks, evaluation procedures, and examples of interactive video search tools, as well as how they evolved over the years. Participants of this tutorial will be able to gain collective insights from all three challenges and use them for focusing their research efforts on outstanding problems that still remain unsolved in this area.