Evaluation in Music Information Retrieval

TitleEvaluation in Music Information Retrieval
Publication TypeJournal Article
Year of Publication2013
AuthorsUrbano, J., Schedl M., & Serra X.
Journal TitleJournal of Intelligent Information Systems
Volume41
Pages345-369
KeywordsEvaluation and Experimentation, music information retrieval, Survey, Text Information Retrieval
AbstractThe field of Music Information Retrieval has always acknowledged the need for rigorous scientifi c evaluations, and several ef forts have set out to develop and provide the infrastructure, technology and methodologies needed to carry out these evaluations. The community has enormously gained from these evaluation forums, but we have reached a point where we are stuck with evaluation frameworks that do not allow us to improve as much and as well as we want. The community recently acknowledged this problem and showed interest in addressing it, though it is not clear what to do to improve the situation. We argue that a good place to start is again the Text IR field. Based on a formalization of the evaluation process, this paper presents a survey of past evaluation work in the context of Text IR, from the point of view of validity, reliability and eficiency of the experiments. We show the problems that our community currently has in terms of evaluation, point to several lines of research to improve it and make various proposals in that line.
Final publicationhttps://doi.org/10.1007/s10844-013-0249-4
intranet