Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

@article{alonso2008crowdsourcing, author = {Alonso, Omar and Rose, Daniel E. and Stewart, Benjamin}, title = {Crowdsourcing for relevance evaluation}, journal = {SIGIR Forum}, publisher = {ACM}, address = {New York, NY, USA}, year = {2008}, volume = {42}, number = {2}, pages = {9--15}, url = {http://doi.acm.org/10.1145/1480506.1480508}, doi = {10.1145/1480506.1480508}, issn = {0163-5840}, keywords = {crowdsourcing, evaluation, ir, relevance}, abstract = {Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.} }

%0 = article %A = Alonso, Omar and Rose, Daniel E. and Stewart, Benjamin %C = New York, NY, USA %D = 2008 %I = ACM %T = Crowdsourcing for relevance evaluation %U = http://doi.acm.org/10.1145/1480506.1480508

Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.

@article{jarvelin2002cumulated, author = {Järvelin, Kalervo and Kekäläinen, Jaana}, title = {Cumulated gain-based evaluation of IR techniques}, journal = {ACM Transactions on Information Systems}, publisher = {ACM}, address = {New York, NY, USA}, year = {2002}, volume = {20}, number = {4}, pages = {422--446}, url = {http://portal.acm.org/citation.cfm?id=582418}, doi = {10.1145/582415.582418}, issn = {1046-8188}, keywords = {evaluation, information, ir, retrieval}, abstract = {Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.} }

%0 = article %A = Järvelin, Kalervo and Kekäläinen, Jaana %C = New York, NY, USA %D = 2002 %I = ACM %T = Cumulated gain-based evaluation of IR techniques %U = http://portal.acm.org/citation.cfm?id=582418

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In-Query1) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.

@inproceedings{jarvelin2000ir, author = {Järvelin, Kalervo and Kekäläinen, Jaana}, title = {IR evaluation methods for retrieving highly relevant documents}, booktitle = {SIGIR '00: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval}, publisher = {ACM}, address = {New York, NY, USA}, year = {2000}, pages = {41--48}, url = {http://portal.acm.org/citation.cfm?id=345545}, doi = {10.1145/345508.345545}, isbn = {1-58113-226-3}, keywords = {evaluation, information, ir, retrieval}, abstract = {This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In-Query1) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.} }

%0 = inproceedings %A = Järvelin, Kalervo and Kekäläinen, Jaana %B = SIGIR '00: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval %C = New York, NY, USA %D = 2000 %I = ACM %T = IR evaluation methods for retrieving highly relevant documents %U = http://portal.acm.org/citation.cfm?id=345545

Spreading activation methods have been recommended in information retrieval to expand the search vocabulary and to complement the retrieved document sets. The spreading activation strategy is reminiscent of earlier associative indexing and retrieval systems. Some spreading activation procedures are briefly described, and evaluation output is given, reflecting the effectiveness of one of the proposed procedures.

@inproceedings{salton1988spreading, author = {Salton, G. and Buckley, C.}, title = {On the use of spreading activation methods in automatic information}, booktitle = {SIGIR '88: Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval}, publisher = {ACM Press}, address = {New York, NY, USA}, year = {1988}, pages = {147--160}, url = {http://portal.acm.org/citation.cfm?id=62447&dl=ACM&coll=GUIDE}, doi = {http://doi.acm.org/10.1145/62437.62447}, isbn = {2-7061-0309-4}, keywords = {***, activation, evaluation, ir, msn, network, semantic, spreading}, abstract = {Spreading activation methods have been recommended in information retrieval to expand the search vocabulary and to complement the retrieved document sets. The spreading activation strategy is reminiscent of earlier associative indexing and retrieval systems. Some spreading activation procedures are briefly described, and evaluation output is given, reflecting the effectiveness of one of the proposed procedures.} }

%0 = inproceedings %A = Salton, G. and Buckley, C. %B = SIGIR '88: Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval %C = New York, NY, USA %D = 1988 %I = ACM Press %T = On the use of spreading activation methods in automatic information %U = http://portal.acm.org/citation.cfm?id=62447&dl=ACM&coll=GUIDE