TREC: Experiment and Evaluation in Information Retrieval

Description:

The Text REtrieval Conference (TREC), a yearly workshop hosted by the US government's National Institute of Standards and Technology, provides the infrastructure necessary for large-scale evaluation of text retrieval methodologies. With the goal of accelerating research in this area, TREC created the first large test collections of full-text documents and standardized retrieval evaluation. The impact has been significant; since TREC's beginning in 1992, retrieval effectiveness has approximately doubled. TREC has built a variety of large test collections, including collections for such specialized retrieval tasks as cross-language retrieval and retrieval of speech. Moreover, TREC has accelerated the transfer of research ideas into commercial systems, as demonstrated in the number of retrieval techniques developed in TREC that are now used in Web search engines.

This book provides a comprehensive review of TREC research, summarizing the variety of TREC results, documenting the best practices in experimental information retrieval, and suggesting areas for further research. The first part of the book describes TREC's history, test collections, and retrieval methodology. Next, the book provides "track" reports -- describing the evaluations of specific tasks, including routing and filtering, interactive retrieval, and retrieving noisy text. The final part of the book offers perspectives on TREC from such participants as Microsoft Research, University of Massachusetts, Cornell University, University of Waterloo, City University of New York, and IBM. The book will be of interest to researchers in information retrieval and related technologies, including natural language processing.

Ellen M. Voorhees is Computer Scientist at the National Institute of Standards and Technology (NIST).

Donna K. Harman is Group Leader at the National Institute of Standards and Technology (NIST).

Table of Contents:

Preface	vii
	List of Conference Proceedings	ix
I	INTRODUCTION
1	The Text REtrieval Conference Ellen M. Voorhees and Donna K. Harman	3
2	The TREC Test Collections Donna K. Harman	21
3	Retrieval System Evaluation Chris Buckley and Ellen M. Voorhees	53
II	SELECTED TRACK REPORTS
4	The TREC Ad Hoc Experiments Donna K. Harman	79
5	Routing and Filtering Stephen Robertson and Jamie Callan	99
6	The TREC Interactive Tracks: Putting the User into Search Susan T. Dumais and Nicholas J. Belkin	123
7	Beyond English Donna K. Harman	153
8	Retrieving Noisy Text Ellen M. Voorhees and John S. Garofolo	183
9	The Very Large Collection and Web Tracks David Hawking and Nick Craswell	199
10	Question Answering in TREC Ellen M. Voorhees	233
III	SELECTED PARTICIPANT REPORTS
11	The University of Massachusetts and a Dozen TRECs James Allan, W. Bruce Croft and Jamie Callan	261
12	How Okapi Came to TREC Stephen Robertson	287
13	The SMART Project at TREC Chris Buckley	301
14	Ten Years of Ad Hoc Retrieval at TREC Using PIRCS Kui-Lam Kwok	321
15	MultiText Experiments for TREC Gordon V. Cormack, Charles L. A. Clarke, Christopher R. Palmer and Thomas R. Lynam	347
16	A Language-Modeling Approach to TREC Djoerd Hiemstra and Wessel Kraaij	373
17	IBM Research Activities at TREC Eric W. Brown, David Carmel, Martin Franz, Abraham Ittycheriah, Tapas Kanungo, Yoelle Maarek, J. Scott McCarley, Robert L. Mack, John M. Prager, John R. Smith, Aya Soffer, Jason Y. Zien and Alan D. Marwick	397
	Epilogue: Metareflections on TREC Karen Sparck Jones	421
	List of Contributors	449
	Index	451