Evaluating Information Retrieval: Precision, Recall, and the Trade off between Them

Abstract

ABSTRACT An information retrieval system, such as a search engine, answers a query by returning the items it judges relevant from a large collection, and how good it is depends on two things: whether the items it returns are actually relevant, and whether it returns the relevant items there are; these are measured by precision and recall, and the two are usually in tension, so that improving one tends to worsen the other. This study examines the evaluation of information retrieval through precision and recall and the trade-off between them. Precision, the fraction of the returned items that are relevant, and recall, the fraction of the relevant items that are returned, were examined together as a retrieval system was made to return more or fewer items, so that the trade-off between them, and how it shapes evaluation, could be seen. Precision and recall traded off against each other: when the system returned only the few items it was most confident in, those were mostly relevant, giving high precision, but many relevant items were left out, giving low recall; when it returned many items so as to catch the relevant ones, recall rose, but more irrelevant items were included, so precision fell. Plotting precision against recall traced a curve that fell as recall rose, showing that high precision and high recall could not generally be achieved at once. Which to favour depends on the task: where it matters that what is returned is relevant, precision is preferred, while where it matters that nothing relevant is missed, recall is preferred, and a single operating point on the curve is chosen accordingly. Because neither alone tells the whole story, the two are reported together or combined into a single measure. The study shows that precision and recall trade off in information retrieval, and it discusses how this governs the evaluation and tuning of search systems.

KEYWORDS: Information retrieval, Precision, Recall, Search evaluation, Precision-recall trade-off, Relevance, Search engines, F-measure, Information systems