How to calculate precision and recall

how to calculate precision and recall example and precision and recall in machine learning
WilliamsMcmahon Profile Pic
WilliamsMcmahon,United States,Professional
Published Date:20-07-2017
Your Website URL(Optional)
Comment
Introduction to Information Retrieval Introduction to Information Retrieval Query expansionIntroduction to Information Retrieval Recap: Unranked retrieval evaluation: Precision and Recall  Precision: fraction of retrieved docs that are relevant = P(relevantretrieved)  Recall: fraction of relevant docs that are retrieved = P(retrievedrelevant) Relevant Nonrelevant Retrieved tp fp Not Retrieved fn tn  Precision P = tp/(tp + fp)  Recall R = tp/(tp + fn) 2Introduction to Information Retrieval Recap: A combined measure: F  Combined measure that assesses precision/recall tradeoff is F measure (weighted harmonic mean): 2 1 (1)PR F 2 1 1  PR  (1) P R  People usually use balanced F measure 1  i.e., with  = 1 or  = ½  Harmonic mean is a conservative average  See CJ van Rijsbergen, Information Retrieval 3Introduction to Information Retrieval This lecture  Improving results  For high recall. E.g., searching for aircraft doesn’t match with plane; nor thermodynamic with heat  Options for improving results…  Global methods  Query expansion  Thesauri  Automatic thesaurus generation  Local methods  Relevance feedback  Pseudo relevance feedbackIntroduction to Information Retrieval Sec. 9.1 Relevance Feedback  Relevance feedback: user feedback on relevance of docs in initial set of results  User issues a (short, simple) query  The user marks some results as relevant or non-relevant.  The system computes a better representation of the information need based on feedback.  Relevance feedback can go through one or more iterations.  Idea: it may be difficult to formulate a good query when you don’t know the collection well, so iterateIntroduction to Information Retrieval Sec. 9.1 Relevance feedback  We will use ad hoc retrieval to refer to regular retrieval without relevance feedback.  We now look at four examples of relevance feedback that highlight different aspects.Introduction to Information Retrieval Similar pagesIntroduction to Information Retrieval Sec. 9.1.1 Relevance Feedback: Example  Image search engine http://nayana.ece.ucsb.edu/imsearch/imsearch.htmlIntroduction to Information Retrieval Sec. 9.1.1 Results for Initial QueryIntroduction to Information Retrieval Sec. 9.1.1 Relevance FeedbackIntroduction to Information Retrieval Sec. 9.1.1 Results after Relevance FeedbackIntroduction to Information Retrieval Ad hoc results for query canine source: Fernando DiazIntroduction to Information Retrieval Ad hoc results for query canine source: Fernando DiazIntroduction to Information Retrieval User feedback: Select what is relevant source: Fernando DiazIntroduction to Information Retrieval Results after relevance feedback source: Fernando DiazIntroduction to Information Retrieval Sec. 9.1.1 Initial query/results  Initial query: New space satellite applications 1. 0.539, 08/13/91, NASA Hasn’t Scrapped Imaging Spectrometer + + 2. 0.533, 07/09/91, NASA Scratches Environment Gear From Satellite Plan 3. 0.528, 04/04/90, Science Panel Backs NASA Satellite Plan, But Urges Launches of Smaller Probes 4. 0.526, 09/09/91, A NASA Satellite Project Accomplishes Incredible Feat: Staying Within Budget 5. 0.525, 07/24/90, Scientist Who Exposed Global Warming Proposes Satellites for Climate Research 6. 0.524, 08/22/90, Report Provides Support for the Critics Of Using Big Satellites to Study Climate 7. 0.516, 04/13/87, Arianespace Receives Satellite Launch Pact From Telesat Canada 8. 0.509, 12/02/87, Telecommunications Tale of Two Companies +  User then marks relevant documents with “+”.Introduction to Information Retrieval Sec. 9.1.1 Expanded query after relevance feedback  2.074 new 15.106 space  30.816 satellite 5.660 application  5.991 nasa 5.196 eos  4.196 launch 3.972 aster  3.516 instrument 3.446 arianespace  3.004 bundespost 2.806 ss  2.790 rocket 2.053 scientist  2.003 broadcast 1.172 earth  0.836 oil 0.646 measureIntroduction to Information Retrieval Sec. 9.1.1 Results for expanded query 1. 0.513, 07/09/91, NASA Scratches Environment Gear From Satellite Plan 2 2. 0.500, 08/13/91, NASA Hasn’t Scrapped Imaging Spectrometer 1 3. 0.493, 08/07/89, When the Pentagon Launches a Secret Satellite, Space Sleuths Do Some Spy Work of Their Own 4. 0.493, 07/31/89, NASA Uses ‘Warm’ Superconductors For Fast Circuit 5. 0.492, 12/02/87, Telecommunications Tale of Two Companies 8 6. 0.491, 07/09/91, Soviets May Adapt Parts of SS-20 Missile For Commercial Use 7. 0.490, 07/12/88, Gaping Gap: Pentagon Lags in Race To Match the Soviets In Rocket Launchers 8. 0.490, 06/14/90, Rescue of Satellite By Space Agency To Cost 90 MillionIntroduction to Information Retrieval Sec. 9.1.1 Key concept: Centroid  The centroid is the center of mass of a set of points  Recall that we represent documents as points in a high-dimensional space  Definition: Centroid   1 (C) d  C dC where C is a set of documents.Introduction to Information Retrieval Sec. 9.1.1 Rocchio Algorithm  The Rocchio algorithm uses the vector space model to pick a relevance feedback query  Rocchio seeks the query q that maximizes opt  q cos(q,(C )) cos(q,(C )) arg max opt r nr  q  Tries to separate docs marked relevant and non-  relevant  1 1 q d d opt j j  C C dC dC r nr j r j r  Problem: we don’t know the truly relevant docs