Boolean model Information retrieval example

boolean retrieval model for information retrieval and boolean retrieval model maintains the term frequency
WilliamsMcmahon Profile Pic
WilliamsMcmahon,United States,Professional
Published Date:20-07-2017
Your Website URL(Optional)
Comment
Introduction to Information Retrieval Introduction to Information Retrieval Boolean Retrieval 1Introduction to Information Retrieval Take-away  Administrativa  Boolean Retrieval: Design and data structures of a simple information retrieval system  What topics will be covered in this class? 2 2Introduction to Information Retrieval Definition of information retrieval Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers). 4 4Introduction to Information Retrieval 6 6Introduction to Information Retrieval Boolean retrieval  The Boolean model is arguably the simplest model to base an information retrieval system on.  Queries are Boolean expressions, e.g., CAESAR AND BRUTUS  The seach engine returns all documents that satisfy the  Boolean expression. Does Google use the Boolean model? 7 7Introduction to Information Retrieval Unstructured data in 1650  Which plays of Shakespeare contain the words BRUTUS AND CAESAR, but not CALPURNIA?  One could grep all of Shakespeare’s plays for BRUTUS and CAESAR, then strip out lines containing CALPURNIA  Why is grep not the solution?  Slow (for large collections)  grep is line-oriented, IR is document-oriented  “NOT CALPURNIA” is non-trivial  Other operations (e.g., find the word ROMANS near COUNTRYMAN ) not feasible 10 10Introduction to Information Retrieval Term-document incidence matrix Anthony Julius The Hamlet Othello Macbeth and Caesar Tempest . . . Cleopatra ANTHONY 1 1 0 0 0 1 BRUTUS 1 1 0 1 0 0 CAESAR 1 1 0 1 1 1 CALPURNIA 0 1 0 0 0 0 CLEOPATRA 1 0 0 0 0 0 MERCY 1 0 1 1 1 1 WORSER 1 0 1 1 1 0 . . . Entry is 1 if term occurs. Example: CALPURNIA occurs in Julius Caesar. Entry is 0 if term doesn’t occur. Example: CALPURNIA doesn’t occur in The tempest. 11 11Introduction to Information Retrieval Incidence vectors  So we have a 0/1 vector for each term.  To answer the query BRUTUS AND CAESAR AND NOT CALPURNIA:  Take the vectors for BRUTUS, CAESAR AND NOT CALPURNIA  Complement the vector of CALPURNIA  Do a (bitwise) and on the three vectors  110100 AND 110111 AND 101111 = 100100 12 12Introduction to Information Retrieval 0/1 vector for BRUTUS Anthony Julius The Hamlet Othello Macbeth and Caesar Tempest . . . Cleopatra ANTHONY 1 1 0 0 0 1 BRUTUS 1 1 0 1 0 0 CAESAR 1 1 0 1 1 1 CALPURNIA 0 1 0 0 0 0 CLEOPATRA 1 0 0 0 0 0 MERCY 1 0 1 1 1 1 WORSER 1 0 1 1 1 0 . . . result: 1 0 0 1 0 0 13 13Introduction to Information Retrieval Answers to query Anthony and Cleopatra, Act III, Scene ii Agrippa Aside to Domitius Enobarbus: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain. Hamlet, Act III, Scene ii Lord Polonius: I did enact Julius Caesar: I was killed i’ the Capitol; Brutus killed me. 14 14Introduction to Information Retrieval Bigger collections 6  Consider N = 10 documents, each with about 1000 tokens 9 ⇒ total of 10 tokens  On average 6 bytes per token, including spaces and 9  punctuation⇒ size of document collection is about 6 ・ 10 = 6 GB  Assume there are M = 500,000 distinct terms in the collection  (Notice that we are making a term/token distinction.) 15 15Introduction to Information Retrieval Can’t build the incidence matrix 6  M = 500,000 × 10 = half a trillion 0s and 1s.  But the matrix has no more than one billion 1s.  Matrix is extremely sparse.  What is a better representations?  We only record the 1s. 16 16Introduction to Information Retrieval Inverted Index For each term t, we store a list of all documents that contain t. dictionary postings 17 17Introduction to Information Retrieval Inverted Index For each term t, we store a list of all documents that contain t. dictionary postings 18 18Introduction to Information Retrieval Inverted Index For each term t, we store a list of all documents that contain t. dictionary postings 19 19Introduction to Information Retrieval Inverted index construction ❶ Collect the documents to be indexed: ❷ Tokenize the text, turning each document into a list of tokens: ❸ Do linguistic preprocessing, producing a list of normalized tokens, which are the indexing terms: ❹ Index the documents that each term occurs in by creating an inverted index, consisting of a dictionary and postings. 20Introduction to Information Retrieval Tokenizing and preprocessing 21 21Introduction to Information Retrieval Split the result into dictionary and postings file dictionary postings 25 25Introduction to Information Retrieval Later in this course  Index construction: how can we create inverted indexes for large collections?  How much space do we need for dictionary and index?  Index compression: how can we efficiently store and process indexes for large collections?  Ranked retrieval: what does the inverted index look like when we want the “best” answer? 26 26Introduction to Information Retrieval Simple conjunctive query (two terms)  Consider the query: BRUTUS AND CALPURNIA  To find all matching documents using inverted index: ❶ Locate BRUTUS in the dictionary ❷ Retrieve its postings list from the postings file ❸ Locate CALPURNIA in the dictionary ❹ Retrieve its postings list from the postings file ❺ Intersect the two postings lists ❻ Return intersection to user 28 28