Accurately Interpreting Clickthrough Data as Implicit Feedback

Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05 Introduction Adapt a retrieval system to users and or collections Manual adaptation - time consuming or

even impractical Explore and evaluate implicit feedback Use clickthrough data in WWW search User Study Record and evaluate user actions Provide insight into the decision process Record users eye movements : Eye tracking

Questions used Two Phases of the study Phase 34 participants Start search with Google query, search for answers Phase

I II Investigate how users react to manipulations of search results Same instructions as phase I Each subject assigned to one of three experimental

conditions Normal, Swapped, Reversed Explicit Relevance Judgments Collected explicit relevance judgments for all queries and results pages Inter-judge

agreements Analysis of user behavior Which Do links do users view and click? users scan links from top to bottom? Which

links do users evaluate before clicking? Which links do users view and click? Almost equal frequency of 1st and 2nd link, but more clicks on 1st link Once the user has started scrolling, rank appears to

become less of an influence Do users scan links from top to bottom? Big gap before viewing 3rd ranked abstract Users scan viewable results thoroughly before scrolling Which links do users evaluate before

clicking? Abstracts closer above the clicked link are more likely to be viewed Abstract right below a link is viewed roughly 50% of the time Analysis of Implicit Feedback Does Are

relevance influence user decisions? clicks absolute relevance judgments? Does relevance influence user decisions? Yes Use the reversed condition

Controllably decreases the quality of the retrieval function and relevance of highly ranked abstracts Users react in two ways View lower ranked links more frequently, scan

significantly more abstracts Subjects are much less likely to click on the first link, more likely to click on a lower ranked link Clicks = absolute relevance judgments? Interpretation Trust is problematic Bias

Abstract ranked first receives more clicks than the second First link is more relevant (not influenced by order of presentation) or Users prefer the first link due to some level of trust in the search engine (influenced by order of presentation)

Trust Bias Hypothesis that users are not influenced by presentation order can be rejected Users have substantial trust in search engines ability to estimate relevance Quality Bias Quality

of the ranking influences the users clicking behavior If relevance of retrieved results decreases, users click on abstracts that are on average less relevant Confirmed by the reversed condition Are clicks relative relevance judgments?

An accurate interpretation of clicks needs to take into consideration Users trust into quality of search engine Quality of retrieval function itself Difficult to measure explicitly Interpret clicks as pairwise preference

statements Strategy 1 Takes trust and quality bias into consideration Substantially and significantly better than random Close in accuracy to inter judge agreement Strategy 2

Slightly more accurate than Strategy 1 Not a significant difference in Phase II Strategy 3 Accuracy worse than Strategy 1 Ranking quality has an effect on the accuracy

Strategy 4 No significant differences compared to Strategy 1 Strategy 5 Highly accurate in the normal condition

Misleading Aligned preferences probably less valuable for learning Better results even if user behaves randomly Less accurate than Strategy 1 in the reversed condition Conclusion

Users clicking decisions influenced by search bias and quality bias Strategies for generating relative relevance feedback signals Implicit relevance signals are less consistent with explicit judgments than the explicit judgments among each other Encouraging results

Recently Viewed Presentations

  • Apo Ibuprofen 600 Mg Tablet - Does Ibuprofen 800 Mg Get You High

    Apo Ibuprofen 600 Mg Tablet - Does Ibuprofen 800 Mg Get You High

    Apo Ibuprofen 600 Mg Tablet - Does Ibuprofen 800 Mg Get You High Description: Concentration of ibuprofen in advil, ibuprofen 600 mg tablets bp, is aleve ibuprofen or tylenol, ibuprofen 400 buy online, ibuprofen 400 mg tablets dosage, ibuprofen 400...
  • Excretory System

    Excretory System

    EXCRETORY SYSTEM PP 993-998 FUNCTION Every cell in the body produces metabolic waste. The skin, lungs, and kidneys make up this system. SKIN Excretes water, salts, small amounts of nitrogenous wastes LUNGS Excretes water vapor and CO2 KIDNEYS Formation and...
  • www.noaasis.noaa.gov

    www.noaasis.noaa.gov

    NASA SPoRT (Geoffrey Stano, Kevin McGrath) NWS/NCEP Aviation Weather Center (Josh Scheck, Amanda Terborg) ... GOES-16 GLM FED in AWIPS/CAVE/D2D. Initial GOES-17 GLM FED implementation in AWIPS will be based on above. Next Steps for GLM Implementation Across NWS (1...
  • Mozart-effekten - Fup Eller Fakta?

    Mozart-effekten - Fup Eller Fakta?

    "It is as if the mind and feelings vibrate at a certain frequency and are most satisfied with music that is attuned to that frequency" (Helen Bonny) "Music is your only friend - until the end" (Jim Morrison) E. Clarke:...
  • Présentation PowerPoint

    Présentation PowerPoint

    JAR OPS 1.035 & 1.037 FAA FOQA The level of processing of the recorded information is not specified A wide variety of solutions is available: From simple grids of 12 to 20 parameters up to complete flight reconstruction and automated...
  • Chapter 3 Settling the Northern Colonies

    Chapter 3 Settling the Northern Colonies

    Chapter 3 Settling the Northern ColoniesVUS.2a impact of age of exploration by describing the characteristics of early exploration and evaluating the impact of European settlement in the Americas pg. 14
  • Types of Economic Systems - Council Rock School District

    Types of Economic Systems - Council Rock School District

    Traditional Economy: system in which economic decisions are based on customs & beliefs that have been handed down from generation to generation "Things are done the way they always have been done" Economic decisions are based on customs & beliefs...
  • Infants and Children: Consent and Issues at End of Life

    Infants and Children: Consent and Issues at End of Life

    Infants and Children: Consent and Legal Issues at End of Life . Catherine Ghent. Partner Gallagher Shatter Solicitors