剑桥Information retrieval(信息检索)课件108eval.pdf

《剑桥Information retrieval(信息检索)课件108eval.pdf》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、DRAFT!©April1,2009CambridgeUniversityPress.Feedbackwelcome.1518EvaluationininformationretrievalWehaveseenintheprecedingchaptersmanyalternativesindesigninganIRsystem.Howdoweknowwhichofthesetechniquesareeffectiveinwhichapplications?Shouldweusestoplists?Shouldwestem?Shouldweusein-versedocume

2、ntfrequencyweighting?Informationretrievalhasdevelopedasahighlyempiricaldiscipline,requiringcarefulandthoroughevaluationtodemonstratethesuperiorperformanceofnoveltechniquesonrepresentativedocumentcollections.InthischapterwebeginwithadiscussionofmeasuringtheeffectivenessofIRsystems(Section8

3、.1)andthetestcollectionsthataremostoftenusedforthispurpose(Section8.2).Wethenpresentthestraightforwardnotionofrelevantandnonrelevantdocumentsandtheformalevaluationmethodol-ogythathasbeendevelopedforevaluatingunrankedretrievalresults(Sec-tion8.3).Thisincludesexplainingthekindsofevaluationm

4、easuresthatarestandardlyusedfordocumentretrievalandrelatedtasksliketextclas-sificationandwhytheyareappropriate.Wethenextendthesenotionsanddevelopfurthermeasuresforevaluatingrankedretrievalresults(Section8.4)anddiscussdevelopingreliableandinformativetestcollections(Section8.5).Wethenstepbac

5、ktointroducethenotionofuserutility,andhowitisap-proximatedbytheuseofdocumentrelevance(Section8.6).Thekeyutilitymeasureisuserhappiness.Speedofresponseandthesizeoftheindexarefactorsinuserhappiness.Itseemsreasonabletoassumethatrelevanceofresultsisthemostimportantfactor:blindinglyfast,useless

6、answersdonotmakeauserhappy.However,userperceptionsdonotalwayscoincidewithsystemdesigners’notionsofquality.Forexample,userhappinesscommonlydependsverystronglyonuserinterfacedesignissues,includingthelayout,clarity,andresponsivenessoftheuserinterface,whichareindependentofthequalityoftheresul

7、tsreturned.Wetouchonothermeasuresofthequal-ityofasystem,inparticularthegenerationofhigh-qualityresultsummarysnippets,whichstronglyinfluenceuserutility,butarenotmeasuredinthebasicrelevancerankingparadigm(Section8.7).Onlineedition(c)2009CambridgeUP1528Evaluationininfor