资源描述：
《Crowdsourcing for Relevance Evaluation 关联评价的众包》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、PAPERCrowdsourcingforRelevanceEvaluationOmarAlonsoDanielE.RoseBenjaminStewartA9.comPaloAlto,CA{oalonso,danrose,bstewart}@a9.comAbstractRelevanceevaluationisanessentialpartofthedevelopmentandmaintenanceofinformationretrievalsystems.Yettraditionalevaluationapproacheshaveseverallimitations;inparticul
2、ar,conductingneweditorialevaluationsofasearchsystemcanbeveryexpensive.WedescribeanewapproachtoevaluationcalledTERC,basedonthecrowdsourcingparadigm,inwhichmanyonlineusers,drawnfromalargecommunity,eachperformsasmallevaluationtask.1IntroductionRelevanceevaluationforinformationretrievalisanotoriouslyd
3、ifficultandexpensivetask.Intheearlyyearsofthefield,asetofvolunteereditors–oftengraduatestudents–wouldpainstakinglyreadthrougheverydocumentinacorpustodetermineitsrelevancetoasetoftestqueries.Thisprocesswassufficientlydifficultthatonlyafewsmalltestcollections(Cranfield,CACM,etc.)werecreated.Withthea
4、dventofTRECin1992[9],researchershadaccesstotestcollectionswithmillionsoffull-textdocuments.However,thescaleofTRECwasonlypossiblebyeliminatingthenotionthateverydocumentwouldbereadandevaluated.Instead,thepoolingapproachwasdeveloped,inwhichonlythetopNdocumentsretrievedbyatleastoneoftheparticipatingsy
5、stemswereexamined.TheotherfactorthatmadeTRECpossiblewastheavailabilityofalargenumberofprofessionalassessors–retiredintelligenceanalysts–whowerepaidfortheirworkwithfundsfromthesponsoringagencies.WhiletheTRECcollections–andasimportantly,thequerysetsandevaluations–havebeeninvaluableinfurtheringIRrese
6、archoverthepast15years,theystillhavesomelimitations.ThemostobviousoftheseisthatresearchersarelimitedtothetypesofIRtasksthatTREChasstudied.Forexample,ifaresearcherwishestostudysearchinaparticularverticaldomainarea(forexample,ayellow-pages-stylesearchforlocalbusinesses)orexperimentwithanewsearchinte
7、ractionparadigm(forexample,collaborativesearch),thenexistingTRECcollectionsmaynothelp.Furthermore,despitethepresenceofaWebtrack,evaluatinggeneralWebsearchhasuniquechallenges[7],whichoftenrequireanotherapproach.Fo
