Impact of HIT Design on Crowdsourcing Relevance点击设计对众包关联性的影响

《Impact of HIT Design on Crowdsourcing Relevance点击设计对众包关联性的影响》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、ImpactofHITDesignonCrowdsourcingRelevanceGabriellaKazai1JaapKamps2MarijnKoolen2NatasaMilic-Frayling11MicrosoftResearch,CambridgeUK2UniversityofAmsterdam,TheNetherlandsABSTRACTInthispaperweinvestigatethedesignandimplementationofeectivecrowdsourcingtasks

2、inthecontextofbooksearchevaluation.WeobservetheimpactofaspectsoftheHumanIntelligenceTask(HIT)designonthequalityofrelevancela-belsprovidedbythecrowd.Weassesstheoutputintermsoflabelagreementwithagoldstandarddatasetandob-servetheeectofthecrowdsourcedrelev

3、ancejudgmentsontheresultingsystemrankings.ThisenablesustoobservetheeectofcrowdsourcingontheentireIRevaluationpro-cess.UsingthetestsetandexperimentalrunsfromtheINEX2010BookTrack,wendthatvaryingtheHITde-signandthepoolinganddocumentorderingstrategieslead

4、stoconsiderabledierencesinagreementwiththegoldsetlabels.Wethenobservetheimpactofthecrowdsourcedrel-evancelabelsetsontherelativesystemrankingsusingfourIRperformancemetrics.SystemrankingsbasedonMAPandBprefremainlessaectedbydierentlabelsetswhilethePreci

5、sion@10andnDCG@10leadtodramaticallydif-ferentsystemrankings,especiallyforlabelsacquiredfromHITswithweakerqualitycontrols.Overall,wendthatcrowdsourcingcanbeaneectivetoolfortheevaluationofIRsystems,providedthatcareistakenwhendesigningtheHITs.1.INTRODUCT

6、IONFigure1:PartofaHITshowingquestionseriestoTheevaluationandtuningofInformationRetrieval(IR)solicitrelevancelabelsforbookpagesfromworkerssystemsbasedontheCraneldparadigmrequirespurpose-onAmazonMechanicalTurk:Fulldesign.builttestcollections,attheheartof

7、whichliethehumanrelevancejudgments.Withtheeverincreasingsizeanddi-umentorderingwithinaHITforpresentationtothework-versityofboththedocumentcollectionsandthequerysets,ers.Basedontheanalysisofthecollecteddata,weprovidegatheringrelevancelabelsbyeditorialjud

8、geshasbecomeainsightson1)howdesigndecisionsin uenceboththerawchallenge.Recently,crowdsourcinghasemergedasafea-labelquality,i.e.,agreementwithgoldstandard(GS)ob-sibleapproachtogatheringrelevancedata.However,thetainedfromtraditiona