资源描述：
《Assessing Crowdsourcing Quality through Objective Tasks通过目标任务评估众包质量》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、AssessingCrowdsourcingQualitythroughObjectiveTasksAhmetAker,MahmoudEl-Hajy,M-DyaaAlbakoury,UdoKruschwitzyDepartmentofComputerScience,UniversityofSheffield,UnitedKingdomySchoolofComputerScienceandElectronicEngineering,UniversityofEssex,UnitedKingdomEmail:a.aker@dcs.shef.ac.
2、uk,melhaj@essex.ac.uk,malbak@essex.ac.uk,udo@essex.ac.ukAbstractTheemergenceofcrowdsourcingasacommonlyusedapproachtocollectvastquantitiesofhumanassessmentsonavarietyoftasksrepresentsnothinglessthanaparadigmshift.Thisisparticularlytrueinacademicresearchwhereithassuddenlybeco
3、mepossibletocollect(high-quality)annotationsrapidlywithouttheneedofanexpert.InthispaperweinvestigatefactorswhichcaninfluencethequalityoftheresultsobtainedthroughAmazon’sMechanicalTurkcrowdsourcingplatform.Weinvestigatedtheimpactofdifferentpresentationmethods(freetextversusra
4、diobuttons),workers’base(USAversusIndiaasthemainbasesofMTurkworkers)andpaymentscale(about$4,$8and$10perhour)onthequalityoftheresults.Foreachrunweassessedtheresultsprovidedby25workersonasetof10tasks.Weruntwodifferentexperimentsusingobjectivetasks:mathsandgeneraltextquestions
5、.Inbothtaskstheanswersareunique,whicheliminatestheuncertaintyusuallypresentinsubjectivetasks,whereitisnotclearwhethertheunexpectedansweriscausedbyalackofworker’smotivation,theworker’sinterpretationofthetaskorgenuineambiguity.Inthisworkwepresentourresultscomparingtheinfluence
6、ofthedifferentfactorsused.Oneoftheinterestingfindingsisthatourresultsdonotconfirmpreviousstudieswhichconcludedthatanincreaseinpaymentattractsmorenoise.Wealsofindthatthecountryoforiginonlyhasanimpactinsomeofthecategoriesandonlyingeneraltextquestionsbutthereisnosignificantdiffere
7、nceatthetoppay.1.Introductioningplatform,weinvestigatedtheimpactofdiffer-entpresentationmethods,workers’baseandpay-Crowdsourcinghasbecomeaseriousalternativementscaleonthequalityoftheresults.Foreachincreatingresourcesfornaturallanguagepro-runweassessedtheresultsprovidedby25w
8、ork-cessing,e.g.(KaisserandLowe,2008;Alonsoersonasetof10tasks.Forourexperimentswea
