资源描述：
《Worker Types and Personality Traits in Crowdsourcing Relevance Labels群体相关标签中的员工类型与人格特征》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、WorkerTypesandPersonalityTraitsinCrowdsourcingRelevanceLabelsGabriellaKazaiJaapKampsNatasaMilic-FraylingMicrosoftResearchUniversityofAmsterdamMicrosoftResearchCambridge,UKTheNetherlandsCambridge,UKv-gabkaz@microsoft.comkamps@uva.nlnatasamf@microsoft.comABSTRACT[4,14,19,21],orbyobserving
2、agreementlevelsamongthework-ersandwitha‘goldstandard’setoftrustedlabels[1,6,11,12].Crowdsourcingplatformsofferunprecedentedopportunitiesforcre-Morerecently,increasingattentionisbeingpaidtostudyingchar-atingevaluationbenchmarks,butsufferfromvariedoutputqualityacteristicsoftheworkersthems
3、elvesandhowtheserelatetothefromcrowdworkerswhopossessdifferentlevelsofcompetencequalityoftheirwork.Forexample,[3]experimentwitharangeandaspiration.Thisraisesnewchallengesforqualitycontrolandofhypotheticalassessorarchetypesandexaminetheeffectsofas-requiresanin-depthunderstandingofhowwork
4、ers’characteristicssociatedassessorerrorsontheevaluationofIRsystems.Theyrelatetothequalityoftheirwork.Inthispaper,weusebehavioraldifferentiatebetweenunenthusiastic,optimistic,pessimistic,topic-observations(HITcompletiontime,fractionofusefullabels,labeldisgruntled,lazy,fatiguedand’Markov
5、ian’assessormodelsandaccuracy)todefinefiveworkertypes:Spammer,Sloppy,Incom-usethesemodelstosimulateassessorsgoingthroughtheprocesspetent,Competent,Diligent.UsingdatacollectedfromworkersofmakingjudgmentsintheTRECMillionQuerytracktestcollec-engagedinthecrowdsourcedevaluationoftheINEX2010Boo
6、ktions.Theworkof[20]presentsaclassificationofcrowdwork-TrackProveIttask,werelatetheworkertypestolabelaccuracyers,differentiatingbetweentwotypesofethicalworkers(sloppyandpersonalitytraitinformationalongthe‘BigFive’personalityandproper),andtwotypesofunethicalworkers(randomspam-dimensions.W
7、eexpectthatthesenewinsightsaboutthetypesofmersanduniformspammers).Atotalof850votesobtainedfromcrowdworkersandthequalityoftheirworkwillinformhowto33crowdworkersfor10topicsfromtheTREC2010WebTrackdesignHITstoattractthebestworkerstoataskandexplainwhyad-hoctaskareusedtoconfirmthiscla
