资源描述：
《Crowdsourcing Inference-Rule Evaluation众包推理规则评价》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、CrowdsourcingInference-RuleEvaluationNaomiZeichnerJonathanBerantIdoDaganBar-IlanUniversityTel-AvivUniversityBar-IlanUniversityRamat-Gan,IsraelTel-Aviv,IsraelRamat-Gan,Israelzeichner.naomi@gmail.comjonatha6@post.tau.ac.ildagan@cs.biu.ac.ilAbstractandgenerateknowledgeresourcesforinferencesy
2、s-tems.However,despitetheirpotential,utilizationofTheimportanceofinferencerulestosemanticinferenceruleresourcesiscurrentlysomewhatlim-applicationshaslongbeenrecognizedandex-ited.Thisislargelyduetothefactthattheseal-tensiveworkhasbeencarriedouttoautomat-gorithmsoftenproduceinvalidrules.Thu
3、s,evalu-icallyacquireinference-ruleresources.How-ationisnecessarybothforresourcedevelopersasever,evaluatingsuchresourceshasturnedoutwellasforinferencesystemdeveloperswhowanttotobeanon-trivialtask,slowingprogressinthefield.Inthispaper,wesuggestaframeworkassesthequalityofeachresource.Unfortu
4、nately,asforevaluatinginference-ruleresources.Ourevaluatinginferencerulesishardandcostly,thereisframeworksimplifiesapreviouslyproposednoclearevaluationstandard,andthishasbecomea“instance-basedevaluation”methodthatin-slowingfactorforprogressinthefield.volvedsubstantialannotatortraining,makin
5、gOneoptionforevaluatinginferenceruleresourcesitsuitableforcrowdsourcing.Weshowthatourmethodproducesalargeamountofan-istomeasuretheirimpactonanendtask,asthatisnotationswithhighinter-annotatoragreementwhatultimatelyinterestsaninferencesystemdevel-foralowcostatashortperiodoftime,withoutoper.
6、However,thisisoftenproblematicsinceinfer-requiringtrainingexpertannotators.encesystemshavemanycomponentsthataddressmultiplephenomena,andthusitishardtoassesstheeffectofasingleresource.AnexampleistheRecog-1IntroductionnizingTextualEntailment(RTE)framework(DaganInferencerulesareanimportantco
7、mponentinse-etal.,2009),inwhichgivenatextTandatextualmanticapplications,suchasQuestionAnsweringhypothesisH,asystemdetermineswhetherHcan(QA)(RavichandranandHovy,2002)andInforma-beinferredfromT.Thistypeofevaluationwases-tionExtraction(IE)(ShinyamaandSekine,2006),tabli
