资源描述：
《chinese word segmentation as character taggin》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、ComputationalLinguisticsandChineseLanguageProcessingVol.8,No.1,February2003,pp.29-4829ÓTheAssociationforComputationalLinguisticsandChineseLanguageProcessingChineseWordSegmentationasCharacterTagging*NianwenXueAbstractInthispaperwereportresultsofasupervisedmachine-learningapproa
2、chtoChinesewordsegmentation.AmaximumentropytaggeristrainedonmanuallyannotateddatatoautomaticallyassigntoChinesecharacters,orhanzi,tagsthatindicatethepositionofahanziwithinaword.Thetaggedoutputisthenconvertedintosegmentedtextforevaluation.Preliminaryresultsshowthatthisapproachi
3、scompetitiveagainstothersupervisedmachine-learningsegmentersreportedinpreviousstudies,achievingprecisionandrecallratesof95.01%and94.94%respectively,trainedona237K-wordtrainingset.KeyWords:Chinesewordsegmentation,supervisedmachine-learning,maximumentropy,charactertagging1.Intro
4、ductionItisgenerallyagreedamongresearchersthatwordsegmentationisanecessaryfirststepinChineselanguageprocessing.However,unlikeEnglishtextinwhichsentencesaresequencesofwordsdelimitedbywhitespaces,inChinesetext,sentencesarerepresentedasstringsofChinesecharactersorhanziwithoutsimi
5、larnaturaldelimiters.Therefore,thefirststepinaChineselanguageprocessingtaskistoidentifythesequenceofwordsinasentenceandmarkboundariesinappropriateplaces.ThismaysoundsimpleenoughbutinrealityidentifyingwordsinChineseisanon-trivialproblemthathasdrawnalargebodyofresearchintheChine
6、selanguageprocessingcommunity[FanandTsai,1988;Gan,1995;Gan,Palmer,andLua,1996;Guo,1997;JinandChen,1998;SproatandShih,1990;Sproatetal.,1996;WuandJiang,1998;Wu,2003].Itiseasytodemonstratethatthelackofnaturaldelimitersitselfisnottheheartoftheproblem.Inahypotheticallanguagewhereal
7、lwordsarerepresentedwithafinitesetofsymbols,ifonesubsetofthesymbolsalwaysstartawordandanothersubset,mutuallyexclusivefromtheprevioussubset,alwaysendaword,identifyingwordswouldbeatrivial*InstituteforResearchinCognitiveScience,Suite400A,3401WalnutStreetUniversityofPennsylvania,P
8、hiladelphia,PA19104,USAE-mail:xueniwen@linc.cis.upenn.edu30Ni
