Very Deep Convolutional Networks for Natural Language Processing

《Very Deep Convolutional Networks for Natural Language Processing》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、VeryDeepConvolutionalNetworksforNaturalLanguageProcessingAlexisConneauHolgerSchwenkYannLeCunFacebookAIResearchFacebookAIResearchFacebookAIResearchaconneau@fb.comschwenk@fb.comyann@fb.comLoïcBarreauLIUM,UniversityofLeMans,Franceloic.barrault@univ-lemans.frAbstract

2、ThedominantapproachformanyNLPtasksarerecurrentneuralnetworks,inpar-ticularLSTMs,andconvolutionalneuralnetworks.However,thesearchitecturesarerathershallowincomparisontothedeepconvolutionalnetworkswhichareverysuccessfulincomputervision.Wepresentanewarchitecturefort

3、extprocess-ingwhichoperatesdirectlyonthecharacterlevelandusesonlysmallconvolutionsandpoolingoperations.Weareabletoshowthattheperformanceofthismodelincreaseswiththedepth:usingupto29convolutionallayers,wereportsignificantimprovementsoverthestate-of-the-artonseveralp

4、ublictextclassificationtasks.Tothebestofourknowledge,thisisthefirsttimethatverydeepconvolutionalnetshavebeenappliedtoNLP.1IntroductionThegoalofnaturallanguageprocessing(NLP)istoprocesstextwithcomputersinordertoanalyzeit,toextractinformationandeventuallytorepresentt

5、hesameinformationdifferently.Wemaywanttoassociatecategoriestopartsofthetext(e.g.POStaggingorsentimentanalysis),structuretextdifferently(e.g.parsing),orconvertittosomeotherformwhichpreservesallorpartofthecontentarXiv:1606.01781v1[cs.CL]6Jun2016(e.g.machinetranslat

6、ion,summarization).Thelevelofgranularityofthisprocessingcanrangefromindividualcharactersorwordsuptowholesentencesorevenparagraphs.Afteracoupleofpioneerworks([2,3,4]amongothers),theuseofneuralnetworksforNLPapplica-tionsisattractinghugeinterestintheresearchcommunit

7、yandtheyaresystematicallyappliedtoallNLPtasks.However,whiletheuseof(deep)neuralnetworksinNLPhasshownverygoodresultsformanytasks,itseemsthattheyhavenotyetreachedtheleveltooutperformthestate-of-the-artbyalargemargin,asitwasobservedincomputervisionandspeechrecogniti

8、on.Convolutionalneuralnetworks,inshortConvNets,areverysuccessfulincomputervision.Inearlyapproachestocomputervision,handcraftedfeatureswereused,forinstance“scal