Attention-Based Models for Speech Recognition

《Attention-Based Models for Speech Recognition》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Attention-BasedModelsforSpeechRecognitionJanChorowskiDzmitryBahdanauUniversityofWrocław,PolandJacobsUniversityBremen,Germanyjan.chorowski@ii.uni.wroc.plDmitriySerdyukKyunghyunChoYoshuaBengioUniversitedeMontr´eal´UniversitedeMontr´eal´UniversitedeMontr´eal´CIFARSeniorFello

2、wAbstractRecurrentsequencegeneratorsconditionedoninputdatathroughanattentionmechanismhaverecentlyshownverygoodperformanceonarangeoftasksin-cludingmachinetranslation,handwritingsynthesis[1,2]andimagecaptiongen-eration[3].Weextendtheattention-mechanismwithfeaturesneededfors

3、peechrecognition.Weshowthatwhileanadaptationofthemodelusedformachinetranslationin[2]reachesacompetitive18.7%phonemeerrorrate(PER)ontheTIMITphonemerecognitiontask,itcanonlybeappliedtoutteranceswhichareroughlyaslongastheonesitwastrainedon.Weofferaqualitativeexplanationofthi

4、sfailureandproposeanovelandgenericmethodofaddinglocation-awarenesstotheattentionmechanismtoalleviatethisissue.Thenewmethodyieldsamodelthatisrobusttolonginputsandachieves18%PERinsingleutterancesand20%in10-timeslonger(repeated)utterances.Finally,weproposeachangetotheat-tent

5、ionmechanismthatpreventsitfromconcentratingtoomuchonsingleframes,whichfurtherreducesPERto17.6%level.1IntroductionRecently,attention-basedrecurrentnetworkshavebeensuccessfullyappliedtoawidevarietyoftasks,suchashandwritingsynthesis[1],machinetranslation[2],imagecaptiongener

6、ation[3]andvisualobjectclassification[4].1Suchmodelsiterativelyprocesstheirinputbyselectingrelevantcontentateverystep.Thisbasicideasignificantlyextendstheapplicabilityrangeofend-to-endarXiv:1506.07503v1[cs.CL]24Jun2015trainingmethods,forinstance,makingitpossibletoconstructn

7、etworkswithexternalmemory[6,7].Weintroduceextensionstoattention-basedrecurrentnetworksthatmakethemapplicabletospeechrecognition.Learningtorecognizespeechcanbeviewedaslearningtogenerateasequence(tran-scription)givenanothersequence(speech).Fromthisperspectiveitissimilartoma

8、chinetranslationandhandwritingsynthesistasks,forwhichattention-basedmethodshavebeenfoundsuitable