[NIPS 2013] Generalized Denoising Auto-Encoders as Generative Models

《[NIPS 2013] Generalized Denoising Auto-Encoders as Generative Models 》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、GeneralizedDenoisingAuto-EncodersasGenerativeModelsYoshuaBengio,LiYao,GuillaumeAlain,andPascalVincentDepartementd’informatiqueetrechercheop´erationnelle,Universit´edeMontr´eal´AbstractRecentworkhasshownhowdenoisingandcontractiveautoencodersimplicitlycapturethestructureofthedata-generati

2、ngdensity,inthecasewherethecor-ruptionnoiseisGaussian,thereconstructionerroristhesquarederror,andthedataiscontinuous-valued.Thishasledtovariousproposalsforsamplingfromthisimplicitlylearneddensityfunction,usingLangevinandMetropolis-HastingsMCMC.However,itremainedunclearhowtoconnectthetra

3、iningprocedureofregularizedauto-encoderstotheimplicitestimationoftheunderlyingdata-generatingdistributionwhenthedataarediscrete,orusingotherformsofcorrup-tionprocessandreconstructionerrors.Anotherissueisthemathematicaljustifi-cationwhichisonlyvalidinthelimitofsmallcorruptionnoise.Wepropo

4、sehereadifferentattackontheproblem,whichdealswithalltheseissues:arbitrary(butnoisyenough)corruption,arbitraryreconstructionloss(seenasalog-likelihood),handlingbothdiscreteandcontinuous-valuedvariables,andremovingthebiasduetonon-infinitesimalcorruptionnoise(ornon-infinitesimalcontractivepe

5、nalty).1IntroductionAuto-encoderslearnanencoderfunctionfrominputtorepresentationandadecoderfunctionbackfromrepresentationtoinputspace,suchthatthereconstruction(compositionofencoderandde-coder)isgoodfortrainingexamples.Regularizedauto-encodersalsoinvolvesomeformofregu-larizationthatpreve

6、ntstheauto-encoderfromsimplylearningtheidentityfunction,sothatrecon-structionerrorwillbelowattrainingexamples(andhopefullyattestexamples)buthighingeneral.Differentvariantsofauto-encodersandsparsecodinghavebeen,alongwithRBMs,amongthemostsuccessfulbuildingblocksinrecentresearchindeeplearn

7、ing(Bengioetal.,2013b).Whereastheusefulnessofauto-encodervariantsasfeaturelearnersforsupervisedlearningcandirectlybeassessedbyperformingsupervisedlearningexperimentswithunsupervisedpre-training,whathasremaineduntilrecentlyratherunclearistheinterpretationofthesealgorithmsintheco