Greedy Layer-Wise Training of Deep Networks

《Greedy Layer-Wise Training of Deep Networks》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、GreedyLayer-WiseTrainingofDeepNetworksYoshuaBengio,PascalLamblin,DanPopoviciandHugoLarochelleDept.IRO,UniversitedeMontrealP.O.Box6128,DowntownBranch,Montreal,H3C3J7,QC,Canadafbengioy,lamblinp,popovicd,larochehg@iro.umontreal.caTechnicalReport1282Departementd'InformatiqueetRechercheOpe

2、rationnelleAugust21st,2006AbstractDeepmulti-layerneuralnetworkshavemanylevelsofnon-linearities,whichallowsthemtopotentiallyrepresentverycompactlyhighlynon-linearandhighly-varyingfunctions.However,untilrecentlyitwasnotclearhowtotrainsuchdeepnetworks,sincegradient-basedoptimizationstartingf

3、romrandominitializationappearstooftengetstuckinpoorsolutions.Hintonetal.recentlyintroducedagreedylayer-wiseunsupervisedlearningalgorithmforDeepBeliefNetworks(DBN),agenerativemodelwithmanylayersofhiddencausalvariables.Inthecontextoftheaboveoptimizationproblem,westudythisalgorithmempiricall

4、yandexplorevariantstobetterunderstanditssuccessandextendittocaseswheretheinputsarecontinuousorwherethestructureoftheinputdistributionisnotrevealingenoughaboutthevariabletobepredictedinasupervisedtask.1IntroductionRecenttheoreticalanalyses(Bengio,Delalleau,&LeRoux,2006)ofmodernnon-parametr

5、icmachinelearningalgorithmsuchaskernelmachinesandgraph-basedmanifoldandsemi-supervisedlearningalgo-rithmssuggestfundamentallimitationsofsomelearningalgorithms.Theproblemisclearinkernel-basedapproacheswhenthekernelislocal"(e.g.theGaussiankernel),i.e.K(x;y)convergestoaconstantwhenjjxyjjin

6、creases.Theseanalysespointtothedicultyoflearninghighly-varyingfunctions",i.e.functionsthathavealargenumberofvariations"inthedomainofinterest,e.g.,theywouldrequirealargenumberofpiecestobewellrepresentedbyapiecewise-linearapproximation.Sincethenumberofpiecescanbemadetogrowexponentiallywi

7、ththenumberofinputvariables,thisproblemisdirectlyconnectedwiththewell-knowncurseofdimensionalityforclassicalnon-parametriclearningalgorithms(forregression,classicationanddensityestimation).Iftheshapesofallthesepiecesareunrelated,oneneedsenoughexamplesforeachpiecein