[PhD Thesis 2013] Advances in Deep Learning

《[PhD Thesis 2013] Advances in Deep Learning》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、(DRAFT)AdvancesinDeepLearningKyunghyunChoJuly17,2013AbstractDeepneuralnetworkshavebecomeincreasinglymorepopularunderthenameofdeeplearningrecentlyduetotheirsuccessinchallengingmachinelearningtasks.Althoughthepopularityismainlyduetotherecentsuccesses,thehi

2、storyofneuralnetworksgoesasfarbackas1958whenRosenblattpresentedaperceptronlearningalgorithm.Sincethen,variouskindsofartificialneuralnetworkshavebeenpro-posed.TheyincludeHopfieldnetwork,self-organizingmaps,neuralprincipalcom-ponentanalysis,Boltzmannmachines

3、,multi-layerperceptrons,radial-basisfunctionnetworks,autoencoders,sigmoidbeliefnetwork,supportvectormachinesanddeepbeliefnetworks.Inthefirstpartofthisthesis,theauthoraimsatinvestigatingthesemodelsandfindingacommonsetofbasicprinciplesfordeepneuralnetworks.T

4、hethesisstartsfromsomeoftheearlierideasandmodelsinthefieldofartificialneuralnetworksandarriveatautoencodersandBoltzmannmachineswhicharetwomostwidelystudiedneuralnetworksthesedays.Theauthorthoroughlydiscusseshowthosevariousneu-ralnetworksarerelatedtoeachoth

5、erandhowtheprinciplesbehindthosenetworksformfoundationforautoencodersandBoltzmannmachines.Thesecondpartisthecollectionofthetenrecentpublicationsbytheauthor.ThesepublicationsmainlyfocusonlearningandinferencealgorithmsofBoltzmannma-chinesandautoencoders.Es

6、pecially,Boltzmannmachineswhichareknowntobedifficulttotrainhavebeeninthemainfocus.Throughoutseveralpublicationstheau-thorandtheco-authorshavedevisedandproposedanewsetoflearningalgorithmswhichincludestheenhancedgradient,adaptivelearningrateandparalleltempe

7、ring.ThesealgorithmsarefurtherappliedtoarestrictedBoltzmannmachinewithGaus-sianvisibleunits.InadditiontothesealgorithmsforrestrictedBoltzmannmachinestheauthorpro-posedatwo-stagepretrainingalgorithmthatinitializestheparametersofadeepBoltzmannmachinetomatc

8、hthevariationalposteriordistributionofasimilarlystructureddeepautoencoder.Finally,deepneuralnetworksareappliedtoimagedenoisingandspeechrecognition.34PrefaceThisworkhasbeencarriedoutintheDepartmentofInformationandComputerSc