Deep_Learning_and_Unsupervised_Feature_Learning

《Deep_Learning_and_Unsupervised_Feature_Learning》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、NIPS2010WorkshoponDeepLearningandUnsupervisedFeatureLearningTutorialonDeepLearningandApplicationsHonglakLeeUniversityofMichiganCo-organizers:YoshuaBengio,GeoffHinton,YannLeCun,AndrewNg,andMarc’AurelioRanzato*Includesslidematerialsourcedfromtheco-organizers1Outline•

2、Deeplearning–Greedylayer-wisetraining(forsupervisedlearning)–Deepbeliefnets–Stackeddenoisingauto-encoders–Stackedpredictivesparsecoding–DeepBoltzmannmachines•Applications–Vision–Audio–Language2Outline•Deeplearning–Greedylayer-wisetraining(forsupervisedlearning)–Dee

3、pbeliefnets–Stackeddenoisingauto-encoders–Stackedpredictivesparsecoding–DeepBoltzmannmachines•Applications–Vision–Audio–Language3Motivation:whygodeep?•DeepArchitecturescanberepresentationallyefficient–Fewercomputationalunitsforsamefunction•DeepRepresentationsmighta

4、llowforahierarchyorrepresentation–Allowsnon-localgeneralization–Comprehensibility•Multiplelevelsoflatentvariablesallowcombinatorialsharingofstatisticalstrength•Deeparchitecturesworkwell(vision,audio,NLP,etc.)!4DifferentLevelsofAbstraction•HierarchicalLearning–Natur

5、alprogressionfromlowleveltohighlevelstructureasseeninnaturalcomplexity–Easiertomonitorwhatisbeinglearntandtoguidethemachinetobettersubspaces–Agoodlowerlevelrepresentationcanbeusedformanydistincttasks5GeneralizableLearning•SharedLowLevel•PartialFeatureSharingReprese

6、ntations–MixedModeLearning–Multi-TaskLearning–CompositionofFunctions–UnsupervisedTrainingtask1taskNtask1task2task3outputy1outputyNoutputoutputoutput…High-levelfeatures…shared…intermediaterepresentationLow-levelfeatures…rawinput…6ANeuralNetwork•ForwardPropagation:–S

7、uminputs,produceactivation,feed-forward7ANeuralNetwork•Training:BackPropagationofError–Calculatetotalerroratthetop–Calculatecontributionstoerrorateachstepgoingbackwardst2t18DeepNeuralNetworks•Simpletoconstruct–Sigmoidnonlinearityforhiddenlayers–Softmaxfortheoutputl

8、ayer•But,backpropagationdoesnotworkwell(ifrandomlyinitialized)–Deepnetworkstrainedwithbackpropagation(withoutunsupervisedpretraining)performworse