斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15

斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15

ID:36282241

大小:6.73 MB

页数:71页

时间:2019-05-08

斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15_第1页
斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15_第2页
斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15_第3页
斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15_第4页
斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15_第5页
资源描述:

《斯坦福深度学习课件8 Topology and Geometry of Half-Rectified Network Optimizationstanford_nov15》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库

1、TopologyandGeometryofHalf-RectifiedNetworkOptimizationDanielFreeman1andJoanBruna21UCBerkeley2CourantInstituteandCenterforDataScience,NYUStats385StanfordNov15thMotivation•WeconsiderthestandardMLsetup:X1Pˆ=(xi,yi)Eˆ(⇥)=E`((X;⇥),Y)+R(⇥)ni(X,Y)⇠Pˆ`(z)convexE(⇥)=E(X,Y)⇠P`((X;⇥),Y).

2、R(⇥):regularizationMotivation•WeconsiderthestandardMLsetup:X1Pˆ=(xi,yi)Eˆ(⇥)=E`((X;⇥),Y)+R(⇥)ni(X,Y)⇠Pˆ`(z)convexE(⇥)=E(X,Y)⇠P`((X;⇥),Y).R(⇥):regularization•Populationlossdecomposition(aka“fundamentaltheoremofML”):⇤⇤⇤⇤E(⇥)=Eˆ(⇥)+E(⇥)Eˆ(⇥).

3、{z}

4、{z}trainingerrorgeneralizationg

5、ap•Longhistoryoftechniquestoprovablycontrolgeneralizationerrorviaappropriateregularization.•Generalizationerrorandoptimizationareentangled[Bottou&Bousquet]Motivation•However,whenisalarge,deepnetwork,currentbest(X;⇥)mechanismtocontrolgeneralizationgaphastwokeyingredients:–Stochas

6、ticOptimization❖“Duringtraining,itaddsthesamplingnoisethatcorrespondstoempirical-populationmismatch”[LéonBottou].–Makethemodelaslargeaspossible.❖seee.g.“UnderstandingDeepLearningRequiresRethinkingGeneralization”,[Ch.Zhangetal,ICLR’17].Motivation•However,whenisalarge,deepnetwork,c

7、urrentbest(X;⇥)mechanismtocontrolgeneralizationgaphastwokeyingredients:–StochasticOptimization❖“duringtraining,itaddsthesamplingnoisethatcorrespondstoempirical-populationmismatch”[LéonBottou].–Makethemodelaslargeaspossible.❖seee.g.“UnderstandingDeepLearningRequiresRethinkingGene

8、ralization”,[Ch.Zhangetal,ICLR’17].•Wefirstaddresshowoverparametrizationaffectstheenergylandscapes.E(⇥),Eˆ(⇥)•Goal1:Studysimpletopologicalpropertiesoftheselandscapesforhalf-rectifiedneuralnetworks.•Goal2:Estimatesimplegeometricpropertieswithefficient,scalablealgorithms.Diagnostictoo

9、l.OutlineoftheLecture•TopologyofDeepNetworkEnergyLandscapes•GeometryofDeepNetworkEnergyLandscapes•EnergyLandscapes,StatisticalInferenceandPhaseTransitions.PriorRelatedWork•ModelsfromStatisticalphysicshavebeenconsideredaspossibleapproximations[Dauphinetal.’14,Choromanskaetal.’15,S

10、egunetal.’15]•Tensorfactorizationmodelsc

当前文档最多预览五页,下载文档查看全文

此文档下载收益归作者所有

当前文档最多预览五页,下载文档查看全文
温馨提示:
1. 部分包含数学公式或PPT动画的文件,查看预览时可能会显示错乱或异常,文件下载后无此问题,请放心下载。
2. 本文档由用户上传,版权归属用户,天天文库负责整理代发布。如果您对本文档版权有争议请及时联系客服。
3. 下载前请仔细阅读文档内容,确认文档内容符合您的需求后进行下载,若出现内容与标题不符可向本站投诉处理。
4. 下载文档时可能由于网络波动等原因无法下载或下载错误,付费完成后未能成功下载的用户请联系客服处理。