斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory

斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory

ID:36282483

大小:4.72 MB

页数:42页

时间:2019-05-08

斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory_第1页
斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory_第2页
斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory_第3页
斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory_第4页
斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory_第5页
资源描述:

《斯坦福深度学习课件7 Understanding_and_improving_deep_learing_with_random_matrix_theory》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库

1、UnderstandingandImprovingDeepLearningwithRandomMatrixTheoryJeffreyPenningtonGoogleBrain,NYCNovember8,2017Stats385,StanfordConfidential&ProprietaryOutline1.Motivation2.Essentialsofrandommatrixtheory3.Geometryofneuralnetworklosslandscapes4.Resurrectingthesigmoidindeeplearning5.Nonli

2、nearrandommatrixtheory6.ConclusionsConfidential&ProprietaryMotivation:WhyRandomMatrices?Confidential&ProprietaryWhyrandommatrices?●Theinitialweightconfigurationisrandom○Trainingmayinduceonlylow-rankperturbationsaroundtherandomconfiguration●Anexacttheoryofdeeplearningislikelytobein

3、tractableoruninformative○Largecomplexsystemsareoftenwell-modeledwithrandomvariables■E.g.statisticalphysicsandthermodynamics●Manyimportantquantitiesarespecifictomatrixstructure○E.g.eigenvaluesandeigenvectorsConfidential&ProprietaryWhichmatricesdowecareabout?●Activations●Hessians●Ja

4、cobiansConfidential&ProprietaryEssentialsofrandommatrixtheoryConfidential&ProprietarySpectraldensityForanymatrix,theempiricalspectraldensityis:Forasequenceofmatriceswithincreasingsize,,thelimitingspectraldensityis:StieltjestransformFortheStieltjestransformisdefinedas:Usingtheident

5、ities,ThespectraldensitycanberecoveredfromGusingtheinversionformula,R-transformandS-transformTheStieltjestransformcanbeusedtodefinetwousefulauxiliaryobjects:theR-transform,definedbythefunctionalequation,andtheS-transform,definedbyasimilarfunctionalequation,Freeadditionandfreemulti

6、plicationIfAandBarefreelyindependent,thenthespectrumofthesumA+BcomputedusingtheR-transform:AndthespectrumoftheproductABcanbecomputedusingtheS-transform:Confidential&ProprietaryFreeIndependenceClassicalindependenceFreeindependenceareindependentifonehasarefreelyindependentifwhenever

7、andaresuchthatwheneverandaresuchthatConfidential&ProprietaryGeometryofneuralnetworklosssurfaceswithYasamanBahriConfidential&ProprietarySinglecriticalpointConfidential&ProprietaryMultiplecriticalpointsA)AllminimaareroughlyB)GlobalminimummuchC)Allminimaandindex1equivalent,butindex1l

8、owerthanlocalminimacriticalpoints

当前文档最多预览五页,下载文档查看全文

此文档下载收益归作者所有

当前文档最多预览五页,下载文档查看全文
温馨提示:
1. 部分包含数学公式或PPT动画的文件,查看预览时可能会显示错乱或异常,文件下载后无此问题,请放心下载。
2. 本文档由用户上传,版权归属用户,天天文库负责整理代发布。如果您对本文档版权有争议请及时联系客服。
3. 下载前请仔细阅读文档内容,确认文档内容符合您的需求后进行下载,若出现内容与标题不符可向本站投诉处理。
4. 下载文档时可能由于网络波动等原因无法下载或下载错误,付费完成后未能成功下载的用户请联系客服处理。