资源描述：
《Why Momentum Really Works》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、WhyMomentumReallyWorksStartingPointOptimumSolutionStepsizeα=0.02Momentumβ=0.99WeoftenthinkofMomentumasameansofdampeningoscillationsandspeedinguptheiterations,leadingtofasterconvergence.Butithasotherinterestingbehavior.Itallowsalargerrangeofstep‑sizestobeused,and0
2、0.0030.0060.000.5000.990createsitsownoscillations.Whatisgoingon?GABRIELGOHUCDavisApril.42017Citation:Goh,2017Here’sapopularstoryaboutmomentum[1,2,3]:gradientdescentisamanwalkingdownahill.Hefollowsthesteepestpathdownwards,hisprogressisslow,butsteady.Momentumisahea
3、vyballrollingdownthesamehill.Theaddedinertiaactsbothasasmootherandanaccelerator,dampeningoscillationsandcausingustobarrelthroughnarrowvalleys,smallhumpsandlocalminima.Thisstandardstoryisn’twrong,butitfailstoexplainmanyimportantbehaviorsofmomentum.Infact,momentumc
4、anbeunderstoodfarmorepreciselyifwestudyitontherightmodel.Onenicemodelistheconvexquadratic.Thismodelisrichenoughtoreproducemomentum’slocaldynamicsinrealproblems,andyetsimpleenoughtobeunderstoodinclosedform.Thisbalancegivesuspowerfultractionforunderstandingthisalgo
5、rithm.WebeginwithGradientDescent.Thealgorithmhasasmanyvirtues,butspeedisnotoneofthem.Itissimple--whenoptimizingasmoothfunctionf,wemakeasmallstepinthegradientwk+1=wk−α∇f(wk).Forastepsizesmallenough,gradientdescentmakesamonotonicimprovementateveryiteration.Italways
6、converges,albeittoalocalminimum.Andunderafewweakcurvatureconditionsitcanevengetthereatanexponentialrate.Buttheexponentialdecrease,thoughappealingintheory,canoftenbeinfuriatinglysmall.Thingsoftenbeginquitewell--withanimpressive,almostimmediatedecreaseintheloss.But
7、astheiterationsprogress,thingsstarttoslowdown.Youstarttogetanaggingfeelingyou'renotmakingasmuchprogressasyoushouldbe.Whathasgonewrong?Theproblemcouldbetheoptimizer'soldnemesis,pathologicalcurvature.Pathologicalcurvatureis,simplyput,regionsoffwhicharen'tscaledprop
8、erly.Thelandscapesareoftendescribedasavalleys,trenches,canalsandravines.Theiterateseitherjumpbetweenvalleys,orapproachtheoptimuminsmall,timidsteps.Progressalon
