Decoupled Neural Interfaces using Synthetic Gradients

《Decoupled Neural Interfaces using Synthetic Gradients》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、ˆA!BA!BBSBMA→BBSBMA→BDecoupledNeuralInterfacesusingSyntheticGradientshA→BA!BˆA!BˆA!BˆhA→BAA!BAMaxJaderberg1WojciechMarianCzarnecki1SimonOsindero1OriolVinyals1AlexGraves1DavidSilver1KorayKavukcuoglu1…AbstractLegend:…ATrainingdirectedneuralnetworkstypicallyre-Forwardconnection

2、,fi+2updatelockedquiresforward-propagatingdatathroughacom-fBBSBMBi+1putationgraph,followedbybackpropagatinger-rorsignal,toproduceweightupdates.Alllay-Forwardconnection,cnotupdatelockedfi+1NFi+1ers,ormoregenerally,modules,ofthenetworkarethereforelocked,inthesensethattheymustErrorgr

3、adienthAiFiwaitfortheremainderofthenetworktoexecute1fAfiforwardsandpropagateerrorbackwardsbeforeSyntheticerror…gradienttheycanbeupdated.Inthisworkwebreakthis…constraintbydecouplingmodulesbyintroduc-(a)(b)ingamodelofthefuturecomputationofthenet-Figure1.Generalcommunicationprotocolb

4、etweenAandB.Af-workgraph.Thesemodelspredictwhatthere-terreceivingthemessagehAfromA,BcanuseitsmodelofA,sultofthemodelledsubgraphwillproduceusingMB,tosendbacksyntheticgradients^Awhicharetrainedtoap-onlylocalinformation.InparticularwefocusonproximaterealerrorgradientsA.NotethatAdoes

5、notneedtomodellingerrorgradients:byusingthemodelledwaitforanyextracomputationafteritselftogetthecorrecter-rorgradients,hencedecouplingthebackwardcomputation.Thesyntheticgradientinplaceoftruebackpropa-feedbackmodelMBcanalsobeconditionedonanyprivilegedin-gatederrorgradientswedecouple

6、subgraphs,formationorcontext,c,availableduringtrainingsuchasalabel.andcanupdatethemindependentlyandasyn-chronouslyi.e.werealisedecoupledneuralin-terfaces.Weshowresultsforfeed-forwardmod-els,whereeverylayeristrainedasynchronously,creatingaforwardprocessinggraphwhichdefinestheflowrecur

7、rentneuralnetworks(RNNs)wherepredict-ofdatafromthenetworkinputs,througheachmodule,pro-ingone’sfuturegradientextendsthetimeoverducingnetworkoutputs.DefiningalossonoutputsallowswhichtheRNNcaneffectivelymodel,andalsoerrorstobegenerated,andpropagatedbackthroughtheahierarchicalRNNsystemw

8、ithtickingatdiffer-network