Fitting larger networks into memory

《Fitting larger networks into memory》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、ApplausefromThomasWolfand14othersYaroslavBulatovFollowJan14·12minreadFittinglargernetworksintomemory.TLDR;wereleasethepython/Tensorowpackageopenai/gradient-checkpointing,thatletsyout10xlargerneuralnetsintomemoryatthecostofanadditional20%computationtime.GPUmemoryisoftenthelimitingfactorf

2、ormodernneuralnetworkarchitectures.Memoryrequirementtotrainaneuralnetworkincreaseslinearlywithbothnetworkdepthandbatch-size.Youwanttogodeeperforstandardreasons,butalsotoincreasethebatch-sizetomakeuseofsecondordermethodslikeKFAC.Suchmethodsneedfewerexamplestolearncomparedtomini-batchSGD.To

3、day,wereleaseapython/Tensorowpackage,openai/gradient-checkpointing,thatextendsthetechniquein“TrainingDeepNetswithSublinearMemoryCost”,TianqiChenetal,torewriteyourTensorFlowmodeltouselessmemory.Itgivesequivalentmemorysavingforsimplefeed-forwardnetworks,butitalsoletsyousavememoryforgeneral

4、neuralnetworks,suchasmulti-towerarchitecture.ThepackageisjointworkbyYaroslavBulatovandTimSalimans.ApplyingittoTensorFlowocialCIFAR10resnetexampleproducesthefollowingmemoryandexecutiontimesforbatchsize=1280.Whileregularbackpropscaleslinearly,thismethodscalesassquarerootofdepth.Thedierenc

5、eismoreapparentwhenwetryitoutfordeepernetworks.Extrapolatingmemoryrequirementofstandardapproachgives60GBofmemorytorunthisiteration,meanwhilememorysavinggradientsaccomplishesitin6GBofRAM.Computationoverheadis1extraforwardpassregardlessofdepth.Inexperimentsthistranslatedto20%increaseinwall-

6、clocktimeonGTX1080,and30%increaseinwall-clocktimeonV100GPU.Howmemorysavingworks:ThepebblegameTounderstandmemoryrequirementsofgeneralcomputation,computerscientistsusetheconceptofthe“pebblegame”,introducedin“CompleteRegisterAllocationProblems”bySethiin1975.ConsiderthefollowingcomputationYou

7、canvisualizethiscomputationasacomputationgraph:Inordertocomputeeachvalue,youneedtohaveitsdependenciesloadedintomemory.Thisisrepresentedbyplacing“pebbles”onthechildrenofthenode.Onceallchildrenofanodehavepebblesonthem,thenodeisreadyforexecution.Computingitsvalueisrepr