Attention & Memory

《Attention & Memory》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Attention&Memory集智俱乐部钟翰廷目录●RNNanddynamicsinRNN○FromvanillaRNNtoLSTM/GRU○UnitaryEvolutionRNN○Delay&HierarchiestoReachFather○DynamicsinRNN●AttentionMechanism○Introduction○AttentioninImageCaption/Generation○AttentioninNLP○BetterAttentionMechanism●MemoryNetwork○NeuralTuringMachineandMemory

2、Network○DynamicMemoryNetwork○HierarchicalMemoryNetworkPartI:RNNanddynamicsinRNN●FromvanillaRNNtoLSTM/GRU●UnitaryEvolutionRNN●Delay&HierarchiestoReachFarther●DynamicsinRNNFromvanillaRNNtoLSTM/GRUBasicstructureFromvanillaRNNtoLSTM/GRUBPTT:BackPropagationThroughTimeVanisinggradientproblem

3、FromvanillaRNNtoLSTM/GRUVanisinggradientproblemFromvanillaRNNtoLSTM/GRUPascanu,Razvan,TomasMikolov,andYoshuaBengio."Onthedifficultyoftrainingrecurrentneuralnetworks."ICML(3)28(2013):1310-1318.●Usearegularizationtermthatrepresentsapreferenceforparametervaluessuchthatback-propagatedgradi

4、entsneitherincreaseordecreaseinmagnitude●FromthedynamicalsystemsperspectivewecanseethatpreventingthevanishinggradientproblemimpliesthatwearepushingthemodeltowardstheboundaryofthecurrentbasinofattractionFromvanillaRNNtoLSTM/GRULSTMGRUUnitaryEvolutionRecurrentNeuralNetworks●Whentheeigenv

5、aluesofthehiddentohiddenweightmatrixdeviatefromabsolutevalue1,optimizationbecomesdifficult●uRNNlearnsaunitaryweightmatrix,witheigenvaluesofabsolutevalueexactly1.●uRNNusesReLUasactivationfunction.uRNNCopyingmemoryproblemuRNNAddingProblemDelay&HierarchiestoReachFartherThetemporaldependen

6、ciesarestructuredhierarchicallyDelay&HierarchiestoReachFarther●Delaysandmultipletimescales:Elhihi&BengioNIPS1995,KoutniketalICML2014●HierarchicalRNNs:SordonietalCIKM2015,Mehri&SoroushICLR2017DynamicsinRNNLSTM:ASearchSpaceOdyssey●Theforgetgateandtheoutputactivationfunctionarethecritical

7、componentsoftheLSTMblock●CertainmodificationssuchascouplingtheinputandforgetgatesorremovingpeepholeconnectionssimplifyLSTMwithoutsignificantlyhurtingperformance●LearningrateandnetworksizearethemostcrucialtunableLSTMhyperparameters.Surprisingly,theuseofmomentumwasfoundtobeunimportantD