Discriminative RBM are universal approximators for Discrete Data

《Discriminative RBM are universal approximators for Discrete Data》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、PatternRecognition&BioinformaticsLaboratoryP.O.Box5013DelftUniversityofTechnology2600GADelfthttp://prb.tudelft.nl/TheNetherlandsCopyrightcLaurensvanderMaaten2011.February2,2011EWI-PRBTR2011{001DiscriminativeRestrictedBoltzmannMachinesareUniversalApproximatorsforDiscreteDataLaurensvanderMaatenPa

2、tternRecognition&BioinformaticsLaboratoryDelftUniversityofTechnologyAbstractThisreportproofsthatdiscriminativeRestrictedBoltzmannMachines(RBMs)areuniversalapproximatorsfordiscretedatabyadaptingexistinguniversalapproximationproofsforgenerativeRBMs.DiscriminativeRestrictedBoltzmannMachinesareUniv

3、ersalApproximatorsforDiscreteDataLaurensvanderMaatenPatternRecognition&BioinformaticsLaboratoryDelftUniversityofTechnology1IntroductionAdiscriminativeRestrictedBoltzmannMachine(RBM)modelsisaconditionalvariantoftheRBM[1,2,4]thatmodelstheconditionaldistributionp(yjx)as1Xp(yjx)=exp(xTWz+zTVy+bTz+c

4、Ty);(1)Z(x)zwhereZ(x)representsthepartitionfunctionXXZ(x)=exp(xTWz0+z0TVy0+bTz0+cTy0):(2)y0z0Inthisnote,weproofthefollowingtheoremfordiscriminativeRBMs:UniversalApproximationTheorem.Fordatax2X=f0;1gD,adiscriminativeRBMcanrepresentanyconditionaldistributionp(yjx)arbitrarilywellintermsofKullback-

5、Leiblerdivergence.Sincediscretedatacanbeexpressedexactlyintermsofabinaryrepresentation(e.g.,usinga1-of-Drepresentation),thetheoremappliestodiscretedata,too.Theproofoftheabovethe-oremisanadaptationofanearlierproofontherepresentationalpowerofgenerativeRBMs[3].2ProofDenotepotentialsbyF(x;y;z),thec

6、onditionaldistributionmodeledbyadiscriminativeRBMcanbewrittenasPPF(x;y;z)exp(xTWz+zTVy+bTz+cTy)p(yjx)=Pz=PPz;(3)0;z)0TWz0+z0TVy0+bTz0+cTy0)y0;z0F(x;yy0z0exp(xwhereweassumethatxisabinaryvector,andyisa1-of-Kvector.Wedenotethediscrimi-nativeRBMthathasoneadditionalhiddenunitwithparametersw,v,andbby

7、pwvb(yjx):P(1+exp(wTx+vTy+b))F(x;y;z)p(yjx)=PPz:(4)wvbTx+vTy0+b))0;z0)y0(1+exp(wz0F(x;yLet(x~;y~)beanarbitrary(x;y)-pairforwhichwewishtochangeincreasetheconditionalprobabilityp(y~jx~).Also,wedenetheparametersw^=a(x~1),v^=a(y~1)