CMU高级机器学习变分推理.pdf

《CMU高级机器学习变分推理.pdf》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、AdvancedMachineLearningVariationalInferenceEricXingLecture12,August12,2009Reading:EricXing©EricXing@CMU,2006-20091AnIsingmodelon2-DimageòNodesencodehiddeninformation(patch-identity).òTheyreceivelocalinformationfromtheimage(brightness,color).òInformationispropagatedthoughthe

2、graphoveritsedges.òEdgesencode‘compatibility’betweennodes.airorwater??EricXing©EricXing@CMU,2006-200921WhyApproximateInference?òTree-widthofNxNgraphisO(N)òNcanbeahugenumber(~1000sofpixels)òExactinferencewillbetooexpensive1⎧⎫p(X)=exp⎨∑θijXiXj+∑θi0Xi⎬Z⎩i

3、CMU,2006-20093VariationalMethodsòForadistributionp(X

4、θ)associatedwithacomplexgraph,computingthemarginal(orconditional)probabilityofarbitraryrandomvariable(s)isintractableòVariationalmethodsòformulatingprobabilisticinferenceasanoptimizationproblem:e.g.f*=argmaxormin{F(f)}f∈S

5、a(tractable)probabilitydistributionf:or,solutionstocertainprobabilisticqueriesEricXing©EricXing@CMU,2006-200942BetheEnergyMinimizationEricXing©EricXing@CMU,2006-20095TheObjectiveòLetuscalltheactualdistributionPP(X)=1/Z∏fa(Xa)fa∈FòWewishtofindadistributionQsuchthatQisa“good”

6、approximationtoPòRecallthedefinitionofKL-divergenceQ(X)1KL(Q1

7、

8、Q2)=∑Q1(X)log()XQ2(X)òKL(Q

9、

10、Q)>=012òKL(Q

11、

12、Q)=0iffQ=Q1212òBut,KL(Q

13、

14、Q)≠KL(Q

15、

16、Q)1221EricXing©EricXing@CMU,2006-200963WhichKL?òComputingKL(P

17、

18、Q)requiresinference!òButKL(P

19、

20、Q)canbecomputedwithoutperforminginferenceo

21、nPQ(X)KL(Q

22、

23、P)=∑Q(X)log()XP(X)=∑Q(X)logQ(X)−∑Q(X)logP(X)XX=−H(X)−ElogP(X)QQòUsingP(X)=1/Z∏fa(Xa)fa∈FKL(Q

24、

25、P)=−HQ(X)−EQlog(1/Z∏fa(Xa))fa∈F=−HQ(X)−log1/Z−∑EQlogfa(Xa)fa∈FEricXing©EricXing@CMU,2006-20097TheObjectiveòKL(Q

26、

27、P)=−HQ(X)−∑EQlogfa(Xa)+logZfa∈FF(P,Q)òWewillcallthe“Ene

28、rgyFunctional”*F(P,Q)òF(P,P)=?òF(P,Q)>=F(P,P)*alsocalledGibbsFreeEnergyEricXing©EricXing@CMU,2006-200984TheEnergyFunctionalòLetuslookatthefunctionalF(P,Q)=−HQ(X)−∑EQlogfa(Xa)fa∈Fò∑EQlogfa(Xa)canbecomputedifwehavemarginalsovereachfafa∈FòHQ=−∑Q(X)logQ(X)isharder!Requiressumma

29、tionoverallXpossiblevaluesòComputingF,isthereforehardingeneral.∧òApproach1:Approxi