资源描述:
《entropy and mutual information-[from book of cover]》由会员上传分享,免费在线阅读,更多相关内容在教育资源-天天文库。
1、CHAPTER2ENTROPY,RELATIVEENTROPY,ANDMUTUALINFORMATIONInthischapterweintroducemostofthebasicdefinitionsrequiredforsubsequentdevelopmentofthetheory.Itisirresistibletoplaywiththeirrelationshipsandinterpretations,takingfaithintheirlaterutility.Afterdefiningentropya
2、ndmutualinformation,weestablishchainrules,thenonnegativityofmutualinformation,thedata-processinginequality,andillustratethesedefinitionsbyexaminingsufficientstatisticsandFano’sinequality.Theconceptofinformationistoobroadtobecapturedcompletelybyasingledefinition
3、.However,foranyprobabilitydistribution,wedefineaquantitycalledtheentropy,whichhasmanypropertiesthatagreewiththeintuitivenotionofwhatameasureofinformationshouldbe.Thisnotionisextendedtodefinemutualinformation,whichisameasureoftheamountofinformationonerandomvari
4、ablecontainsaboutanother.Entropythenbecomestheself-informationofarandomvariable.Mutualinformationisaspecialcaseofamoregeneralquantitycalledrelativeentropy,whichisameasureofthedistancebetweentwoprobabilitydistributions.Allthesequantitiesarecloselyrelatedandsh
5、areanumberofsimpleproperties,someofwhichwederiveinthischapter.Inlaterchaptersweshowhowthesequantitiesariseasnaturalanswerstoanumberofquestionsincommunication,statistics,complexity,andgambling.Thatwillbetheultimatetestofthevalueofthesedefinitions.2.1ENTROPYWefi
6、rstintroducetheconceptofentropy,whichisameasureoftheuncertaintyofarandomvariable.LetXbeadiscreterandomvariablewithalphabetXandprobabilitymassfunctionp(x)=Pr{X=x},x∈X.ElementsofInformationTheory,SecondEdition,ByThomasM.CoverandJoyA.ThomasCopyright2006JohnWil
7、ey&Sons,Inc.1314ENTROPY,RELATIVEENTROPY,ANDMUTUALINFORMATIONWedenotetheprobabilitymassfunctionbyp(x)ratherthanpX(x),forconvenience.Thus,p(x)andp(y)refertotwodifferentrandomvariablesandareinfactdifferentprobabilitymassfunctions,pX(x)andpY(y),respectively.Defin
8、itionTheentropyH(X)ofadiscreterandomvariableXisdefinedbyH(X)=−p(x)logp(x).(2.1)x∈XWealsowriteH(p)fortheabovequantity.Thelogistothebase2andentropyisexpressedinbits.Forexample,theentropyofafaircoi