maximum entropy and minimal mutual information in a nonlinear model

《maximum entropy and minimal mutual information in a nonlinear model》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、MAXIMUMENTROPYANDMINIMALMUTUALINFORMATIONINANONLINEARMODELFabianJ.Theis,ElmarW.LangInstituteofBiophysicsUniversityofRegensburg,D-93040Regensburg,Germanyemail:fabian.theis@mathematik.uni-regensburg.deABSTRACTClassically,linearBSShasbeentreatedmostthoroughlyInblindsour

2、ceseparation,twodifferentseparationtech-[7][8],wherethemixingfunctionofthesourcesignalscor-niquesaremainlyused:MinimalMutualInformation(MMI),respondstoalinearfunction(matrix).Twoprinciplesbe-whereminimizationofthemutualoutputinformationyieldscamepopular:MinimalMutual

3、Information(MMI),whereanindependentrandomvector,andMaximumEntropy(ME),minimizationofthemutualoutputinformationyieldsanwheretheoutputentropyismaximized.However,itisyetindependentrandomvector,andMaximumEntropy(ME),unclearwhyMEshouldsolvetheseparationproblem,ie.wherethe

4、outputentropyismaximized.ConcerningMMIresultinanindependentvector.andME,ComonproposedtousethemutualinformationAmarihasgivenapartialconfirmationforMEintheoftheoutputasacontrastfunctionbecauseminimizingthelinearcasein[1],whereheprovesthatundertheassumptionmutualinformat

5、ion(MMI)inducesstatisticalindependenceofvanishingexpectancyofthesourcesMEdoesnotchangeoftheoutput.ThishastobecomparedwithBellandSe-thesolutionsofMMIuptoscalingandpermutation.jnowski’ssuggestiontomaximizetheentropy(ME)oftheInthispaper,wegeneralizeAmari’sapproachtononl

6、in-output.ButMEdoesnotalwaysinduceMMI([4],sec-earICAproblems,whererandomvectorshavebeenmixedtion4and[9])andthereforestatisticalindependence.MEbyoutputfunctionsoflayeredneuralnetworks.Weshowperformsbestwhenthenon-lineardemixingfunctioninthethatcertainsolutionpointsofM

7、MIarekeptfixedbyMEifMEalgorithmmatcheswiththecumulativedistributionofnoscalingoftheweightvectorsisallowed.Ingeneral,MEthegivensource.howevermightleavethoseMMIsolutionsusingdiagonalHowever,nowadaysmanyalgorithmsarebasedontheweightsinthefirstnetworklayer.Therefore,weconc

8、ludeMEcontrastfunction.ThisraisesthequestionhowthisthispaperbysuggestingthatinnonlinearMEalgorithmslargebranchofICAresearchcompares