资源描述:
《r for additive model》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、AdditiveModels36-350,DataMining,Fall20092November2009Readings:PrinciplesofDataMining,pp.393{395;Berk,ch.2.Contents1PartialResidualsandBackttingforLinearModels12AdditiveModels33TheCurseofDimensionality44Example:CaliforniaHousePricesRevisited71PartialRes
2、idualsandBackttingforLinearModelsThegeneralformofalinearregressionmodelishiXpEYjX~=~x=0+~~x=jxj(1)j=0whereforj21:p,thexjarethecomponentsof~x,andx0isalwaystheconstant1.(Addingactitiousconstantfeature"likethisisastandardwayofhandlingtheinterceptjustli
3、keanyotherregressioncoecient.)Supposewedon'tconditiononallofX~butjustonecomponentofit,sayXk.WhatistheconditionalexpectationofY?E[YjXk=xk]=E[E[YjX1;X2;:::Xk;:::Xp]jXk=xk](2)23Xp=E4jXjjXk=xk5(3)j=023X=kxk+E4jXjjXk=xk5(4)j6=k1wheretherstlineusesthelawoft
4、otalexpectation1,andthesecondlineusesEq.1.Turnedaround,23Xkxk=E[YjXk=xk] E4jXjjXk=xk5(5)j6=k2013X=E4Y @jXjAjXk=xk5(6)j6=kTheexpressionintheexpectationisthekthpartialresidual
5、the(total)residualisthedierencebetweenYanditsexpectation,thepartialresidualist
6、hedierencebetweenYandwhatweexpectittobeignoringthecontributionfromX.Let'sintroduceasymbolforthis,sayY(k).khix=EY(k)jX=x(7)kkkkInwords,iftheover-allmodelislinear,thenthepartialresidualsarelinear.AndnoticethatXkistheonlyinputfeatureappearinghere
7、ifwecoul
8、dsomehowgetholdofthepartialresiduals,thenwecanndkbydoingasimpleregression,ratherthanamultipleregression.Ofcoursetogetthepartialresidualweneedtoknowalltheotherjs...Thissuggeststhefollowingestimationschemeforlinearmodels,knownastheGauss-Seidelalgorithm,o
9、rmorecommonlyandtransparentlyasback-tting;thepseudo-codeisinExample1.Thisisaniterativeapproximationalgorithm.Initially,welookathowfareachpointisfromtheglobalmean,anddosimpleregressionsofthosedeviationsontheinputfeatures.Thisthengivesusabetterideaofwhat
10、theregressionsurfacereallyis,andweusethedeviationsfromthatsurfaceinournextsetofsimpleregressions.Ateachstep,eachcoecientisadjustedtotinwithwhatwealreadyknowabouttheothercoecients
11、that'swhyit'scalledbacktting".Itisnotobvious2