model selection for small sample regression

《model selection for small sample regression》由会员上传分享，免费在线阅读，更多相关内容在工程资料-天天文库。

1、MachineLearning,48,9–23,2002c2002KluwerAcademicPublishers.ManufacturedinTheNetherlands.ModelSelectionforSmallSampleRegressionOLIVIERCHAPELLEolivier.chapelle@liple.frLIP6,15rueduCapitaineScott,75015Paris,FranceVLADIMIRVAPNIKvlad@research.att.comAT&TResearchL

2、abs,200LaurelAvenue,Middletown,NJ07748,USAYOSHUABENGIObengioy@IRO.UMontreal.CADept.IRO,CP6128,UniversitedeMontr´eal,Succ.Centre-Ville,2920Chemindelatour,Montr´eal,Qu´ebec,´Canada,H3C3J7Editor:DaleSchuurmansAbstract.Modelselectionisanimportantingredientofmany

3、machinelearningalgorithms,inparticularwhenthesamplesizeinsmall,inordertostriketherighttrade-offbetweenoverfittingandunderfitting.Previousclassicalresultsforlinearregressionarebasedonanasymptoticanalysis.Wepresentanewpenalizationmethodforperformingmodelselectio

4、nforregressionthatisappropriateevenforsmallsamples.Ourpenalizationisbasedonanaccurateestimatoroftheratiooftheexpectedtrainingerrorandtheexpectedgeneralizationerror,intermsoftheexpectedeigenvaluesoftheinputcovariancematrix.Keywords:modelselection,parametricre

5、gression,uniformconvergencebounds1.IntroductionConsidertheproblemofestimatingaregressionfunctioninthesetoffunctions∞f(x,α)=αkϕk(x)(1)k=1where{ϕ}formabasisofL(Rp),e.g.aFourierorwaveletbasis.k2Givenacollectionofdata(x1,y1),...,(xn,yn),whereyi=f(xi,α0)+ξiandxi

6、,ξiareindependentlygeneratedbyunknowndistributionsP(x)andP(ξ),onewantstofindthefunctionf(x,α∗)thatprovidesthesmallestvalueoftheexpectedlossR(α)=L(y,f(x,α))dP(x)dP(ξ)(2)whereL(y,f(x,α))isagivenlossfunction,usuallythequadraticlossL(y,f(x,α))=(y−f(x,α))2.Tomini

7、mizetheexpectedrisk(2),oneminimizestheempiricalrisk10O.CHAPELLE,V.VAPNIK,ANDY.BENGIOfunctional1nRemp(α)=L(yi,f(xi,α))ni=1Howeversincetheset(1)hasaninfiniteexpansion,thisideadoesnotwork:foranyfinitenumberof(different)examplestherearefunctionswhichhavezeroempir

8、icalriskandalargevalueoftheexpectedloss.Toguaranteeasmallexpectedrisk,onecanminimizetheempiricalfunctionaloveronlythefirstd=d(n)functionsϕk(x).Thisisreasonableiftheϕkareorderedinsuchwaythatputsth