Model Assessment and Selection

《Model Assessment and Selection》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、ModelAssessmentandSelectionKwok-LeungTsuiSystemsEngineering&EngineeringManagementCityUniversityofHongKong2/28/20121DataMining(KDD)ProcessDetermineBusinessObjectivesDataPreparationMining&ModelingConsolidationandApplication2/28/20122DataMining&ModelingStartConsiderChooseModelsAlternateModelsTrainD

2、ataBuild/FitModelCollectSamplemoreDataValidationRefine/TuneModeldataData(modelsize&diagnosis)EvaluateModelTestData(e.g.Predictionerror)(EvaluationData)NOMeetaccuracyreqt.YESScoreDataPredictionMakeDecisions2/28/20123Supervised&UnsupervisedLearning•Supervisedlearning:–Learningwithateacher–Classifi

3、cation,e.g.onlineshoppers(buyersVs.non-buyers)•Unsupervisedlearning:–Learningwithoutateacher–Clustering,e.g.onlineshoppers(segmentationofnon-buyers)•Otherrelatedterms:–MachineLearning(analogiestohumanreceiving)–NeuralNetworks(biologicalanalogiestobrain)2/28/20124SupervisedLearning•Inputs:(Predic

4、tors,independentvariables,y)–Asetofvariableswhicharemeasuredorpreset.•Outputs:(Responses,dependentvariables,x)–Asetofmeasurablevariableswhichareinfluencedbytheinputs•Steps:–Establishmodels/systems(yhat)basedoncollectedinputs&outputs(xandy).–Predictthevaluesofoutputsbasedontheestablishedmodels/sy

5、stemsandanewsetofspecifiedinputs.2/28/20125Training,Validation,andTestError•TwoBasicObjectives–ModelSelection:Choosebestmodelbasedoncertainperformancemeasures–Modelassessment:estimatingpredictionerroroffinalmodel•ErrorTypes–TrainingError:fittingmodel–ValidationError:selectingmodel–Test(Generaliz

6、ation)Error:assessingmodel•Methods–Analytical:Cp,AIC,BIC,etc.–Re-sampling:cross-validation,bootstrap.2/28/20126PredictionorClassificationErrorOverfittingTesterrorPredictionErrorTrainingerrorLowModelComplexityHigh2/28/20127TrainingError,Cross-ValidationError,TestErrorTestingdataTrainingdataCross-

7、Validation123...KFittedmodelusingtrainingdataTestingerrorbasedontestingdataTrainingerror&CVErrorbasedontrainingdata2/28/20128BiasVarianceDecomposition2/28/201292/28/2012Figure7.3-110BiasVarianceDecomposition2/28/201211BiasVa