第三章_数据预处理

《第三章_数据预处理》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、KnowledgeDiscoveryinDatabasesDataPreprocessingSchoolofSoftware,NanjingUniversityChapter3:DataPreprocessing•Whypreprocessthedata?•Datacleaning•Dataintegrationandtransformation•Datareduction•Discretizationandconcepthierarchygeneration•SummaryWhyDataPreprocessing?•Dataintherea

2、lworldisdirtyuincomplete:lackingattributevalues,lackingcertainattributesofinterest,orcontainingonlyaggregatedataunoisy:containingerrorsoroutliersuinconsistent:containingdiscrepanciesincodesornames•Noqualitydata,noqualityminingresults!uQualitydecisionsmustbebasedonqualitydat

3、auDatawarehouseneedsconsistentintegrationofqualitydataMajorTasksinDataPreprocessing•DatacleaninguFillinmissingvalues,smoothnoisydata,identifyorremoveoutliers,andresolveinconsistencies•DataintegrationuIntegrationofmultipledatabases,datacubes,orfiles•DatatransformationuNormal

4、izationandaggregation•DatareductionuObtainsreducedrepresentationinvolumebutproducesthesameorsimilaranalyticalresults•DatadiscretizationuPartofdatareductionbutwithparticularimportance,especiallyfornumericaldataFormsofdatapreprocessingChapter3:DataPreprocessing•Whypreprocesst

5、hedata?•Datacleaning•Dataintegrationandtransformation•Datareduction•Discretizationandconcepthierarchygeneration•SummaryDataCleaning•DatacleaningtasksuFillinmissingvaluesuIdentifyoutliersandsmoothoutnoisydatauCorrectinconsistentdataMissingData•DataisnotalwaysavailableuE.g.,m

6、anytupleshavenorecordedvalueforseveralattributes,suchascustomerincomeinsalesdata•Missingdatamaybeduetouequipmentmalfunctionuinconsistentwithotherrecordeddataandthusdeletedudatanotenteredduetomisunderstandingucertaindatamaynotbeconsideredimportantatthetimeofentryunotregister

7、historyorchangesofthedata•Missingdatamayneedtobeinferred.HowtoHandleMissingData?•Ignorethetuple:usuallydonewhenclasslabelismissing(assumingthetasksinclassification—noteffectivewhenthepercentageofmissingvaluesperattributevariesconsiderably.•Fillinthemissingvaluemanually:tedi

8、ous+infeasible?•Useaglobalconstanttofillinthemissingvalue:e.g.,“unknown”,anewclass