资源描述：
《第2课数据预处理技术》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。
1、第2课数据预处理技术徐从富，副教授浙江大学人工智能研究所浙江大学本科生《数据挖掘导论》课件内容提纲Whypreprocessthedata?DatacleaningDataintegrationandtransformationDatareductionDiscretizationandconcepthierarchygenerationSummaryWhyDataPreprocessing?Dataintherealworldisdirtyincomplete:lackingattributevalues,lackingcertainat
2、tributesofinterest,orcontainingonlyaggregatedatae.g.,occupation=“”noisy:containingerrorsoroutlierse.g.,Salary=“-10”inconsistent:containingdiscrepanciesincodesornamese.g.,Age=“42”Birthday=“03/07/1997”e.g.,Wasrating“1,2,3”,nowrating“A,B,C”e.g.,discrepancybetweenduplicatereco
3、rdsWhyIsDataDirty?Incompletedatacomesfromn/adatavaluewhencollecteddifferentconsiderationbetweenthetimewhenthedatawascollectedandwhenitisanalyzed.human/hardware/softwareproblemsNoisydatacomesfromtheprocessofdatacollectionentrytransmissionInconsistentdatacomesfromDifferentda
4、tasourcesFunctionaldependencyviolationWhyIsDataPreprocessingImportant?Noqualitydata,noqualityminingresults!Qualitydecisionsmustbebasedonqualitydatae.g.,duplicateormissingdatamaycauseincorrectorevenmisleadingstatistics.DatawarehouseneedsconsistentintegrationofqualitydataDat
5、aextraction,cleaning,andtransformationcomprisesthemajorityoftheworkofbuildingadatawarehouse.—BillInmonMulti-DimensionalMeasureofDataQualityAwell-acceptedmultidimensionalview:AccuracyCompletenessConsistencyTimelinessBelievabilityValueaddedInterpretabilityAccessibilityBroadc
6、ategories:intrinsic,contextual,representational,andaccessibility.MajorTasksinDataPreprocessingDatacleaningFillinmissingvalues,smoothnoisydata,identifyorremoveoutliers,andresolveinconsistenciesDataintegrationIntegrationofmultipledatabases,datacubes,orfilesDatatransformation
7、NormalizationandaggregationDatareductionObtainsreducedrepresentationinvolumebutproducesthesameorsimilaranalyticalresultsDatadiscretizationPartofdatareductionbutwithparticularimportance,especiallyfornumericaldataFormsofdatapreprocessingDataCleaningImportance“Datacleaningiso
8、neofthethreebiggestproblemsindatawarehousing”—RalphKimball“Datacleaningisthenumberoneprob
