基于预处理技术的数据清理系统研究与实现

《基于预处理技术的数据清理系统研究与实现》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、南京航空航天大学硕士学位论文基于预处理技术的数据清理系统研究与实现姓名：李俊娴申请学位级别：硕士专业：计算机应用技术指导教师：郑洪源20070101基于预处理技术的数据清理系统的研究与实现ABSTRACTWiththerapiddevelopmentofinformationtechnologies,enterpriseshaveaccumulatedalotofdataafterlong-termoperation.Howtousethesedataandthentodirectthedecisionanalysisofen

2、terprisesisthekeythattheywinandmakemaximalbenefits.Decisionsupportsystemisakindofapplicationsbasedondata,butthedatawithmanyqualititiveproblemsresultinfalsedecisveanalysis.Socorrectingtheerrorsofdirtydataplayscentralroleofreduingtheriskofwrongdecision.Therefore,thest

3、udyondatacleaninghasgaineditstheoreticalandpracticalvalue.Inthisdissertation,datacleaningisstudiedandadatacleaningsystemisdesigned.Thetechnologiesofdatacleaningpre-processinganddatacleaningareresearchedandanalysedinthisthesis.Thestudyofdatapre-prcessingfocusesontheo

4、utlierdetectionandabbreviation-discovered.Andtheapproximatelyduplicatedrecordscleaningthekeytechnologyofdatacleaningisstudied.Outlierdetectionalgorithmusesadistance-basedmethodwithlocalpruning,andaverticaldatastructure(PeanoCountTree,P-tree)isadoptedtofacilitateeffi

5、cientoutlierdetectionfurther.Andwemaketwomodificationsonthealgorithm.Anabbreviation-discoveredalgorithmbasedondynamicprogrammingisstudied.ItcandetectabbreviationsinChineseaswellasinEnglish.Inordertocleanapproximatelyduplicatedrecords,asyntheticallycleaningmethodisgi

6、ven:AMulti-PassSorted-Neighborhoodalgorithmisbeingusedtosortthedataset.Thenthisthesisanalysesthemethodsthatcalculatethesimilaritybetweenfieldsaswellassimilaritybetweenrecords.Andthecombinationofapproximatelyduplicatedrecordsisdiscussedinthethesis.Andwemakemodificati

7、onsoneachalgorithm.Basedontheresearchondatapre-processinganddatacleaning,adatacleaningsystemisproposed.Attheendofthisdissertation,theresultsofexperimentsshowthatthedatacleaningsystemandalgorithmshavegoodcleaningeffectandhighefficiency.Keywords:datacleaning,datapre-p

8、rocessing,outlierdetection,abbreviationdiscovery,approximatelyduplicatedidentificationii南京航空航天大学硕士学位论文图清单图1.1数据一致性与数据冗余成反比关系..............