数据挖掘之异常检测.ppt

《数据挖掘之异常检测.ppt》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、AnomalyDetection:AintroductionSourceofslides:TutorialAtAmericanStatisticalAssociation(ASA2008)JiaweiHan-datamining:conceptsandtechniquesTutorialattheEuropeanConferenceonPrinciplesandPracticeofKnowledgeDiscoveryinDatabasesSpeaker:WentaoLiOutlineDefinitionApplicationMethodsLim

2、itedtime,SoIjustdrawthepictureofanomalydetection,formoredetail,pleaseturntothepaperforhelp.WhatareAnomalies?AnomalyisapatterninthedatathatdoesnotconformtotheexpectedbehaviorAnomalyisAdataobjectthatdeviatessignificantlyfromthenormalobjectsasifitweregeneratedbyadifferentmechan

3、ismAlsoreferredtoasoutliers,exceptions,peculiarities,surprises,etc.Anomaliestranslatetosignificant(oftencritical)reallifeentitiesCyberintrusionsCreditcardfraudFaultsinmechanicalsystemsRelatedproblemsOutliersaredifferentfromthenoisedataNoiseisrandomerrororvarianceinameasuredv

4、ariableNoiseshouldberemovedbeforeoutlierdetectionOutliersareinteresting:ItviolatesthemechanismthatgeneratesthenormaldataOutlierdetectionvs.noveltydetection:earlystage,outlier;butlatermergedintothemodelKeyChallengesDefiningarepresentativenormalregionischallengingTheboundarybe

5、tweennormalandoutlyingbehaviorisoftennotpreciseAvailabilityoflabeleddatafortraining/validationTheexactnotionofanoutlierisdifferentfordifferentapplicationdomainsDatamightcontainnoiseNormalbehaviorkeepsevolvingAppropriateselectionofrelevantfeaturesMapRelatedareas(theory)Applic

6、ation(practice)ProblemformulationDetectioneffect+AspectsofAnomalyDetectionProblemNatureofinputdataWhatisthecharacteristicofinputdataAvailabilityofsupervisionNumberoflabelTypeofanomaly:point,contextual,structuralTypeofanomalyOutputofanomalydetectionScorevslabelEvaluationofano

7、malydetectiontechniquesWhatkindofdetectionisgoodInputDataMostcommonformofdatahandledbyanomalydetectiontechniquesisRecordDataUnivariateMultivariateInputDataMostcommonformofdatahandledbyanomalydetectiontechniquesisRecordDataUnivariateMultivariateInputData–NatureofAttributesNat

8、ureofattributesBinaryCategoricalContinuousHybridcategoricalcontinuouscontin