12[Nov 12]Time series&ampamp;amp;sequential hypothesis testing&ampamp;amp;anomaly detection[Alex Shyr].pdf

《12[Nov 12]Time series&ampamp;amp;sequential hypothesis testing&ampamp;amp;anomaly detection[Alex Shyr].pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、AnomalydetectionandsequentialstatisticssequentialstatisticsintimeseriesAlexShyrCS294PracticalMachineLearning11/12/2009(manyslidesfromXuanLongNguyenandCharlesSutton)TwotopicsTwotopicsAnomaldetectionAnomalydetectionSequentialstatisticsReview•DimensionalityReduction•e.g.PCA•HMM•ROCcurve

2、sROCcurvesOutline•ItdtiIntroduction•AnomalyDetection–StaticExample–TimeSeries•SequentialTests–StaticHypothesisTestingStaticHypothesisTesting–SequentialHypothesisTesting–Change-pointDetectionpointDetectionAnomaliesintimeseriesdata•Timeseriesisasequenceofdatapoints,,ymeasuredtyppically

3、yatsuccessivetimes,spacedat(oftenuniform)timeintervals•AnomaliesintimeseriesdataaredataAnomaliesintimeseriesdataaredatapointsthatsignificantlydeviatefromthenormalpatternnormalpatternofthedatasequenceofthedatasequenceExamplesoftimeseriesdataExamplesoftimeseriesdataNetworktrafficdataFi

4、nancedataFinancedataHumanActivitydataApplications•Failuredetection•Frauddetection(creditcardtelephone)Frauddetection(creditcard,telephone)•Spamdetection•Biosurveillance–detectinggeographichotspots•ComputerintrusiondetectionOutline•ItdtiIntroduction•AnomalyDetection–StaticExample–Time

5、Series•SequentialTests–StaticHypothesisTestingStaticHypothesisTesting–SequentialHypothesisTesting–Change-pointDetectionpointDetectionExample:Networktraffic[Lakhinaetal,2004]Goal:Findsource-destinationpairswithhightraffic(e.g.,byratevolume)rate,volume)Backbonenetwork….Y=10030422121729

6、13….Example:NetworktrafficDatamatrixPerformPCAonmatrixY….Y=1003042212172913….Low-dimensionaldataEigenvectors….Yv=yTvyTvt1t2vv…12….Example:NetworktrafficAbilenebackbonenetworktrafficvolumeover41linkscollectedover4weeksPerformPCAon41-dimdataSelecttop5componentsSelecttop5componentsanoma

7、liesthresholdProjectiontoresidualsubspaceConceptualframework•Learnamodelofnormalbehavior•FindoutliersundersomestatisticFindoutliersundersomestatisticalarmCriteriainanomalydetection•Falsealarmrate(typeIerror)•Misdetectionrate(typeIIerror)•Neyman-Pearsoncriteria–minimizemisdetectionrat

8、ewhilefalsea