Druid 实时分析架构设计内容思路—Imply.pdf

《Druid 实时分析架构设计内容思路—Imply.pdf》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、DRUIDINTERACTIVEEXPLORATORYANALYTICSATSCALEFANGJINYANG·DRUIDCOMMITTEROVERVIEWDEMOSEESOMENEATTHINGSMOTIVATIONWHYDRUID?ARCHITECTUREPICTURESWITHARROWSCOMMUNITYCONTRIBUTETODRUIDTHEPROBLEM‣Arbitraryandinteractiveexplorationoftimeseriesdata•Ad-tech,system/appmetrics,ne

2、twork/websitetrafficanalysis‣Multi-tenancy:lotsofconcurrentusers‣Scalability:10+TB/day,ad-hocqueriesontrillionsofevents‣Recencymatters!Real-timeanalysis2013DEMOINCASETHEINTERNETDIDN’TWORKPRETENDYOUSAWSOMETHINGCOOLREQUIREMENTS‣Scalable&highlyavailable‣Real-timedata

3、ingestion‣Arbitrarydataexplorationwithad-hocqueries‣Sub-secondqueries‣Manyconcurrentreads2015FINDINGASOLUTION‣LoadallyourdataintoHadoop.Queryit.Done!‣Goodjobguys,let’sgohome2015FINDINGASOLUTIONHadoopInsightEventStreams2015PROBLEMSWITHTHENAIVESOLUTION‣MapReducecan

4、handlealmosteverydistributedcomputingproblem‣MapReduceoveryourrawdataisflexiblebutslow‣Hadoopisnotoptimizedforquerylatency‣Tooptimizequeries,weneedaquerylayer2015FINDINGASOLUTIONHadoopHadoop(pre-processingandstorage)QueryLayerInsightEventStreams2015MAKEQUERIESFAST

5、ER‣Whattypesofqueriestooptimizefor?•Businessintelligence/OLAP/pivottablesqueries•Aggregations,filters,groupBys2015WHATWETRIEDFINDINGASOLUTIONHadoopHadoop(pre-processingandstorage)RDBMS?InsightEventStreams2015I.RDBMS-THESETUP‣Commonsolutionindatawarehousing:•StarSc

6、hema•AggregateTables•QueryCaching2015I.RDBMS-THERESULTS‣Queriesthatwerecached•fast‣Queriesagainstaggregatetables•fasttoacceptable‣Queriesagainstbasefacttable•generallyunacceptable2015I.RDBMS-PERFORMANCENaivebenchmarkscanrate~5.5Mrows/second/core1dayofsummarizedag

7、gregates60M+rows1queryover1week,16cores~5secondsPageloadwith20queriesoveraweeklongtimeofdata2015FINDINGASOLUTIONHadoopNoSQLK/VHadoop(pre-processingandstorage)Stores?InsightEventStreams2015II.NOSQL-THESETUP‣Pre-aggregatealldimensionalcombinations‣StoreresultsinaNo

8、SQLstoreKeyValue1revenue=$1.19tsgenderagerevenue1,Mrevenue=$0.151M18$0.151,Frevenue=$1.041F25$1.031,18revenue=$0.161F18$0.011,25revenue=$1.031,M,18revenue=$0.1