Druid 实时分析架构设计思路—Imply

《Druid 实时分析架构设计思路—Imply》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、DRUIDINTERACTIVEEXPLORATORYANALYTICSATSCALEFANGJINYANG·DRUIDCOMMITTEROVERVIEWDEMOSEESOMENEATTHINGSMOTIVATIONWHYDRUID?ARCHITECTUREPICTURESWITHARROWSCOMMUNITYCONTRIBUTETODRUIDTHEPROBLEM‣Arbitraryandinteractiveexplorationoftimeseriesdata•Ad-tech,system/appmetrics,network/websitetrafficanalysis‣Multi-ten

2、ancy:lotsofconcurrentusers‣Scalability:10+TB/day,ad-hocqueriesontrillionsofevents‣Recencymatters!Real-timeanalysis2013DEMOINCASETHEINTERNETDIDN’TWORKPRETENDYOUSAWSOMETHINGCOOLREQUIREMENTS‣Scalable&highlyavailable‣Real-timedataingestion‣Arbitrarydataexplorationwithad-hocqueries‣Sub-secondqueries‣Many

3、concurrentreads2015FINDINGASOLUTION‣LoadallyourdataintoHadoop.Queryit.Done!‣Goodjobguys,let’sgohome2015FINDINGASOLUTIONHadoopInsightEventStreams2015PROBLEMSWITHTHENAIVESOLUTION‣MapReducecanhandlealmosteverydistributedcomputingproblem‣MapReduceoveryourrawdataisflexiblebutslow‣Hadoopisnotoptimizedforqu

4、erylatency‣Tooptimizequeries,weneedaquerylayer2015FINDINGASOLUTIONHadoopHadoop(pre-processingandstorage)QueryLayerInsightEventStreams2015MAKEQUERIESFASTER‣Whattypesofqueriestooptimizefor?•Businessintelligence/OLAP/pivottablesqueries•Aggregations,filters,groupBys2015WHATWETRIEDFINDINGASOLUTIONHadoopHa

5、doop(pre-processingandstorage)RDBMS?InsightEventStreams2015I.RDBMS-THESETUP‣Commonsolutionindatawarehousing:•StarSchema•AggregateTables•QueryCaching2015I.RDBMS-THERESULTS‣Queriesthatwerecached•fast‣Queriesagainstaggregatetables•fasttoacceptable‣Queriesagainstbasefacttable•generallyunacceptable2015I.

6、RDBMS-PERFORMANCENaivebenchmarkscanrate~5.5Mrows/second/core1dayofsummarizedaggregates60M+rows1queryover1week,16cores~5secondsPageloadwith20queriesoveraweeklongtimeofdata2015FINDINGASOLUTIONHadoopNoSQLK/VHadoop(pre-processingandstorage)Stores?InsightEventStreams2015II.NOSQL-THESETUP‣Pre-aggregateall

7、dimensionalcombinations‣StoreresultsinaNoSQLstoreKeyValue1revenue=$1.19tsgenderagerevenue1,Mrevenue=$0.151M18$0.151,Frevenue=$1.041F25$1.031,18revenue=$0.161F18$0.011,25revenue=$1.031,M,18revenue=$0.1