HiC2011-Realtime Data Streams and Analytics-Data Freeway and Puma-Facebook

《HiC2011-Realtime Data Streams and Analytics-Data Freeway and Puma-Facebook》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Real-timeAnalyticsatFacebook:DataFreewayandPumaZhengShao12/2/2011Agenda1AnalyticsandReal-time2DataFreeway3Puma4FutureWorksAnalyticsandReal-timewhatandwhyFacebookInsights•Usecases▪Websites/Ads/Apps/Pages▪Timeseries▪Demographicbreak-downs▪Uniquecounts/heavyhitters•Majorchallenges▪Scalabil

2、ity▪LatencyAnalyticsbasedonHadoop/HiveHourlyDailysecondssecondsCopier/LoaderPipelineJobsHTTPScribeNFSHiveMySQLHadoop•3000-nodeHadoopcluster•Copier/Loader:Map-Reducehidesmachinefailures•PipelineJobs:HiveallowsSQL-likesyntax•Goodscalability,butpoorlatency!24–48hours.HowtoGetLowerLatency?•

3、Small-batchProcessing•StreamProcessing▪RunMap-reduce/Hiveeveryhour,every▪Aggregatethedataassoonasitarrives15min,every5min,…▪Howtosolvethereliabilityproblem?▪Howdowereduceper-batchoverhead?Decisions•StreamProcessingwins!•DataFreeway▪ScalableDataStreamFramework•Puma▪ReliableStreamAggregat

4、ionEngineDataFreewayscalabledatastreamScribeBatchCopierHDFStail/fopenScribeScribeScribeMid-TierWritersNFSClientsLog•Simplepush/RPC-basedloggingsystemConsumer•Open-sourcedin2008.100logcategoriesatthattime.•Routingdrivenbystaticconfiguration.DataFreewayContinuousCopierC1C2DataNodeHDFSPTai

5、lC1C2DataNode(intheplan)ScribePTailClientsCalligraphusCalligraphusHDFSMid-tierWritersLogConsumerZookeeper•9GB/secatpeak,10seclatency,2500logcategoriesCalligraphus•RPCFileSystem▪Eachlogcategoryisrepresentedby1ormoreFSdirectories▪Eachdirectoryisanorderedlistoffiles•Bucketingsupport▪Appli

6、cationbucketsareapplication-definedshards.▪InfrastructurebucketsallowslogstreamsfromxB/stoxGB/s•Performance▪Latency:Callsyncevery7seconds▪Throughput:Easilysaturate1GbitNICContinuousCopier•FileSystemFileSystem•Lowlatencyandsmoothnetworkusage•Deployment▪Implementedaslong-runningmap-onlyj

7、ob▪Canmovetoanysimplejobscheduler•Coordination▪UselockfilesonHDFSfornow▪PlantomovetoZookeeperPTailfilescheckpointdirectorydirectorydirectory•FileSystemStream(RPC)•Reliability▪Checkpointsinsertedintothedatastream▪Canrollbacktotailfromanydatacheckpoints▪Nodataloss/duplicatesCha