超越hadoop的大数据技术：用spark 和shark进行基于内存的实时大数据分析

《超越hadoop的大数据技术：用spark 和shark进行基于内存的实时大数据分析》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、CCFYOCSEFShanghaiBigDataBeyondHadoopReal-TimeAnalyticalProcessing(RTAP)UsingSparkandSharkJasonDaiEngineeringDirector&PrincipalEngineerIntelSoftwareandServicesGroupAgendaBigDatabeyondHadoopIntroductiontoSparkandSharkCasestudy:real-timeanalyticalprocessing(RTAP)BigDatabeyondHadoop

2、BigDtatoday•TheisintheroomBigDatabeyondHadoop•Real-timeanalyticalprocessing(RTAP)–Discoverandexploredataiterativelyandinteractivelyforreal-timeinsights•Advancedmachineleaninganddatamining(MLDM)–Graph-parallelpredictiveanalytics(non-SQL)•Distributedin-memoryanalytics–Exploitavail

3、ablemainmemoryintheentireclusterfor>100xspeedupRTAP:Real-TimeAnalyticalProcessingReal-TimeAnalyticalProcessing(RTAP)•Dataingested&processedinastreamingfashion•Real-timedataqueriedandpresentedinanonlinefashion•Real-timeandhistorydatacombinedandminedinteractively•PredominantlyRAM-

4、basedprocessingAdvanced,Graph-ParallelMLDMAdvancedmachinelearninganddatamining(MLDM)•Informationretrieval(e.g.,pagerank)•Recommendationengine(e.g.,ALS)•Socialnetworkanalysis(e.g.,clustering)•Naturallanguageprocessing(e.g.,NER)•…Graphparallelcomputations•AsparsegraphG(V,E)•Averte

5、xprogramPrunsoneachvertexinparallel&repeatedly•VerticesinteractalongedgesAdvanced,Graph-ParallelMLDMMLDMData-ParallelGraph-ParallelMapReducePregel/GraphLab•Independentdata•(Sparse)datadependence•Single-pass•Iterative•(Bulk)synchronous•Dynamicallyprioritized10x~100xspeedup•Exploi

6、tgraphstructuretoreducecomputation&communications•Efficientgraphpartitiontobalancecomputation/storage,andminimizenetworktransferDistributedIn-MemoryAnalyticsMemoryisking•64GB/nodemainstream,192GBnotuncommon,fastcheapNVRAMonthehorizonHadoopinherentlydisk-basedarchitecture•Fulltab

7、lescaninHivefromRAMonly~40%speedup•Readallthemain-memoryDBliteraturesDistributedin-memoryanalytics•Efficientcomputeintegratedwithcolumnarcompression•ReliableRAM-orientedstoragelayeracrossthecluster•Holisticallocationofmemoryinthecluster–Inputs,intermediateresults,temporarydata,

8、computationstate,etc.AgendaBigDatabeyondHadoopI