基于成本模型的hadoop与storm混合系统的研究

《基于成本模型的hadoop与storm混合系统的研究》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、AbstractAbstractOurworldissteppingintoanerawheredatasizegrowsrapidly.Themainstreambigdatatechnologiescanbedividedintofourcategories:batchcomputing,real-timecomputing,streamcomputinganduniformresourcemanagementplatform.Eachcomputingframeworkhasitsownapplicablearea.

2、Onceuser’sdemandexceedstheareathataframeworkworksfor,thisframeworkdoesn’tperformwell.Forexample,MapReduceissuitableforbatchcomputingandhashighthroughout.WhileitslongresponsetimelimitsusagescenarioofMapReduce.Similarly,Stormcangiveuseraquickresponseforanevent,butit

3、sufferslowthroughout.Fortheseproblems,mysolutionistocombinesomeframeworksintoasystem.TheaimofthisresearchistodesignahybridsystemtointegrateHadoopandStormtoimprovetheoverallexecutionefficiency.ThespecificmethodistouseStormtoprocessreal-timedataandsmall-scaledatawhi

4、lelarge-scalehistoricaldataisfeedtoHadoopinordertoincreasingsystemthroughput.Mostofothersimilarsystemsareaimedatblockingtheunderlyingimplementationdetailsandprovidinguserswithaunifiedqueryinterface.Comparedtothesesystems,thesolutionofthispaperhastwoadvantages:1.de

5、signanautoframework-pickingalgorithmwhichcansmartlyassigntasktoproperframework.2.SettingcachetableforStorminHBasetoimproveitsperformance.Theimplementationofprototypesystemincludesthreelayers:querylanguage,task-schedulinglayerandtaskexecutionlayer.Theframe-pickinga

6、lgorithmbecomesapartofquerylanguage.Accordingtoperformancetests,hybridsystemcanmeetourexpectations.SinglestatementexecutiontimeisclosetoSummingbirdOnHadoopandis20%~40%fastertoSummingbirdOnStorm.Andforlarge-scaledata,hybridsystemwillchooseHadoopasexecutionframework

7、.Asaresultitsthroughouthasbeenhighlyimproved40%thanSummingbirdOnStorminthisscenario.Finallythisnewarchitecturehasthreeadvantagesbelow:1)SupportawiderangeofapplicationscenariosbecauseofcombiningHadoopandStorm.2)Provideuserswithunifiedquerylanguageandblockstheimplem

8、entationdetails.3)Chooseproperframeworksmartly.Keywords:Hadoop,Storm,HybridSystemII目录目录第一章概述............................................................