资源描述:
《HCE Performance Report英文文献资料》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、HadoopC++ExtensionPerformanceReportHadoopC++ExtensionPerformanceReportNameHadoopC++ExtensionSystemBaidu/sys/dpf/hadoopGroupYangDong,HanFusheng,XiaPeng,YangZongkuiDate2010-6-12百度在线网络技术(北京)有限公司-1-HadoopC++ExtensionPerformanceReportTableofContentsHadoopC++ExtensionPerfo
2、rmanceReport................................................................-1-1.Introduction.........................................................................................................-3-2.Environment.....................................................
3、...................................................-3-3.GenericJobConfiguration(partial)...............................................................-4-4.ExperimentCases..............................................................................................-5
4、-5.Conclusion.........................................................................................................-11-百度在线网络技术(北京)有限公司-2-HadoopC++ExtensionPerformanceReport1.IntroductionHCEisaninternalprojectdevelopedbyDPFgroupinBaiduthatistargettoimprovetheabili
5、tyforprocessingtremendousnumberofdata(PBscale),aswellastoprovidemoreeffectivememorymanagementandcontrolmechanism.ComparingwithJava,PIPES&Streamingwhicharepronetosufferfromout-of-memoryfailureinourdailyjobs,HCEcaneliminatethisbyimplementingdataprocessingstages(e.g.map
6、per,reducer,recorderreader/writer,partitioner,combiner,etc.)withc++code.TodeterminetheperformanceofHCE,wearrangeaseriesofexperimentswithdifferentenvironmentdeployments.Werecordseveralperformancemetricsincludingaveragejobdurationtime,correspondingCPUutility,memoryfoot
7、print,networkthroughput,etc.Forcomparison,wealsoevaluateJavaandPipesimplementationswiththesameconfigurations.ExperimentresultshowsthatHCEcanoutperformeitherJavaorPipesinallexperimentscenariosandconsumelessmemoryfootprint(onlyoccupiesabout80%relativetotheothertwoimple
8、mentations).Moresignificantly,HCEisshowntoreducetheexecutiontimebyapproximately21-43%whencomparedtotheJavaandPipes.Theremainderofth