资源描述：
《HCE Performance Report英文文献资料》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、HadoopC++ExtensionPerformanceReportHadoopC++ExtensionPerformanceReportNameHadoopC++ExtensionSystemBaidu/sys/dpf/hadoopGroupYangDong,HanFusheng,XiaPeng,YangZongkuiDate2010-6-12百度在线网络技术（北京）有限公司-1-HadoopC++ExtensionPerformanceReportTableofContentsHadoopC++ExtensionPerfo
2、rmanceReport................................................................-1-1．Introduction.........................................................................................................-3-2．Environment.....................................................
3、...................................................-3-3．GenericJobConfiguration(partial)...............................................................-4-4．ExperimentCases..............................................................................................-5
4、-5．Conclusion.........................................................................................................-11-百度在线网络技术（北京）有限公司-2-HadoopC++ExtensionPerformanceReport1．IntroductionHCEisaninternalprojectdevelopedbyDPFgroupinBaiduthatistargettoimprovetheabili
5、tyforprocessingtremendousnumberofdata(PBscale),aswellastoprovidemoreeffectivememorymanagementandcontrolmechanism.ComparingwithJava,PIPES&Streamingwhicharepronetosufferfromout-of-memoryfailureinourdailyjobs,HCEcaneliminatethisbyimplementingdataprocessingstages(e.g.map
6、per,reducer,recorderreader/writer,partitioner,combiner,etc.)withc++code.TodeterminetheperformanceofHCE,wearrangeaseriesofexperimentswithdifferentenvironmentdeployments.Werecordseveralperformancemetricsincludingaveragejobdurationtime,correspondingCPUutility,memoryfoot
7、print,networkthroughput,etc.Forcomparison,wealsoevaluateJavaandPipesimplementationswiththesameconfigurations.ExperimentresultshowsthatHCEcanoutperformeitherJavaorPipesinallexperimentscenariosandconsumelessmemoryfootprint(onlyoccupiesabout80%relativetotheothertwoimple
8、mentations).Moresignificantly,HCEisshowntoreducetheexecutiontimebyapproximately21-43%whencomparedtotheJavaandPipes.Theremainderofth
