欢迎来到天天文库
浏览记录
ID:39712491
大小:566.23 KB
页数:16页
时间:2019-07-09
《HiC2009-Using_Simulation_for_Large-scale_Distributed_System》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、MumakUsingSimulationforLarge-scaleDistributedSystemVerificationandDebuggingHongTang2009.11-HadoopWorld,BeijingHadoopPerformanceEngineering@Yahoo!ProductionCloudInvestigation-optimizationCycleAnalyticsBenchmarkingImplementationEvaluationToolsMetricsCollectionBenchmarkDevelopm
2、entDevelopmentOutlineMotivationsOverviewandStatusArchitectureLessonsandExperiencesConclusionsandFutureWorkMotivationsLarge-scaledistributedsystemishardtoverifyanddebug-Cannotafforda2000-nodeclusterforeverydeveloper,totesteveryfeatureenhancementorbugfix-Timeconsumingtor
3、unbenchmarks-HardtoreproducecornercaseconditionsMotivations(cont.)JobTrackerisafertileareaforexperimentation-Schedulingpolicies–wehavefourschedulersalready-SynergybetweenMapReduceandHDFS(e.g.blockplacementpolicies)-Improvingspeculativeexecutionpolicies-WEWANTMOREPEOPLETOHEL
4、PUSINNOVATE!But,JobTrackeristoocomplextomodifycorrectly-Manyfactorstoconsider:fairness,capacity/SLAguarantees,datalocality,loadbalance,failurehandlingandrecovery,etc-Manycontrolknobsincurrentimplementationwithsubtleinteractions–cantrickevenseasonedHadoopdevelopers-Generalad
5、vicetopeoplenewtoHadoopMapReducecodebase:DONOTPLAYWITHFIRE!Mumak–TheVisionDiscrete-eventsimulation-Cansimulateaclusterwiththousandsofnodesinoneprocess-DoesnotperformactualIOorcomputation-Virtualclock“spins”fasterthanwallclockPluggingintherealJobTrackerandSchedulercode-Good
6、:Noneedtoseparatelymodeltheschedulingpolicies-Bad/Good:InheritbothfeaturesandbugsinJTandSchedulerSimulateallconditionsofaproductioncluster-WorkloadandclusterconfigurationdrivenbytracegeneratedbyRumen-Jobsubmission,inter-arrival,jobdependencies,high-ramjobs-Allkindsoffailure
7、sandfailurerecoverylogic-ResourcecontentionProjectStatusWork-in-progressFirst-cutversioncommittedtoHadoop0.21andtrunk-Basicframework-Nomodelingofresourcecontention•Taskexecutiontimetakenas-isfromtraces-Onlyindividualtaskfailures,nonodefailuresorfailurecorrelations-Nojobdep
8、endencies-NospeculativeexecutionTheTeam-YahooHadoopPerformanceEngineeringT
此文档下载收益归作者所有
点击更多查看相关文章~~