Hadoop内核在虚拟化平台上的优化与扩展.pdf

《Hadoop内核在虚拟化平台上的优化与扩展.pdf》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、HadoopVirtualizationExtensionsJunpingDuSr.MTS,VMware,Inc©2009VMwareInc.AllrightsreservedCloud:BigShiftsinSimplificationandOptimization1.ReducetheComplexity2.DramaticallyLower3.EnableFlexible,AgileCostsITServiceDeliverytosimplifyoperationstoredirectinvestmentintotomeetandanticipatetheandma

2、intenancevalue-addopportunitiesneedsofthebusiness2AUnifiedAnalyticsCloudSignificantlySimplifiesSimplify•SingleHardwareInfrastructure•Faster/EasierprovisioningSQLClusterBigSQLNoSQLHadoopNoSQLClusterUnifedAnalyticsInfrastructurePrivatePublicHadoopClusterOptimize•SharedResources=higherutil

3、ization•Elasticresources=fasteron-demandaccessDecisionSupportCluster3UnifyingtheBigDataPlatformusingVirtualizationGoals•Makeitfastandeasytoprovisionnewdataclustersondemand•AllowMixingofWorkloads•Leveragevirtualmachinestoprovideisolation(esp.forMulti-tenant)•Optimizedataperformancebasedon

4、virtualtopologies•MakethesystemreliablebasedonvirtualtopologiesLeveragingVirtualization•Elasticscale•Usehigh-availabilitytoprotectkeyservices,e.g.,Hadoop’snamenode/jobtracker•Resourcecontrolsandsharing:re-useunderutilizedmemory,cpu•PrioritizeWorkloads:limitorguaranteeresourceusageinamixe

5、denvironmentCloudInfrastructurePrivatePublic4VMwareiscommittedtobetheBestVirtualplatformforHadoopPerformanceStudiesandBestPractices•Studiesthrough2010-2011ofHadoop0.20onvSphere5•Whitepaper,includingdetailedconfigurationsandrecommendationsMakingHadooprunwellonvSphere•Performanceoptimizat

6、ionsinvSpherereleases•VMwareengagementinHadoopCommunityeffort•SupportingkeypartnerswiththeirdistributionsonvSphere•ContributingenhancementstoHadoop•AutomateHadoopdeploymentonvSphereHadoopFrameworkIntegrationSerengeti•SpringforHadoop:EnablingSpringtosimplifyMap-ReduceJobs•SpringBatch:Soph

7、isticatedbatchmanagement5EvolutionofHadooponVMsSlaveNodeVMVMVMVMCurrentComputeT1T2Hadoop:CombinedVMVMStorage/StorageStorageComputeHadoopinVMSeparateStorageSeparateComputeClusters-VMlifecycle-Separatecompute-SeparatevirtualclustersdeterminedfromdatapertenantbyDatanod