HiC2010-Adaptive_and_Scalable_Metadata_Manage...

《HiC2010-Adaptive_and_Scalable_Metadata_Manage...》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、AdaptiveandScalableMetadataManagementtoSupportATrillionFileJingXingNationalResearchCenterForIntelligentComputingSystemsInstituteofComputingTechnologyChineseAcademyofSciencesOutline1.Introduction2.Ourapproach3.Evaluation4.Conclusion5.FutureWorksINTRODUCTIONIntroductionOurapproac

2、hEvaluationConclusionFutureworkTrendofDataProcessingAsthedevelopmentofglobalinformatization,moreandmoredataneedtobestored.(InternetService,BusinessTransaction,PersonalLife)Ontheotherhand,thepopularizationofInternetapplications,theincrementofscientificcomputingrequirement,andthe

3、enhancementofhardware,weneedmuchhigherdataprocessingperformance.FromIDC&EMCFromPDSIHIC2010BeijingXINGJINGNCIC@ICT@CASIntroductionOurapproachEvaluationConclusionFutureworkTrendofMetadataProcessingMetadataprocessingrequirementincreaseswithdataprocessingrequirement.Onthefieldsofsc

4、alabilityandperformance.•WithIncreaseddatastorage,metadataserversneedtoefficientlyorganizeandmaintainlargenumberoffiles--Facebook550kops/sec--Flickr38kops/sec--LSST9*10^18files•DifferentmetadataserversneedtobebalancedFromIDC&EMCdynamically--dynamicworkload--dynamicserverconfigu

5、rationHIC2010BeijingXINGJINGNCIC@ICT@CASIntroductionOurapproachEvaluationConclusionFutureworkWhatmatermostindistributedmetadatamanagement?1.Howtodistributenamespaceonmultiplemetadataservers•Thedefinitionofmetadataprocessingunit•Thelocatingmechanismofmetadataprocessingunit•None-

6、partition/Sub-treepartition/sub-dirpartition/filepartition2.Howtoimprovetheconcurrencyofmetadataprocessing3.Howtobetterutilizemetadatacache4.HowtobalanceworkloadamongmultiplemetadataserversHIC2010BeijingXINGJINGNCIC@ICT@CASIntroductionOurapproachEvaluationConclusionFutureworkNo

7、nepartition•Example(GoogleFS,HDFS)•Thereisnometadatapartitioninthisarchitecture.•Severalcopyofnamespacemetadataismaintainedinmultiplemetadataserverstoensurereliability.•Metadataiskeptinmemorytoimprovetheperformance.•WeakscalabilityatbothscaleandperformanceHIC2010BeijingXINGJING

8、NCIC@ICT@CASIntroductionOurapproachEvaluationConclusio