资源描述:
《hdfs高可用性方案的分析与设计》由会员上传分享,免费在线阅读,更多相关内容在行业资料-天天文库。
1、杭州电子科技大学硕士学位论文ABSTRACTWiththerapidgrowthofthedata-intensiveapplications,allkindsofdataarebecomingmoreandmorelargesincethebeginning15yearsofthiscentury.Traditionalstoragesystemsarebecomingmoreandmorepowerlessinthefieldofhugeamountofdataprocessingandmanagement,anddis
2、tributedcomputingframeworkisbecomingsuitableinhugedataprocessingandmanagement.Hadoopisanimplementationoflarge-scaledistributedcomputingframework,withhighthroughput,highreliability,highscalability,etc.Thereforeitiswidelyusedinthefieldofbigdataprocessingandstorage.Ha
3、doopDistributedFileSystem(HDFS),asafundamentalcomponentofHadoop,isresponsibleforprovidinghighperformancedatastorageandmanagementserviceforHadoop.HDFSisdesignedofseparatingrealdatafrommetadata,whichadoptsmaster-slavesarchitecture,whilethesinglemasternodemanagesthewh
4、olesetofmetadataofthedatalocatedintheHDFS.However,thisarchitecturebringsasinglepointoffailure(SPOF)problem.IfthemasternodeofHDFSfails,thewholesystemwouldbeunavailable.ToaddresstheSPOFproblem,weanalyzedsomepopularhighavailablesolutionsintheindustryandafterwemakeagoo
5、dunderstandingoftheHDFSarchitecture,weproposeahigh-availabilitysolutionwhichimplementsahot-standbymechanismintheHDFS.ThissolutionsolvestheSPOFproblemintheHadoopwhileitbringslittleoverheadfortheHDFS.BaseontheoriginalHadoopHDFSarchitecture,weintroducedahotstandbyName
6、node.Asthesystemruns,themetadataissynchronizedcontinuouslybetweentheprimarymasterandstandbynodetomaintaintheconsistenceofthenamespace.Inordertoreducetheswitchtimeofthefailover,itonlyimportstheEdits.ThenthesystemcanswitchtothestandbyNamenoderapidlywhentheactiveNamen
7、odeisunavailable.ExceptthisSPOFproblem,thereisanotherproblemintheHDFS.Thevisitingrateofthedatablocksisdifferent,someishighandsomeislow.Ifweconfiguredafixedreplicationnumberofthedatablocks,wecan’tmakegooduseoftheresourceinthecluster.Soafterweanalyzedtheblockstorages
8、olutioninHDFSandtheparitycheckcodealgorithm,weproposedanoptimizedsolutionwhichisbasedontheparitycheckcodealgorithmfortheblockstorage.Weappliedthe