《HDFS简介详尽版》PPT课件

《《HDFS简介详尽版》PPT课件》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、HDFS简介杨林三2012.08.28提纲文件系统HDFS实验文件系统概览文件系统数据结构文件系统数据结构文件系统工作流程LinuxExt2LinuxExt3NFS小结HadoopHadoopHDFS目标Targets&CharactersVerylargefilesGB,TB,PB+Millionsoffiles10K+nodesStreamingdataaccessWrite-once,read-many-timesGooddatacoherencyTimetoreadwholedatasetmattersmorethantimet

2、oseekCommodityhardwareCheapNodefailure:thechanceoverlargeclusterishighRedundancythroughreplicasFailurehandlingandrecoveryHDFS目标Targets&CharactersOptimizedforbatchprocessingmovecomputationratherthandatalocationsofdataexposedtocomputationNotgoodforLow-latencydataaccessmsHi

3、ghthroughputagainstLow-latencyLotsofsmallfilesMetadatamayrunoutofmemorySeektimeoverreadtimeMultiplewriters,arbitraryfilemodificationAsinglewriterWriteatendWrite-once,read-many-timesHDFS架构Master/WorkersNameNodeMaster:onesingleFilenamespaceDirectoriesandfilesMetadataInmemo

4、ryMappingfilename->blocksblocks->datanodeClusterconfigurationReplicationManagementDataNodesWorkers:manyA"block"serverStoreblocksinlocalFSStoremetadataofblocksServedatatoclientsFacilitatepipelinetootherdatanodesHDFS架构DataStorageBlocks/ChunksBlocksize64/128/256MBConfigurab

5、lewhysobig?transferoverseekseek:10ms,transferrate:100MB,seektime/transfertime=1%=>blocksize=100MBblocksofafiledistibutedoverdatanodesaspossibleasingleFILE,existingofNblocks,isstoredonMnodesNamenodeFileSystemImageEditsNamenodeLogofcreation,deletion,renameofthenamespaceNam

6、espacetreeiscachedinRAMandstoredpermanentlyinFSImageEditlogtakesrecodeofopen,close,renamefileordirectory,etcSecondaryNamenodeWhy?PeroidicallyMergeeditsandfsimagePreventeditlogfrombeingtoolargeProvidecheckpointofNamenode'sfsimageasbackupincaseofNamenodecrashNamenodeHigh-A

7、vailabilityFsImageandEditLogarecentraldatastructuresofHDFS.AcorruptionofthesefilescancauseaHDFSinstancetobenon-functional.Forthisreason,aNamenodecanbeconfiguredtomaintainmultiplecopiesoftheFsImageandEditLog.MultiplecopiesoftheFsImageandEditLogfilesareupdatedsynchronously

8、.Meta-dataisnotdata-intensive.NamenodeSafemodeMergeFSImageandEditsEntersafemodeOffersread-onlyviewoffil