hadoop-数据挖掘研究组

《hadoop-数据挖掘研究组》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、HadoopIntroducingInstallationandConfiguration数据挖掘研究组DataMiningGroup@XiamenUniversityADistributeddata-intensiveProgrammingFrameworkHDFSMapReduceHadoopDistributedstorageParallelcomputing数据挖掘研究组DataMiningGroup@XiamenUniversityIntroducingtoHDFSHadoopDistributedFileSystem(HDFS)Anopen-sourceimp

2、lementationofGFShasmanysimilaritieswithdistributedfilesystems.However,comesdifferenceswithit.HDFSishighlyfault-tolerantandisdesignedtobedeployedonlow-costhardware.HDFSprovideshighthroughputaccesstoapplicationdataandissuitableforapplicationsthathavelargedatasets.数据挖掘研究组DataMiningGroup@Xiame

3、nUniversityHowitworks?FeaturesofitAnimportantfeatureofthedesign:dataisnevermovedthroughthenamenode.Instead,alldatatransferoccursdirectlybetweenclientsanddatanodes数据挖掘研究组DataMiningGroup@XiamenUniversityMapReduce?Let’stalkitnexttime………数据挖掘研究组DataMiningGroup@XiamenUniversity“RunningHadoop?”W

4、hatmeansforit?“RunningHadoop”meansrunningasetofdaemons.NameNodeDataNodeSecondaryNameNodeJobTrackerTaskTracker数据挖掘研究组DataMiningGroup@XiamenUniversityWhoWorksforwho?HDFSMapReduceHadoopNameNodeSecNDTaskTrackerJobTrackerDataNodeNameNodeHadoopemploysamaster/slavearchitectureforbothdistributeds

5、torageanddistributedcomputation.NameNodeisthemasterofHDFSthatdirectstheslaveDataNodedaemonstoperformthelow-levelI/OtasksNameNodeisthebookkeeperofHDFSkeepstrackofhowyourfilesarebrokendownintofileblockskeepstrackoftheoverallhealthofthedistributedfilesystemDataNodereadingandwritingHDFSblocksforc

6、lientscommunicatewithotherDataNodestoreplicateitsdatablocksforredundancy数据挖掘研究组DataMiningGroup@XiamenUniversityNameNodeandDataNodeSecondaryNameNodeSNNisanassistantdaemonformonitoringthestateoftheclusterHDFSdiffersfromtheNameNodeinthatthisprocessdoesn’treceiveorrecordanyreal-timechangestoHD

7、FScommunicateswiththeNameNodetotakesnapshotsoftheHDFSmetadataRecovery:NameNodefailure????WereconfiguretheclustertousetheSNNastheprimaryNameNodeJobTrackertheliaisonbetweenyourapplicationandHadoopsubmityourcodetoyourcluster,theJobTrackerdeterminestheexecutionplan