Optimizing Hadoop for the cluster.pdf

《Optimizing Hadoop for the cluster.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、OptimizingHadoopfortheclusterChristerA.HansenInstituteforComputerScienceUniversityofTromsø,Norwayhansen.christer@gmail.comABSTRACTbrie ysomerelatedwork.Section7concludesthispaperandsuggestssomefuturework.ThetotalnumberofclustersrunningHadoopincreasesev-eryday.Thereasonforthisisthatcomp

2、anieshavefoundasimplemodelthatworkswell.Themodelisbuilttoworkon2.TERMINOLOGYthousandsofmachinesandmassivedatasets.MapReduceTheHadoopopen-sourceframeworkconsistsofadistributedisascalableandfault-tolerantmodelthathidesallthedirtylesystemandtheprogrammingmodelitself.Thelesys-workforthepr

3、ogrammers.SinceHadoopisbeinginstalledtemiscalledHadoopDistributedFileSystem(HDFS)andonmoreandmoreclusters,congurationbecomesanimpor-issimilartotheGoogleFileSystem(GFS)[4].Hadoopistanttopic.ThisfactinspiredustondoutifweareablebasedontheMapReduceprogrammingmodeldesignedbytooptimizeHadoo

4、ptorunfaster.WehaverunexperimentsGoogle[3].Inthissectionweprovidesomehighleveltermi-ondierentcongurationsandfoundthatHadoopshouldbenologyregardingMapReduceandHadoop.conguredinawaythatitutilizestheresourcestheclusterAtypicalHadoopjobisasfollows:TheHadoopframe-hastooer.Thedefaultcong

5、urationofHadoophasprovenworkiscalledwithinaregularprogram.Theuserprogramtobeslow.WeuseourowncongurationandCloudera'sthemapandreducefunctions,andsuggestshowmanymap-Conguratortoprovethatperformancemayincreasesignif-perandreducertasksthatshouldbeexecuted.Theinputicantlybyjustchangingaset

6、ofparameters.leissplitupinto64MBblocksbydefault(butiscong-urable),andisdistributedamongtheslaves.ThelesareKeywordsstoredonthesamemachinesthatexecutethemappertasks.Theresultofamaptaskisasetofintermediatekey/valueConguringHadoop,Clouderapairs,thisdataisnotstoredinthedistributedlesyst

7、embutonthelocallesystemoftheslave.Thelocationsof1.INTRODUCTIONtheseintermediatekey/valuepairsarepassedbacktotheConguringHadooptoperformwellonaclusterislikemasterwhichisresponsibleforforwardingtheselocationstuningacar.Thereareseveralswitchesandknobsthattothereduceworkers.Whent