Hadoop添加节点的方法

《Hadoop添加节点的方法》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、Hadoop添加节点的方法自己实际添加节点过程：1.先在slave上配置好环境，包括ssh，jdk，相关config，lib，bin等的拷贝；2.将新的datanode的host加到集群namenode及其他datanode中去；3.将新的datanode的ip加到master的conf/slaves中；4.重启cluster,在cluster中看到新的datanode节点；5.运行bin/start-balancer.sh，这个会很耗时间备注：1.如果不balance，那么cluster会把新的数据都存放在新的node上，这样会降低mr的工作效率；2.也可调用bin/start-bal

2、ancer.sh命令执行，也可加参数-threshold5  threshold是平衡阈值，默认是10%，值越低各节点越平衡，但消耗时间也更长。3.balancer也可以在有mrjob的cluster上运行，默认dfs.balance.bandwidthPerSec很低，为1M/s。在没有mrjob时，可以提高该设置加快负载均衡时间。其他备注：1.必须确保slave的firewall已关闭;2.确保新的slave的ip已经添加到master及其他slaves的/etc/hosts中，反之也要将master及其他slave的ip添加到新的slave的/etc/hosts中mapper及re

3、ducer个数url地址：http://wiki.apache.org/hadoop/HowManyMapsAndReducesHowManyMapsAndReducesPartitioningyourjobintomapsandreducesPickingtheappropriatesizeforthetasksforyourjobcanradicallychangetheperformanceofHadoop.Increasingthenumberoftasksincreasestheframeworkoverhead,butincreasesloadbalancingandlowe

4、rsthecostoffailures.Atoneextremeisthe1map/1reducecasewherenothingisdistributed.Theotherextremeistohave1,000,000maps/1,000,000reduceswheretheframeworkrunsoutofresourcesfortheoverhead.NumberofMapsThenumberofmapsisusuallydrivenbythenumberofDFSblocksintheinputfiles.Althoughthatcausespeopletoadjustthe

5、irDFSblocksizetoadjustthenumberofmaps.Therightlevelofparallelismformapsseemstobearound10-100maps/node,althoughwehavetakenitupto300orsoforverycpu-lightmaptasks.Tasksetuptakesawhile,soitisbestifthemapstakeatleastaminutetoexecute.Actuallycontrollingthenumberofmapsissubtle.Themapred.map.tasksparamete

6、risjustahinttotheInputFormatforthenumberofmaps.ThedefaultInputFormatbehavioristosplitthetotalnumberofbytesintotherightnumberoffragments.However,inthedefaultcasetheDFSblocksizeoftheinputfilesistreatedasanupperboundforinputsplits.Alowerboundonthesplitsizecanbesetviamapred.min.split.size.Thus,ifyoue

7、xpect10TBofinputdataandhave128MBDFSblocks,you'llendupwith82kmaps,unlessyourmapred.map.tasksisevenlarger.Ultimatelythe[WWW]InputFormatdeterminesthenumberofmaps.ThenumberofmaptaskscanalsobeincreasedmanuallyusingtheJobCon