2023年IT十八掌徐培成HADOOP笔记

《2023年IT十八掌徐培成HADOOP笔记》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

Hadoop

ᘤ211212215216slOOs101S102slO3s104UbUntu204-server-64-2-00100osl00211nOdelUbuntU204-server-64-2-010101oslOlo212node2Ubuntu204-server-64-2-02oo102slO2oo215node3UbUDtu204-Server-64-2-O31Ooo3s103t>215oonode3Ubuntu204-serVer-64-2-04104ooS104o216node4a)bigdateᦪo1byte=8bito1024B=IM2A101024M=1G2A101024G=IT2A101024TIP2A10

11024P=1E2A101024E=1Z2A101024Z=1Y2A101231231024Y=IN2A10oᑖ஺ᑖHadoop(ᜮᜧ)DoUgcUttingHadoop☠!ᑖḄ#$%&஺HDFS'IOEo((BM+oracle+EMe)MapReduceMR//᧜+,ᓄ./0123₞Bigdata4V1)Volumn//⚪ᜧ2)Variaty//᪵73)VelOcity//89:4)ValueleSs//;<=9>

2b)Hado0pḄ?@:B?@UbuntuCDE?@jdkLn-s/soft/jdk-xxXjdk஺FGH᛻JJAVA_HOME=/SOft/jdkoPATH="…/sOft/jdk/b1n”SoUrCe/etc/environmentJava-version?@hadoopHadoop.tar.gzHadoOpVerSionFGH᛻JHADOOP_HOMEoPATHFGhadoopᑖKL11.stande1One|1oca1/MN/ᙢ1PᵨḄᙢᦻSCDNOthingTUᦻSCDḄVTUᦻSCDhadoopfs-Is/WᨵYZ[0ᡠᨵ0]^⃩ᙠaJVMb^ᵨtest,debug.

32.PaeudodistributedModeo//cᑖ13.Ful1ydistributedmodeo//dᐰᑖFGSSH1?@ssh$sudoapt-getinsta11ssh2fᡂ=hiSSh-keygen-trsa-Pஹ'-f~/.ssh/Cd-.ssh3lᐭnhᦪᑮᣴᩗrbCat~/.ssh/1d_rsa.pub>>~/.ssh/aUthoriZed_keys4EstuijloCalhOstSshlocalhOst5vᓄhdfsᦻSCDHadOopnamenode-format6wxᡠy[0Start-all.sh7ETU[0

4Jps//5RMNMNNDN2NN8)TUᦻSCDHadoopfs-Is9)z{ᦻSCDHadoopfs-mkdir-p/user/Ubuntu/dataHadoopfs-Is-R/c)Hadoopᒹ}KL1ᙽ1)HadoopCommOn:0஺ᢝᐸ1ᙽḄ(ᐹ1ᙽ2)HadoOpDistributedFileSystem(HDFS)0ᑖᦻSCDᵨ0]ᦪḄᔢ஺஺[0ooNameNOdeoo⁚஺ooNNoDataNodeoᦪ⁚°oDNooSecondaryNamenode⁚o002ndNN3)HadoopYARN09$ᳮḄ᪾஺

5஺஺[0oooResourCeManager$ᳮRMooNodeManager⁚ᳮᘤ-NM4)HadoopMapReduCeyarnCDḄiᜧᦪ[¡¢ᢈ¤⎽¦GhadOop1)Standelone/loCald)dᐰᑖ?@1)§ᜓ5©ᡝ

ᘤ211212215216si00slOlS102s103S1040000iP00«0ip001.Ubuntu204-Server-64-2-00100si00211nodel2.Ubuntu204-serVer-64-2-011oo01ooslOloo212onode2

63.UbuntU204-server-64-2-02o102osi02215nOde34.Ubuntu204-server-64-2-03oo103osi03oo215node35.UbUntU204-server-64-2-0410oo4os104o216node42?@ssh1?@sSh$sudoapt-getinstallssh2fᡂ=hissh-keygen-trsa-P''-f~/.ssh/cd~/.ssh3lᐭnhᦪᑮᣴᩗrbcat~/.ssh/id_rsa.pub>>~/.Ssh/authorized_keySScp/root/.sSh/*nOde2@:/rOot/.sSh/SCp/rOot/.sSh/*node3@:/rO஺t/.ssh/Sep/root/.Ssh/*node2@:/root/.sSh/

749stᐸ
ᘤsShnode1ifConfig39?@jdk1.rpm-iVh/opt/jdk-7u79-linux-X64.rpm2.In-s/soft/jdk-xxxjdk3.FGH᛻J4.JAVA_HOME=/s0ft/jdk5.PATH="…

86.⎽¦GhadoopTUᦻSCDhadoopfs-Is/FGᦻS/etC/hadoop/core-site.XmlfS.default.namehdfs//nOdel:8020Hdfs-site.xmIீname>dfs.replication3

9Mapred-Site.xmlmapreduce.framew0rk.nameyarnYarn-site.Xmlyarn.resourcemanager1ocalhostyarn.nOdemanager.aux-serViCeSmapreduCeshuffle

1059FGᦻS-rw-r--r.1rootroot8616ᨴ610:41core-s1te.Xml-rw-r--r—.1rOotrOot9506ᨴ610:41hdfS-site.Xml-rw-r—r--.1rOotroOt8446ᨴ610:41mapred-Site.Xml-rw-r—r.1rOOtroOt7286ᨴ61043yarn-site.xml-rw-r---r—.1rootroot126ᨴ610:43slaves/soft/hadoop/etc/hadoop/core-site.xmfs.defaultFS=hdfS//nodel//Soft/hadoop/etC/hadoop/hdfs-site.xmlreplicat1on=3

11dfs.namenode.secondary,http-address=node450090/soft/hadOOp/etc/hadoOp/mapred-S1te.xm1mapreduce.fraITlework.name=yarnsoft/hadoop/etc/hadoop/yarn-site.xmlyarn.resourcemanager.hostname=nodel/soft/hadoop/etc/hadoop/SlaveSnode2node3ᙠ®ᑖ¯y®KLᦻSCd/sOft/hadoOp/etc/hadoopxsynCcore-site.XmlxsynCyarn-site.xmlxsyncslaveS69°±wxhadoOp19vᓄᦻSCD$>hadoopnamenode-format2wxᡠᨵ[0

12$>Start-all.sh3)T²[0ேsp4)´µᡠᨵ[0Stop-al1.shPᵨwebuihado0phdfS1)hdfshttp/nodel:500702)dataNodehttp://node2:500753)2nnhttp://node4:50090ᑤ◀had஺Op¸¹ºtᦪ»¼/tmp/hadoop-roothadoop-root-datanode.pid(cᑖ½f)e)IJKLscprsyncxSyncXCalI1)scp2)rsync¾0a¿(ᐹÀ⌕ᜓÂ,ÃÄᢝÅÆÇᜓ஺rSync-rvl/soft/*ubuntU@sl01:/soft39ÈÉÊËXSync,ᙠ®ᑖ¯ᦻS

13ÌHÍᑴᦻSᑮᡠy⁚ḄaºtÏrsynC-rvl/home/UbuntUUbuntu@s101:Xsynche11O.txt[/usr/loca1/bin/xsynC][root@nOde1bln]#vimXSync#!/bin/bashpcount=$#if((pcount<1));theneChonOargs

14Cuser=whoamifor((host=2;hoSt<5;host=host+1));doecho--node$host-------#echO$pdir/$fname$cuser@nOde$hoSt:$pd1rSCp$pdir/$fname$cuser@nOde$hOst:$pdirdone4)/ᑏ/usr71oCal/bin/xcallËᙠᡠᨵ«
®᡻aḄÙÚXcal]rm-rf/soft/jdk[/usr/10ca1/bin/xcal1][root@nodelbin]#cd/usr/loca1/bin[root@nodelbin]#xca11Is-1/soft/[root@nodelbln]#xca11rmhe11O.txt[root@nOde1bin]#vimxCall

15#!/b1n/bashpCount=$#if((pcount<1));thenechOnOargs;ex1t;fieCho----------1OcaIhost------------$@echo$@for((host=2;host<5;host=hOSt+1))

16᦮ᳮjarᒹ஺ÞÒᡠᨵFGᦻS[core_default.xml]ohadoop-Common-2.7.2.jar/Core-default.xmlo[hdfs-default.xml]hadoop-hdfs-2.7.2.jar/hdfs-default.Xmloo[yarn-default.xml]ohadoOp-yarn-cOmlTon-2.7.3.jar/hdfs-defau1t.xml0o[mapred-default.xml]ohadOop-mapreduce-Client-core-2.7.2.jar/mapred-default.Xmlg)hadoopfs/Xᵨ:hadoopfS-mkd1r-p/user/UbUntu/datahadoopfs-puthe1lo.txt/user/Ubuntu/data/

17hdfsdfs-isr/hdfsdfS-cat/user/hello.txt./hadoopfS-Is/./hadoopfs-Is-R/./hadoOpfs-mkdirtest./hadoopfs-put/opt/test.Sh/userßᙢᦻSuhadoop./hadoopfs-get/user/t/Ok.txt/home/tßhadoop®àLᦻSdownuᙢáᨵºtÏ./hadoopfS-cat/user/test.shhadoopfs-rmr/user7tᑤ◀had஺஺p®ᢣÉᦻSᜳBᒹ}äºtåEhadoOpfs-toUchz/user/new.txtoᙠhadOopᢣɺtÏæ{LçᦻShadoopfs-mV/user/test.txt/user/ok.txtBßtest,txtÀÙèOk.txthadoopdfs-getmerge/user/home/tßhado஺pᢣɺtÏᡠᨵᑁêëèLᦻSa¹downuᙢ

18hadOOpjob-ki]1[job-id]ßíᙠ^⃩Ḅhadoopki11ᣵhZᦋLᙢḄ^_`ahad0op.tmp.dir1îᦋhado0p.tmp.dir[core-Site.xml]hado0p.tmp.d1r/soft/hadoop-root2ᑖ¯core-sIte.xml3hadoopnamenode-formatðÀwhadoOpXcallmkd1r/soft/had0Op-roOtoXCallrm-rf/soft/hadoop/1ogsihadoopbᙽ1.COmm0n2.hdfsstart-dfs.sh/stop-dfs.shBᐜwxòEnndn

192nn3.yarnstart-yarn.Sh/stop-dfs.shrnnn4.mapredj)hadoopKLᑖ᪆1.start-al].shlibexec/hadoop-config.sh--ÇNJsb1n/start-dfs.Sh-config$HADOOP_C0NF_DIRowxhdfsSb1n/start-yarn,sh-Config$HADOOP_CONF_DIRoowxyarn2.11bexec/hadoop-conf1g.shoooooo--ÇNJCOMMON_DIR0...HADOOP_CONF_DIR=...HEAPSIEZ=1000M,

20oCLAsSPATHi.3.Sb1n/Start-dfS.Sh-Config$HADOOP_CONF_DIRoowxhdfsoo1.libexeC/hdfs-COnfig.sho2.ÑÒ⁚«
ooNAMENODES=hdfSgetcOnf-namenodeS3.wx⁚$HADOOP_prefix/Sbin/hadOop-daemons.sh—cOnfig$hadoop_cOnp_diroo--hostnames$namenodes--script$bin/hdfsstartnamenode$nameStartOpt4.wxdatanode$HADOOP_prefiX/sbin/hadOop-daemOns.sho—config$hadOop_conp_diro—sCript$bin/hdfsstartdatanode$nameStartOpt5.wx2nn$HADOOP_prefix/sbin/hadoop-daemons.Sh

21ooo—config$hadoop_conp_d1roo--hostnames$secondarynamenodeo--script$bin/hdfsStartsecondarynamenode4.11bexeC/hdfs-COnfig.sholibeXec/hadoop-config.sh5.sbin/hadoop-daemons.shooo--wxYZ[0Ëo1.libexec/hdfS-COnfig.shoo᡻FGË2.s1aVes.sh—cOnfig$HADOOP_CONF_DIRcd$hadoOp_prefic\;$bin/hadOOp-daemon,sh-config$hadoOp_conf_dirஹ'$@"ÌHsiavesᦻS÷øsshVst¾0«
᡻ÙÚo[bin/hadoop-daemon.Sh]ohadoOp-config.sh[bin/hdfs]hdfSgetconf-namenodeshdfsgetcOnf-secondaryNameNdoes

22kᙠᔜL
ä®wx⁚hadoop-daemon.shstartnamenodehadoop-daemon.shstartsecondarynamenodehadOop-daemon.shStartdatanodehadoop-daemon.shstopdatanodeᦪrLᱯoaatomicûäឋococonsistentoýឋoiisolationooo◞ÿឋoddurableo஺ឋ1ᵨᓄhadoopnamenode-fOrmatpllt===copyFormLocalhadoOpfs-pUt//hdfsdsf-puthdfsdfs-put//ohdfsdfS-mV./sshd.tXt./hOW.txthdfsdfs-get./sSh.tXt./kk.txtohdfsdfs-CopyToLocal

23UhdfShdfsdfs-moVeFromLocalhellO.txt.hdfsᑮᙢᨵ஺ᑤ◀ᦻhdfSdfs-rmdir...ᙠhdfs"#$ᦻ%ᑴhdfsdfs-Cp/uSer/Ubuntu/hello.txt/user/hel1o2.txthadOOp'()*namenodeḄᙢ,-'(ᡂ/0ᑣ20,-34ᑁ6789☠ឋ஺[hdfs-Site.xml]°dfs.Damenode.name.dir=f1le///${hadoOp.tmp.d1r}/dfS/name1,file:///${hadoop.tmp.dir}/dfs/na!Tle2datanode;9<'(/0,-=>?,-஺oo[hdfS-site.xml]oodfS.datanode.name.dir=file:///${hadoop.tITlp.dir}/dfs/nalTe1,file:///${hadoop.tmp.d'ir}/dfs/name2

24hadoophdfsᦻᜧA128MCDḄE⍝GH=10msCDḄI஺J᳛100MB/S10*100=Is*100M/s=1OOMLMAPINOhdfS»@TestpubIicvoidwriteFile()throwsException{ooConfigurat1onconf=newConfiguration஺<஺FileSystemfs=FileSystem.gef(conf);oPathpath=newPath("hdfs://node1:8020/user/hel1o");஺஻ᦪRSTU஺FSDataOutputstreamdos=fs.create(path);஺dos.write("hel1oworld!!!".getBytes());0dos.c1ose();System.out.println("over");6)

25mᙠeclipseVWhad0opXYZ[NOhdfShadoop2x-eclipse-p1ugin.zip஺\]had஺Op2X-ecl1pse-plugin.zip/release/*.jarj39I]eclipseVW,-/plugins/0^eelipse_`hadoopXY>ᔲᡂbᒈ᪆ᦻᑏᐭMg?⁚iḄ⌱klmnopqrstmnopuvwDNSToSW1tchMapp1ngxy஺⎽{(ᦻcOre-Site.xml[core-Site.Xml]net.topO1Ogy.nOde.Switch.mapping.impl=com.it18zhang.rackaWare.MyDNSTOSWichmappingᑖ}core-site.xml~0ᦪR⁚i஺◖mᘤ0⁚i^⁚ihdfs>master/slaven᪀ᵫ^⁚i

26NamenOde34ᐗᦪRl?ᦪᩗ▲ᑡ⊤q=ᒹᦪR⁚i஺rackawareᶍᵨoie஺eVnamenodeḄᦻ¡¢£ᦻ¤¥¦ᐹhdfsoiV-pXML-IdfS/namel/current/fS1mage_000-o~/1mage.xmlhdfSoiv-pXML-ixXx-o~/xxX.xml//fsimagehdfsoev-pXML-ixxx-o~/xxx.Xml//edit1oghadOop-daelTlon.shstopnamenodehadoop-daelTlon.shstartnamenodehdfSdfsadmin-helpsaveNamespacennḄᦻhdfsoiv-pXML-IfaimaQe-OxXxnnḄ¡¢£ᦻhdfsOev-poXMLfaedit-xxx-oxXx

27nhadoop¨©^>Z☢#ᐭVᐰ¬஺Ꮇ᝞¨©T¯Vᐰ¬=°᡻$⌕³´lᑏ³´qO¨©^µ¶·r⌨TVᐰ¬஺Vᐰ¬³´hdfsdfSadm1n-safemodeget//Vᐰ¬º᝱hdfsdfsadmin-safemodeanter//#ᐭVᐰ¬º᝱hdfSdfsadmin-Safemodeleave//¤¼Vᐰ¬º᝱hdfsdfsadmin-safemodewait//½¾Vᐰ¬º᝱¿3ÀÁH/Âᔠ¡¢£#ᐭVᐰ¬oohdfsdfsadmin-safemodeenter஺#$¿3ohdfsdfSadm1n-saVeNameSpaCe஺⌨TVᐰ¬ohdfsdfsadmin-safemodeleaveÄឋ¬Å

28scpÆ0ÇgÈmÉHḄᦻ%ᑴ஺$>scp-rhello.txtroot@nc>de4:/user/xx.txt$>scp-rro஺t@node4:/User/xX.tXt~/$>SCp-r-3rOot@node4:/user/hell஺.tXtrOOt@noed1:/user/distcpÆ0hadoop¨©ÉHḄ⌴ÌᦪR%ᑴohadOopdiStCphdfs://slOl:8020/User/he11O.tXthdfs://S102:8020/uSer/kk.txtÌ᫏ᦻhadooparchive-archiVeNamemyhar.har-p/user/Ubuntu/uSer/myÌ᫏hdfsdfs-1Srhar:///User/my/myhar.har°\Ì᫏hdfsdfs-cphar///user/my/myhar.har/user/yourᦪRµ᦮ឋ0

29oqᙠUbuntU"VWeelipseoeclipase-jee-mars-R-linux-gtk-x86_64.tar.gzotar■ὃijubuntu/SoftTtar-XZvfeClise-xXX.tar.gz'(Ó᛻ÕÖPATH—.:/soft/eclipse^l·qeclipse&pcodec×Ø\×Øᘤ1.deflateoo.def1ate2.gzipo.gz3.bzip2o.bz24.1zOoo.IzO5.Iz4oo.1Z46.snappyo.snappydeflateCOdeC:olength=2054672,compreSsTime=l766,decompress=615GzipCodeC:olength=2054684,ocOmpressTime=680,oodecompreSS=341

30BZip2codec:olength=1457864,compressTime=10271,decompress=1695LzoCodec:ooolength=3636556,ocompreSsTime=499,odecompreSs=303Lz4Code:1ength=3644382,oCompressTime=177,decompress=392SnappyCOdeo1ength=3987117,compresST1me=424,odeCOmpress=434ÁHᦔ°Bzip2>def1ate>Gzip>1zO>lz4>snappy]ÚGHlz4oo>snappy>Izo>GZ1p>def1ate>Bzip2\]ÚGHlzo°o>GZip>1Z4>snappy>deflate>BZip2ᵨhadOopḄchecknatiVeÛᙢÜVWÝÞhadoopcheCknative-aVWlzo-2.06᱐1zo-2.06.tat.gz./cOnfigure-enable-sharedmakemakeinsta11

31Û>ᔲàᡂIz஺Üᦻfind/usr/IOcal/1ib|grep1ZOáhadOop-1ZO-master⚗,hadoop-lzO-master.zipáhadoop-lzooc_INCLUDE_PATH=/usr/1ocal/includeLIBRARY_PATH=/USr/1Ocal/libãὃᦻävmavenÓ᛻஺mvncleantestTIzoᐳæÜç=ᑮèேib1Zo2.so.xxxᦻ%ᑴ/lib,-$>rsync-1/uSr/loca1/lib/*1Zo*roOt@nOdel:/libmvncleanpaCkageᙠhadoop⚗,ê¨ᡂhadoop-lzo-masterḄjarᒹ1.%ᑴàᡂḄjarᒹᑮ⚗,lib2.cptarget/hadoop-1zo-0.4.20-SNAPSHOT.jar~/workSpace/hadoop2-01/lib3.ëᦋíṹ

32q)centos"VWᵨmaVen:oapache-maven-3.3.9-b1n.tar.gziJ/sOft'(mavenḄÓ᛻ÕÖetC/env1ronmentM2_HOME=/soft/mavenPATH=…/soft/maven/bin'(mavenḄsetting.xmlᦻïxᑮᙢðÜñòᘤ஺óô:mavenVWᙠcentos"nexUSVWᙠw1nlO"

33஺'(mavenXYoeclipSeøW1ndOws->✌⌱⚗-ümaVen->usersettingS->ýþ#$ÿ
userSettings/soft/maven/conf/settings.xmllocalrepository:/hoITe/UbuntU/.m2/repoSitoryr)win10maven:1.tomcat2.nexus.Warᦻ3.ᑴnexuS.Warij${t0mcat_holTe}/webappS/4.tomcat5.http://locaIhoSt:8080/neXuS/6.ᙢᘤmavenḄtomcat/webapps/nexus/WEB-INF/clasSes/nexUs.propert1es

34nexus-wOrk=E:/maven-repo/nexuSruntime=${bundேeBasedir}neXus-app=${runtime}ᙠwin10ᵨmaven஺1.ljhello-world⚗ᡠᙠḄᦻᜳᙠhel1O-wOr1dᦻᜳ"#pom.Xm1o/he1lo-world/pom.xmlᡭ%pom.xm14.0.0cOm.itl8zhang.helOo-worldheloo-world1.0-SNAPSHOTmavenhelloWOridproject

35jun1tjunit4.11oteSt2.(ᦻᜳ
cd\workspace_eclipse\hello-Worldmkdir\srC\ma1n\java\com\1t18zhang\helloWorId3.ijHei1oWor1d.javaᦻpackagecom.itl8zhang.helloworid

36,◤⌕mavenapache-maven-3.5.0,/᛻12D\software\apache-maven-3.5.0\bin34mVn-v9mVncleancOmp1leITlvnclean5.(567ᒹooo6•(ᒹoooSᦻ1.txt9ᦻ:;<=>?@)2.SequenceF'ilekey-Va1ue//map3.AᑏSequeneeᦻCṹpUbliccIassTestSequenC6File{0/*o*sequenCeFileo*S6qUenceF1lekey-Valliersmapo*/o@Test

37opUbliCvoidwrite()throwSException{Configurat1onconf=newConfiguratiOn()

38owriter.append(newIntWr1table(1),newText("tom5n));Writer.CloSe()

39Configurationconf=newConf1gurat1on()

40o஺O᣸SḄseqfiieᐹᨵVWI஺index(seq)data(seq)XYkeyᢥ᯿ᜧ]^S_`Ḅ஺indexVabcd2Ḅ᧜f(seq),ghij◞l,no128,pq⎽sᦻio.map.1ndex.interva1ij◞l

413.DataOuputStream4.Text//jaVa.lang.Stringjava?ᓄ஺ᨵᐗᦪJ(ᦪJ7)ᑏᐭ<ᦪJ2ᜧ<ᦪJjOḄ<ᙠaᵨḄ஺java:3867297ohadoop:47591150ClassAddress{StringproVince;oStringcity

42ᵨutf8ᦪJ஺,ᵨ1/2/3/⁚ᦪJ2.writeByteS
''XXx"ᑏᐭ¡⁚஺3.writeCharS
ஹ'xxx"⁚஺4.ArrayWr1tableArray5.MapWritabieMap6.NullWritableᨵ?ᓄ/¤?q¥<¦§ᦪJ¨©bª«¬<®®O¡ᓰ°஺ᓫ²³;,´µ;.SecondaryNamenode1.¶·¸¹⁚º2ndNNoo2.oivooffimagevieweroo3•OSV04.(»Mºq¥5.2nn(»MºḄᕜ½hdfsdfSadmin

43஺1.i¾¿ᳮ0)¾ÁᑴḄ)ᡠᨵᦻ/ᦻᜳḄᦪ<1⊤ÃÄ)஺ᒹÅÆᦻ஺o$>hdfsdfsmain-setQUOta1/USer/Ubuntu/data//i¾o$>hdfsdfSadm1n-clrQuOta/User/Ubuntu/dataoÄj¾o஺o஺ÇᦪÈÉᙠᑁ<ÁᑴḄᡠᨵᦻËᜧ]஺lÌᜧÍ>384m,ÎÏÄjᨬÌÐὑÒO384m.oo$>hdfsdfsadmin-helpsetSpaceQuota//MNÄj¾ooo$>hdfSdfsadmin-setSpaceQUota20mdata//iÄj¾oo$>hdfsdfsadmin-clrSpaceQuotadata//ÓᳮÄj¾02.Ô᯿஺஺Ô᯿¹ÍÕ)Ö¡ᜓØ஺ÙÚÛᑴᡠᨵᦻ<ÜOᢣᔣÝ¡ᦻ஺Þᑏᐭßàá<âÚãàäᦻ஺

44oo$>hdfSdfs-he1pCreateSnapShot//MNÔ᯿o$>hdfsdfSadmin-al1owSnapShotdata஺//ᢣY)ḄÔ᯿åæhdfsdfs-CreateSnapShot/user/UbUntu/data//Õ)(Ô᯿o$>hdfsodfs-createSnapShot/user/UbUntu/dataXpc-0902஺//ᢣY¸¹è¸Ô᯿oo$>hdfsdfs-renameSnapShotdatasO902-0040.890UbUFltU_9020஺//éè¸Ô᯿ooo$>hdfS1SSDapshottableDir//êUëÞ~ᵨᡝᡠᨵÔ᯿)oo$>hdfsdiffsnapshotdata.xpc_0902஺//Ô᯿)ḄÝíᜐ3.ïᦈñtrash,òḄ¡ᦻᜳ஺fS.traSh.interVal=0o//ᑖôᦪ<0⊤ÃἭᵨïᦈñ<ᑤ◀ᦻḄ÷áj஺fs.traSh.checkpoint.1nterval//»Mïᦈñj◞áj<òÞ<=÷ájv0⊤ÃbfS.traSh.interval¡ú஺

451.ïᦈñcore-site.xmlfs.trash.interval=1஺஺2.ᑖßᦻ<Ûàᦔïᦈñ°
/user/UbuntU/.TraSh/...஺3.pq¥Sᑤ◀ḄᦻÚdqïᦈñ<◤⌕pqm஺veTOTrash()âtᐭïᦈñ஺ooTrashtrash=newTreash(conf);ootrash.moVeToTrash(path)<4.üýïᦈñhdfsdfs—expunge//ÓÄïᦈñ5.þᦋhdfsḄwebu1Ḅ☟᝱ᵨᡝḄ஺oocore-site.xmloohadoop.http.StatiCuSer.useroUbuntuoot)ᜧᦪ:1.

46hdfS:hadoopdistr1bUtedfilesystem.GFSᑖ2.〈ᑖ஺MapReduce
MRO//᧜+ᓄ.Key-Valueᑏ1.!Mapper"#$%&'ᫀ)*2.+I,!RedUcer3.+-MapReducer4.⃩5./01ṹ34map5reduceᑖ6⃩ᙠ⁚9:;MapReduce:Job°=>MapTaskmap?@ooooReduce:taSkomap#᧜*+reduce#ᓄ*MapreduceAB᡻D0E1.F1ṹ2.GHjarᒹJᦋpom.xmlᦻM

47஺ᑤ◀junitḄSCopePᑖojunitoojUnitoo3.8.1ᙠ⚗RSḄpom.xmlᦻMTU⃩mavenbuild...Globals:packageVWXjarᒹ3.:Yubuntu4.ᙠubuntu:Z[jobhadoopjarxXx.jarcom.xXx.xxX.xxxApp/User/UbUntu/data/ncdc/uSer/Ubuntu/out25.winl0\]Z[AB^ubuntu.[_`aḄb⚪஺eelipSedḄmaVen⚗R◤⌕ghF1ṹij⚗RklmnoSplmqᵨrstuvᑴSpF1ṹ$>mVndependency:Sources-DdownloadSoUrCes=true

48xyz:[x}(slOO,namenode)]1.ghjavaxyzḄ~c:/>java-agentlib:jdwp=help2.javaḄxyzjaVa-agentlib:jdwp=transport=dt_soCket,server=y,suspend=y,address=8000//JavaḄᦪjᵨxyz3.^namenodesjvmḄxyzja.Jᦋbin/hadfssHADOOP_NAMENODE_OPTS=$HADOOP_NAMENODE_OPTS-agentlib:jdwp=transpOrt=dt_socket,server=y,suspend=y,addresS=8000b.ᙠshelld᛻$>exportHADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS-agentlib:jdwp=transport=dt_sOcket,server=y,suSpend=y,address=8000”c.xyzjᐵxyz

49$>exportHADOOP_NAMENODE_OPTS=4.s⁚9$>hadoop-daemOn.shstartnamenOdeᡝ}1.namenode"jᙠmainᦪ:ᡭ¡9.2.ec1ipseTUyz->remotejavaapplication⚗REmyhad0opConnectionType:Standard(socketattach)hoSt:slOOport8ooo3.yz£¤yzᙠwin5UbUntu:/0xyzjghjobᙠhadoop¥¦:Ḅ᡻D0xooxxs100,UbUntuy1.Y⌴jar¨ijubuntu2.HADOOP_cLIENT_OPTS᛻$>exportHADOOP_CLIENT_oPTS="$HADOOp_cLIENT_OPTS-agentlib:jdwp=transport=d

50t_sOcketserver=y,sUspend=y,address=8f000z/3.sjobZ[tu$>hadoopjarXxx.jarcom.xx.x.x.Appdata/ncdc0ut3oo[ᡝ}jwin7,eclipse]1.ᙠF1ṹd¡9App.main()...2.eclipseTUyz©ுremotejavaapplicat1On⚗RmyhadoopConnectionTypestandard(socketattach)host:al00port:80003.yz£¤MRAMoo:MapReduceAppmasterresMgrDelegateo:ResourceManager1ᳮYARMoResourceManagerooNodeManagergoOgle

51oPBprotocalbuffer,¬®¯஺RPCoogoOg1eprotObufRemoteproCedureCa11,x0yᵨec1ipsesᨴ±@}jᡝ}ipcᑗᱏᜧµ5block¶j஺஺·ᓄ¸ᶍ(map),1.ᦪºᙢᓄ·ᓄ¸ᶍ2.¼ᙢᓄ¸ᶍ3.k½¼⃩?@Partitionᑖ¯Combiner:ᔠᡂᘤ஺ÁÂmap5reduceÃÄḄYÅ஺smrABᔊÇÈ@ᘤ1.sXca1mr-jobhistory-daemon.ShStophistoryserVerwordc0Unt0ᓫÊËÌ

520஺ÍÎᑖᒘḄᓫÊblockEhdfs//Ñjhdfs-Site.xml512ḄÒᦪ//512᪥35ᑗᱏ0Spl1t000000000//mrABᶖITlinSplit,maxSplit,bioCkSize//ÕdÄÖE஺ᑗᱏḄᦪ×ØÙmaptaskḄᦪ¥¦ÚᣚᙢEomapred-s1te.xml1oCaloyarn-site.xmlo01ocalhostocOre-Site.xml00DebugConfigurationS->ArgunentsD/mrd/mr/out/

53UᨬtuEwhichstart-all.Shfind.ocath1>>h0hdfsgetConf-nnRpcAddresseseclipseᓫÜÝ0ᓫÜÞᐭhdfSᑖᦻMàáNameNOdeSecondaryNameNodeᥜ]ᜧqãäåãDataNodeNameNode:NameNOdeḄmetadateæçᙠsèépᑮᑁ

54SecondaryNameNode:SNN᡻Dᔠìijí⌕îA~NNᔠìeditslog,ÁïNNsiÄ஺DataNodeᦪº(block)-rW-r—r--.1rootro0t63851630May1000:09hadOop-1.2.l.tar.gz/opt/hadoop-1.2.1/confvimCOre-Site.Xmlfs.default.namehdfs//10.10.1.206:9000hadoOp.tmp.dir

55/opt/hadoop-1.2vimhdfs-site.xmldfs.rep1icat1on2[rOOt@loca1hostConf]#Vimslaves10.10.1.21210.10.1.215[roOt@1ocalhostconf]#vimmasterS10.10.1.212

56ssh-keygen-tdsa-P-f/•ssh/id_dsacat~/.ssh/id_dSa.pub»~/.ssh/authorized_keysz{ᓄ./hadoopnamenode-formatjava--versionVimhadoop-env.sh#ThejaVaimp-Iementationtouse.ReqUired.exportJAVA_HOME=/USr/jaVa/jdk1.8.0_101./Start-dfs.sh./Stop-dfs.shjPS:5007050070

57(ߟ)MapReduceᑖð᪾¼MapReduceòsðjókòsᦪºsplit->map->Sort->reduce#å*ShUfflerð᪾¼ᙠmapper5reducerdÄḄ©ô⁚õöP÷EMapReduceøJobTracker,ùTaskTracker.õöDataName:vimmapred-site.xmlmapred.job.trackerio.0.27.206:900150030

58date-s"2023-05-171503:00HadoopfstuE1,Hadoopfs-fs[1Ocal|]:üýhadoopqᵨḄᦻMàá஺2,hadoOpfs-Is:þ½ÿᙢḄ1s,ᑡᙠᢣḄᦻᑁᢝpatternᓛ஺᝞fi-Iename(fullpath)s1ze.ᐸn⊤replicaḄ!ᦪsize⊤ᜧ$(ᓫ'byteS)o3,hadoOpfs-Isr:⌴)*ljᓛ⎽,patternḄᦻ-./01s,234⌴)ᑡᡠᨵ7-.஺4,hadoopfs-du:*ljᓛpatterFlḄᢣḄᦻ89:;(ᓫ'bytes),=>?unixḄ┐AḄdu-Sbீpathு/*F┐AᦻḄdu-b᝞Dame(fullpath)Size(inbytes)o5,hadOopfs-duS:=>?-du,HIJ234=>?unixḄdu-sb஺

596,hadOopfS-mvKᑴḄᦻmoveᑮᢣḄḄ'N஺OsrCPQ!ᦻRdstST4!஺7,hadoopfs-Cp:UVᦻᑮḄ'NOsrcPQ!ᦻRdstST4!஺8,hadoOpfS-Pm[-skipTrash]ᑤ◀ᓛpatternḄᢣᦻ=>?unixḄrmo9,hadoOpfs-rmr[skipTrash]:⌴)ᑤᣵᡠᨵḄᦻF=>?unixḄrm-rfo10,hadOopfs-rm1[SkipTrash]:=>?unixḄrm-rfio11,hadoopfs-put>1,:\ᙢUVᦻᑮDFSo12,hadoopfS-copyFrOmLocal>1,?-put013,hadoOpfS-mOveFromLocal•••=J?-put,234]ᦻᙠUV^_ᑤ◀஺14,hadoOpfs-get[-1gnoreCrC][-crc]<10Caldst>:>y

60ᦻᦻᓛpattern,b4Q!ᦻᑣdStST4஺15,hadoopfs-getmerge<1oCa1dstு:defg\DFSUVQ!ᦻஹᔠj᣸lPm!ᦻᑮᙢᦻ஺16,hadOopfS-cat=>?-get஺18,hadoopfs-mkdirᙠᢣ'Nqr஺19,hadoopfs-setrep[-R][-w]PATH…n~ᦋᦻḄᩗ▲-R᪗⌴)~ᦋ஺MODEPa+r,g-w,+rwx=OCTALMODEP755᪵஺21,hadoopfs-chown[-R][OWNER][:[GROUP]]PATH…n~ᦋᦻḄᡠᨵὅF஺-R⊤⌴)஺22,hadOopfs-chgrp[-R]GROUPPATH…:=>?-chOwn'1,:GROUP

6123,hadOOpfS-cOunt[-q]nᦪᦻ!ᦪ}ᡠᓰ89Ḅ⊤ḄᑡḄgPnDIR_COUNT,FILE_COUNT,CONTENT_SIZE,FILE_NAMEᡈὅᎷ᝞-qḄᑡQU஺TA,REMAINING_QUOTA,SPACE_QUOTA,REMAINING_SPACE_QU0TAo⌶n[root@nodelbln]#./hadoOpfs-Is/o./hadoopfs-mkd1rtest./hadoopfs-put/opt/test.sh/userKᙢᦻ`hadoop./hadoopfs-get/user/t/Ok.txt/hOme/tKhadoop!ᦻdown`ᙢᨵ./hadOOpfs-cat/user/test.Shhadoopfs-rmr/User/tᑤ◀hadoopᢣᦻᜳᒹ7=¢hadoopfs-touchz/user/neW.txtᙠhadoopᢣ£rm!8ᦻhadOopfs-ITV/user/test.txt/user/Ok.txtKtest.tXt¤¥ePok.txt

62hadoOpdfs-getmerge/uSer/home/tKhadoopᢣᡠᨵᑁ¦Pm!ᦻJRdown`ᙢhadoopjob-kill[job-id]K§ᙠ¨⃩Ḅhad஺஺pª«ki11ᣵjavaᡭjarᒹn⚗->export->java->JARfile->⌱¯°஺m±./had0opjar/opt/MapRedUceTo01s.jarmrOl.JobRUnssh-keygen-tdSa-P''-f~/.sSh/id_dsacat~/.ssh/id_dsa.pub>>~/.ssh/author1zed_keyshad0op2.7.3=-=---------=cd/opt/hadoop2.7.3/etc

63vimhadOop-enV.ShexportJAVA_HOME=${JAVA_HOME}exportjAVA_HOME=/usr/java/jdk1.8.0_101Vimhdfs-site.xmlvimcore-s1te.xmlVimsiavesnode2node3node4²³´µNnNNDNoZKoZKFCJN0RMoDMHMRSnode111111node21111111node3111111

64nOde411111¶·JournalNodeCd/opt/hadoOp-2.7.3/sbin/./hadoop-daemon.shstartjournaInode./hdfsnamenode-formatᙠnode1±ᓄcd/opt/2hadOop/dfS/name/current¹ᦪºᦻnodel./hadoop-daemon.shStartnamenodenode1¶·namenodecd../IOgStail-n100hadooP-rOot-namenode-nodel.outvim/Opt/hadOop-2.7.3/1Ogs/hadoop-rOot-namenode-node1.Outcd/opt/hadOop-2.7.3/bin/

65./hdfsnamenode-bootstrapstandbynode2UVcd/opt/hadoop2/dfs/nalTe/currentnode2/opt/hadoop-2.7.3/b1n./hdfszkfc-forlTlatZKᓄzkfCcd/opt/hadoop-2.7.3/Sbin/./Stop-dfs.shCd/opt/hadoop-2.7.3/sbln/./start-dfS.sh./hdfsdfs-mkdir-p/Opt/file./hdfSdfS-put/opt/jdk-7u79-1inux-x64.rpm/Opt/f1le:50070Utilit1eS->BrowseDirectory»¼ᦻᜧ$:50070

66ResourceManager¶·ncd/opt/had஺0p-2.7.3/etc/hadoopvimmapred-site.Xmlvimyarn-site.xmlsbin]#./start-yarn.shReSoUrCeManagernodemanager:8088eelipse´µhad0op°ᒹnhadoop-2.7.3\share\hadoop\cOmlTlonhadoop-2.7.3\share\hadoop\common\libhadoop-2.7.3\Share\hadoop\mapreduCehadoop-2.7.3\share\hadoop\yarnhadoop-2.7.3\share\hadoop\hdfsjavaᡭjar■ᒹn

67⚗m>export->java->JARf1le->⌱¯°஺m±1003Cdhad0op-2.7.3/bin/1006./hdfsdfS-mkd1r-p/usr/1npUt/hot1007./hdfsdfS-putdata/usr/1nput/hot1016./hadoOpjar/opt/hadoop2.7.3.jarcom.bjSxt.mr.RunJOb:50070/explorer.html#/usr/Output/hot./hdfsdfs-cat/USr/Output/hot/part*|head-nl0hiveHiVe4ᦪº¾¿Hive4ÀÁᘤÃÄᘤÅᓄᘤ=஺hive´µnapache-hive-2.1.1-bln.tar.gzcd/opt/apache-hive-2.1.1-bin/Conf/

68Cphive-env.sh.templatehive-env.shcphive-default.Xml.temp1ateh1Ve-site.xmlcphiVe-log4j2.properties.templatehive-1og4j2.propertiescphive-exec-log4j2.propQrties.templatehive-eXec-1Og4j2.properties~ᦋhive-env.shᵫ?HiveÇᵨHadoop,◤⌕ᙠhive-env.shᦻᢣHadoop´µ⌶nJAVA_HOME=/opt/java##Java⌶eXportHADOOP_HOME=/Opt/hadOop##Hadoop´µ⌶HIVE_HOME=/opt/hive##Hive´µ⌶HIVE_CONF_DIR=/Opt/hive/conf##HiVeNᦻ⌶qrHDFSᙠHlveqr⊤ËÌ◤⌕qrÍHDFSjÎÏÐÑIÒḄᩗ▲஺

69hdfsdfs-mkd1r-p/user/hive/warehoUsehdfsdfs-mkdir-p/User/hive/tmphdfsdfs-ITkdir-p/User/h1ve/10ghdfsdfs-chmOd777/user/hive/warehousehdfSdfS-chmod777/User/hiVe/tmphdfsdfs-Chmod777/user/hive/1og~ᦋhive-site.xm-IKhive-site.xmlᦻÍÓ!N⚗ḄÔstᡂm±qrḄÓ!⌶஺h1Ve.exeC.Scratehdir/user/hive/tITlpHDFSr0otscratchdirforHiVejobswh1chgetsCreatedw1thwriteall(733)permiss1on.ForeaChconnectingUser,anHDFSscratchd

70ir:${hiVe.exec.scratchdir}/<username>iscreated,With${hive.seratCh.dir.permisS1on}.(property>hive.metastore.warehouse.dir/user/h1Ve/warehoUSelOcationofdefaultdatabaseforthewarehousehive,querylog.1ocatiOn/user/hive/IOgLocationofHiverunt1mestructuredlogflle

71´µmysq1yuminsta11mariadbmar1adb-serVermariadb-deve1servicemar1adbrestartmysq1yuminStal1mysq1-serverservicemysqldreStartqrᦪº¿FᵨᡝᎷ×Ø´µÙMySQL☢qrm!h1ve0ᦪº¿ᵨᩭHiveᐗᦪºÝᦪº¿ÞßḄᵨᡝeFàṹâPhive஺mysq1>CREATEDATABASEhive;mysql>USEhive;mySql>CREATEUSER'hive'@'1oca1host'IDENTIFIEDBY'hive'GRANTALLONhive.*TO1hive'@'localhost'IDENTIFIEDBY1hive';

72mysql>GRANTALLONhive.*TO*hive,@'%'IDENTIFIEDBY1hive'FLUSHPRIVILEGES;mySql>quit;¨⃩Hiveᙠ¥ãä¨⃩h1ve¥ãRST¦åÍæçnHDFSض·஺èÍÇᵨstart-dfs.Shéᩭ¶·HDFS0MySQLJaVaêëᘤìíî`U$HIVE_HOME/lib஺ᡃ´µRÇᵨḄ4mysql-connector-java-5.1.39.jaroschematOOI-dbTypemysqi-1n1tSChema⌕ÇᵨHiveCLI(HiveCOITlmand1ine1nterface),èÍᙠðñᐭÍ¥ãnhivehiVe>Showtables<

73OKMariaDB[(none)]>usehive;MariaDB[h1ve]>Showtab-Iescreatetablet_emp(idint,nameString,age1nt,dept_namestring)rOwformatdelimitedfieldsterminatedby<°ᐭᦪºnloaddataloca1inpath1/opt/emp.txt*intotablet_emp;h1ve>select*fromt_emp

74eclipseqrhiveìjarᒹC\workspaCe_eclipsQ\apaChe-hive-2.1.1-b1n\1ib¶·hiVeServer2÷ønvimh1ve-site.xmlhive.server2.thrift.b1nd.h0stnodelBindhostonwhichtoruntheHiveServer2Thriftservice.[rOot@nOde1bin]#./hive—servicehiveserver2netstat-an|grep10000./beeline!connectjdbehive2//node1/default

75ErrorCouldnotopenclienttranSportW1thJDBCUri:jdbc:hive2://nodel:-1/defaUItCannotopenwithoUtport.(state=08S01,code=0)hiveᨵùôúᦪnUDF,UDAF1,udf:ᐭᦪºPmᩩᦪºᦪºHPmᩩᦪº2,udaf:ᐭᦪºPQᩩᦪºcoUntüᔠúᦪ,Avg,min,ᦪ᪾þMappermrhadoopÿ᝱
hdfshbaSeMapReduCeMahOutPigHivezookeeperoFlumeSqO0phbase:hbase-1.2.5-bin.tar.gzvimhbase-enV.shexportJAVA_HC)ME=/Usr/java/jdkl.8.0_101

76vimhbase-s1te.xmlvimregiOnserversvimhbase-env.shexportHBASE_MANAGES_ZK=falsecp-a/Opt/hadOOp-2.7.3/etc/hadOop/hdfS-site.xml./opt/hbase-1.2.5/b1nshstart-hbaSe.sh./hbase-daemon.shstartmasterntpdate1.asia.pOol.ntp.orgservicentpdStartᦋcp-f/Usr/share/Zone1nfo/Asia/Shanghai/etc/loCaltime**/2***rootntpdate1.asia.pOo1.ntp.org;hwclock-w:16010/master-statUs./hbaseshell

77hbase(ma1n):002:0>ereatet_persOn','cfl'hbaSe(ma1n)003:0>listhbase(main):004:0>put't_person','007','cfl:name','Zs'hbase(main):005:0>flush*t_persOn,./hbasehfile-p-fanpt1