Instructions on MapReduce Assignments.pdf

《Instructions on MapReduce Assignments.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、InstructionsonMapReduceAssignments1.Exampleof‘WordCount’InPythonDescription:useMapReducemethodinPythontocountthenumberofeachwordthatappearsinthetextfile.TextFile:words.json(justanexample)["row1","todayisaniceday"]["row2","howisshetoday"]["row3","sheisverynicet

2、oday"]MapReduceMethodMapfunctionisaprocessforgeneratingoriginal‘word-count’pairsforeachrow.Reducefunctionisaprocessforcomputingthesumofcountsforeachword.Implementation:(Youcanimitatethemapperandreducerbelowtoaccomplishyourassignments.)wordcountMapper.py#!/us

3、r/bin/envpythonimportsysimportjson#inputcomesfromSTDIN(standardinput)forlineinsys.stdin:#removeleadingandtrailingwhitespaceline=line.strip()#parsethelinewithjsonmethodrecord=json.loads(line)key=record[0];value=record[1];#splitthelineintowordswords=value.split()f

4、orwordinwords:#writetheresultstoSTDOUT(standardoutput);print'%st%s'%(word,1)wordcountReducer.py#!/usr/bin/envpythonimportsys#mapswordstotheircountsword2count={}#inputcomesfromSTDINforlineinsys.stdin:#removeleadingandtrailingwhitespaceline=line.strip()#parsethei

5、nputwegotfrommapper.pyword,count=line.split('t',1)#convertcount(currentlyastring)tointtry:count=int(count)word2count[word]=word2count.get(word,0)+countexceptValueError:#countwasnotanumber,sosilently#ignore/discardthislinepass#writetheresultstoSTDOUT(standardout

6、put)forwordinword2count:print'%st%s'%(word,word2count[word])output:a1very1is3how1she2day1today3nice2TipsItdoesn’tmatterifyouhavenoPythonorJavaprogramingexperience.EitherCorC++programingexperienceisfine,consideringthatwewillnotuseadvancedsyntaxinPythonorJava.

7、ThefirstthingyouneedtodoistounderstandtheideaofMapReduce,andtotakeadeeplookatwordcountMapper.pyandwordcountReducer.py.Afterthatyoucansimplyimitatethemethodtoaccomplishtheotherassignments.2.Practicalguidestepbystep(蓝色部分需要修改路径)1.$hadoopfs–ls/(checkthedirectoriesin

8、hdfs)2.$hadoopfs–mkdir/input(makeanew‘input’directory)3.$hadoopfs–put/path/to/words.json/input(puttheinputdatafileintohdfs)4.$hadoopfs–ls/(checkthedirectoriesagain,yo