资源描述:
《基于hadoop的云计算平台搭建业设计翻译毕业论文》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、MapReduce:SimpliedDataProcessingonLargeClustersJeffreyDeanandSanjayGhemawatAbstractMapReduceisaprogrammingmodelandanassociatedimplementationforprocessingandgeneratinglargedatasets.Usersspecifyamapfunctionthatprocessesakey/valuepairtogenerateasetofintermediatekey/valuepai
2、rs,andareducefunctionthatmergesallintermediatevaluesassociatedwiththesameintermediatekey.Manyrealworldtasksareexpressibleinthismodel,asshowninthepaper.Programswritteninthisfunctionalstyleareautomaticallyparallelizedandexecutedonalargeclusterofcommoditymachines.Therun-tim
3、esystemtakescareofthedetailsofpartitioningtheinputdata,schedulingtheprogram’sexecutionacrossasetofmachines,handlingmachinefailures,andmanagingtherequiredinter-machinecommunication.Thisallowsprogrammerswithoutanyexperiencewithparallelanddistributedsystemstoeasilyutilizeth
4、eresourcesofalargedistributedsystem.OurimplementationofMapReducerunsonalargeclusterofcommoditymachinesandishighlyscalable:atypicalMapReducecomputationprocessesmanyterabytesofdataonthousandsofmachines.Programmersfindthesystemeasytouse:hundredsofMapReduceprogramshavebeenimp
5、lementedandupwardsofonethousandMapReducejobsareexecutedonGoogle’sclusterseveryday.1IntroductionOverthepastfiveyears,theauthorsandmanyothersatGooglehaveimplementedhundredsofspecial-purposecomputationsthatprocesslargeamountsofrawdata,suchascrawleddocuments,webrequestlogs,e
6、tc.,tocomputevariouskindsofderiveddata,suchasinvertedindices,variousrepresentationsofthegraphstructureofwebdocuments,summariesofthenumberofpagescrawledperhost,thesetofmostfrequentqueriesinaToappearingivenday,etc.Mostsuchcomputationsareconceptuallystraightforward.However,
7、theinputdataisusuallylargeandthecomputationshavetobedistributedacrosshundredsorthousandsofmachinesinordertofinishinareasonableamountoftime.Theissuesofhowtoparallelizethecomputation,distributethedata,andhandlefailuresconspiretoobscuretheoriginalsimplecomputationwithlargeam
8、ountsofcomplexcodetodealwiththeseissues.Asareactiontothiscomplexity,wedesignedanewabstractionthatallows