资源描述：
《3)MapReduce Simplied Data Processing on Large Clusters1.PDF》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、MapReduce:SimpliedDataProcessingonLargeClustersJeffreyDeanandSanjayGhemawatjeff@google.com,sanjay@google.comGoogle,Inc.Abstractgivenday,etc.Mostsuchcomputationsareconceptu-allystraightforward.However,theinputdataisusuallyMapReduceisaprogrammingmodelandanassoci-largeandthec
2、omputationshavetobedistributedacrossatedimplementationforprocessingandgeneratinglargehundredsorthousandsofmachinesinordertonishindatasets.Usersspecifyamapfunctionthatprocessesaareasonableamountoftime.Theissuesofhowtopar-key/valuepairtogenerateasetofintermediatekey/valueall
3、elizethecomputation,distributethedata,andhandlepairs,andareducefunctionthatmergesallintermediatefailuresconspiretoobscuretheoriginalsimplecompu-valuesassociatedwiththesameintermediatekey.Manytationwithlargeamountsofcomplexcodetodealwithrealworldtasksareexpressibleinthismode
4、l,asshowntheseissues.inthepaper.Asareactiontothiscomplexity,wedesignedanewProgramswritteninthisfunctionalstyleareautomati-abstractionthatallowsustoexpressthesimplecomputa-callyparallelizedandexecutedonalargeclusterofcom-tionsweweretryingtoperformbuthidesthemessyde-moditymac
5、hines.Therun-timesystemtakescareofthetailsofparallelization,fault-tolerance,datadistributiondetailsofpartitioningtheinputdata,schedulingthepro-andloadbalancinginalibrary.Ourabstractionisin-gram'sexecutionacrossasetofmachines,handlingma-spiredbythemapandreduceprimitivesprese
6、ntinLispchinefailures,andmanagingtherequiredinter-machineandmanyotherfunctionallanguages.Werealizedthatcommunication.Thisallowsprogrammerswithoutanymostofourcomputationsinvolvedapplyingamapop-experiencewithparallelanddistributedsystemstoeas-erationtoeachlogical“record”inour
7、inputinordertoilyutilizetheresourcesofalargedistributedsystem.computeasetofintermediatekey/valuepairs,andthenOurimplementationofMapReducerunsonalargeapplyingareduceoperationtoallthevaluesthatsharedclusterofcommoditymachinesandishighlyscalable:thesamekey,inordertocombinethed
8、eriveddataap-atypicalMapReducecomputationprocessesmanyter-propriately.Ouruseofafun
