Google云计算技术MapReduce国外课件.ppt

《Google云计算技术MapReduce国外课件.ppt》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、MapReduce:SimplifiedDataProcessingonLargeClustersJeffreyDean&SanjayGhemawatOSDI’04“Thedensityoftransistorsonachipdoublesevery18months,forthesamecost”(1965)TheFreeLunchIsAlmostOver!!TheFutureisMulti-core!!WebgraphicSuperComputerJanetE.Ward,2000ClusterofDesktopsTheFutureisMulti-core

2、!!ReplacespecializedpowerfulSuper-ComputerswithlargeclustersofcommodityhardwareButDistributedprogrammingisinherentlycomplex.Google’sMapReduceParadigmPlatformforreliable,scalableparallelcomputingAbstractsissuesofdistributedandparallelenvironmentfromprogrammer.RunsoverGoogleFileSyst

3、emsWhatisMapReduce?Aprogrammingmodelandanassociatedimplementation(library)forprocessingandgeneratinglargedatasets(onlargeclusters).Anewabstractionallowingustoexpressthesimplecomputationsweweretryingtoperformbuthidesthemessydetailsofparallelization,fault-tolerance,datadistributiona

4、ndloadbalancinginalibrary.ReferencesJeffreyDean,SanjayGhemawat:MapReduce:SimplifiedDataProcessingonLargeClusters.OSDI2004:137-150Also:InterpretingtheData:ParallelAnalysiswithSawzall.RobPike,SeanDorward,RobertGriesemer,SeanQuinlan.GoogleLabs.GoogleFileSystems(GFS)Highlyscalabledist

5、ributedfilesystemforlargedata-intensiveapplications.ProvidesredundantstorageofmassiveamountsofdataoncheapandunreliablecomputersProvidesaplatformoverwhichothersystemslikeMapReduce,BigTableoperate.GFSArchitectureMapReduce:Insight”Considertheproblemofcountingthenumberofoccurrencesofe

6、achwordinalargecollectionofdocuments”Howwouldyoudoitinparallel?OnepossiblesolutionDividecollectionofdocumentamongtheclass.Eachpersongivescountofindividualwordinadocument.Repeatsforassignedquotaofdocuments.(Donew/ocommunication)Sumupthecountsfromallthedocumentstogivefinalanswer.Map

7、ReduceProgrammingModelInspiredfrommapandreduceoperationscommonlyusedinfunctionalprogramminglanguageslikeLisp.Usersimplementinterfaceoftwoprimarymethods:1.Map:(key1,val1)→(key2,val2)2.Reduce:(key2,[val2])→[val3]Manyrealworldtasksareexpressibleinthismodel.Assumption:datahasnocorrela

8、tion,oritissmall.BigpictureMapope