资源描述：
《Massively Parallel Data Analysis with MapReduce.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、MassivelyParallelDataAnalysiswithMapReduceETHZurichComputerScienceDepartmentFall2008Today•HowwouldyoujointwodatasourceswithintheMapReduceframework?–Map-Reduce-Merge[Yangetal(Yahoo!&UCLA),SIGMODConference,June2007]–HadoopJoin–ImprovementsonHadoopJoin[Raoetal(IBMAlmadenRes
2、earchCenter),BayAreaHadoopUserGroupMeeting,October2008]ETHZurich,Fall2008MassivelyParallelDataAnalysiswithMapReduce2Map-Reduce-MergeETHZurich,Fall2008MassivelyParallelDataAnalysiswithMapReduce3BasicDatabaseOperationsinMapReduce•Projection•Selection•Aggregation•Binaryoper
3、ations–Join,Cartesianproduct,Setoperations•OnlytheunaryoperationscanbedirectlymodeledwiththeoriginalMapReduceframework.•Thereisnodirectsupportforoperationsovermultiple,possiblyheterogeneousinputdatasources.–CanbedoneindirectlybychainingextraMapReducesteps.ETHZurich,Fall2
4、008MassivelyParallelDataAnalysiswithMapReduce4SearchEngineExample•SearchEngineskeepdatainmultiple“databases”.–Crawlerdatabase(crawledURLs+contents)–Indexdatabase(invertedindices)–Logdatabases(clickorexecutionlogs)–Webgraphdatabase(URLlinkages+properties)•Sometasksrequire
5、accesstomultipledatasources.–Example:Indexdatabaseiscreatedbasedonthedatainbothcrawlerandwebgraphdatabases.ETHZurich,Fall2008MassivelyParallelDataAnalysiswithMapReduce5Map-Reduce-MergeVision•Map-Reduce-Mergecanformahierarchicalworkflowwhichissimilarto,butmuchmoregeneralt
6、hanaDBMSqueryexecutionplan.–Noqueryoperators,butarbitraryprogramminglogicspecifiedbythedevelopers–Moregeneralthanrelationalqueryplans–MoregeneralthanMap-ReduceETHZurich,Fall2008MassivelyParallelDataAnalysiswithMapReduce6OriginalMapReduceETHZurich,Fall2008MassivelyParalle
7、lDataAnalysiswithMapReduce7Map-Reduce-MergeETHZurich,Fall2008MassivelyParallelDataAnalysiswithMapReduce8Map-Reducevs.Map-Reduce-Mergekeepthekeydifferentdatasetlineages(α=βself-merge)merge~two-dimensionallistcomprehensioninfunctionalprogrammingETHZurich,Fall2008Massively
8、ParallelDataAnalysiswithMapReduce9Example•Tworelationaltables:DepartmentandEmployee•Goal:Computeemploye
