云计算关键技术

《云计算关键技术》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、云计算关键技术郑伟平2011-7-26Page2虚拟化技术内容1虚拟化定义2虚拟化分类3全虚拟化与半虚拟化4虚拟化实现5虚拟化技术比较与选型6虚拟化带来的好处7虚拟化带来的问题8虚拟化适用范围9服务器虚拟化过程MapReduceMapReduce是一个简单易用的并行编程模型，它极大简化了大规模数据处理问题的实现Page3DivideandConquer“Work”w1w2w3r1r2r3“Result”“worker”“worker”“worker”PartitionCombineParallelizati

2、onChallengesHowdoweassignworkunitstoworkers?Whatifwehavemoreworkunitsthanworkers?Whatifworkersneedtosharepartialresults?Howdoweaggregatepartialresults?Howdoweknowalltheworkershavefinished?Whatifworkersdie?Whatisthecommonthemeofalloftheseproblems?CommonTheme?

3、Parallelizationproblemsarisefrom:Communicationbetweenworkers(e.g.,toexchangestate)Accesstosharedresources(e.g.,data)Thus,weneedasynchronizationmechanismManagingMultipleWorkersDifficultbecauseWedon’tknowtheorderinwhichworkersrunWedon’tknowwhenworkersinterrupt

4、eachotherWedon’tknowtheorderinwhichworkersaccessshareddataThus,weneed:Semaphores(lock,unlock)Conditionalvariables(wait,notify,broadcast)BarriersStill,lotsofproblems:Deadlock,livelock,raceconditions...Diningphilosophers,sleepybarbers,cigarettesmokers...Moralo

5、fthestory:becareful!CurrentToolsProgrammingmodelsSharedmemory(pthreads)Messagepassing(MPI)DesignPatternsMaster-slavesProducer-consumerflowsSharedworkqueuesMessagePassingP1P2P3P4P5SharedMemoryP1P2P3P4P5Memorymasterslavesproducerconsumerproducerconsumerworkque

6、ueBut,nowMapreduce!Mapreduce:Parallel/DistributedComputingProgrammingModelInputsplitshuffleoutputTypicalproblemsolvedbyMapReduce读入数据:key/value对的记录格式数据Map:从每个记录里extractsomethingmap(in_key,in_value)->list(out_key,intermediate_value)处理inputkey/valuepair输出中间结果ke

7、y/valuepairsShuffle:混排交换数据把相同key的中间结果汇集到相同节点上Reduce:aggregate,summarize,filter,etc.reduce(out_key,list(intermediate_value))->list(out_value)归并某一个key的所有values，进行计算输出合并的计算结果(usuallyjustone)输出结果MapreduceFrameworkMapreduceFrameworkShuffleImplementationPartitiona

8、ndSortGroupPartitionfunction:hash(key)%reducernumberGroupfunction:sortbykeyExampleuses:distributedgrepdistributedsortweblink-graphreversalterm-vector/hostwebaccesslogstatsinvertedindexconstructi