基于spark平台的gpu调度研究

《基于spark平台的gpu调度研究》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

1、基于Spark平台的GPU调度研究苏鹏飞ICT-HPC目录2Spark简介目前工作动机现有编程模型（MapReduce）是一种非循环数据流抽象模型，不适用于需要重复使用数据集的应用：迭代计算（机器学习）交互式数据挖掘工具（R,Excel,Python）MapMapMapReduceReduceDiskInputDiskOutput特性分布式内存抽象支持数据延迟查询支持比Map和Reduce更多的函数提供交互式的Scalashell和Pythonshell保留MapReduce优点容错数据局部性可扩展性生态圈架构数据存

2、储资源管理计算架构sc=newSparkContextf=sc.textFile(“…”)f.filter(…).count()...programSparkclient(appmaster/driver)SparkworkerHDFS,HBase,…BlockmanagerTaskthreadsRDDgraphSchedulerBlocktrackerShuffletrackerClustermanagerRDD全称:ResilientDistributedDatasets容错的、只读的、分布式数据集

3、允许用户指定数据的存储级别(内存/外设)提供丰富的并行操作ReduceCollectCount…RDD操作类型变换(Transformation)不计算，仅返回新RDD行动(Action)进行计算，返回新值给driver程序RDD一个例子：日志挖掘lines=spark.textFile(“hdfs://...”)errors=lines.filter(_.startsWith(“ERROR”))messages=errors.map(_.split(‘t’)(2))cachedMsgs=messages.cach

4、e()Block1Block2Block3WorkerWorkerWorkerDrivercachedMsgs.filter(_.contains(“foo”)).countcachedMsgs.filter(_.contains(“bar”)).count...tasksresultsCache1Cache2Cache3BaseRDDTransformedRDDCachedRDDParalleloperation对Wikipedia全文搜索耗时<1s(而对on-disk数据的搜索耗时为20s)RDD依赖关系划分窄依

5、赖RDD的每个分区最多被ChildRDD的一个分区使用宽依赖RDD的一个分区被ChildRDD的多个分区使用好处？RDD存储级别StorageLevelMeaningMEMORY_ONLYStoreRDDasdeserializedJavaobjectsintheJVM.IftheRDDdoesnotfitinmemory,somepartitionswillnotbecachedandwillberecomputedontheflyeachtimethey'reneeded.Thisisthedefaultleve

6、l.MEMORY_AND_DISKStoreRDDasdeserializedJavaobjectsintheJVM.IftheRDDdoesnotfitinmemory,storethepartitionsthatdon'tfitondisk,andreadthemfromtherewhenthey'reneeded.MEMORY_ONLY_SERStoreRDDasserializedJavaobjects(onebytearrayperpartition).Thisisgenerallymorespace-ef

7、ficientthandeserializedobjects,especiallywhenusingafastserializer,butmoreCPU-intensivetoread.MEMORY_AND_DISK_SERSimilartoMEMORY_ONLY_SER,butspillpartitionsthatdon'tfitinmemorytodiskinsteadofrecomputingthemontheflyeachtimethey'reneeded.DISK_ONLYStoretheRDDpartit

8、ionsonlyondisk.MEMORY_ONLY_2,MEMORY_AND_DISK_2,etc.Sameasthelevelsabove,butreplicateeachpartitionontwoclusternodes.OFF_HEAP(experimental)StoreRDDinserializedformatinTachyon.