欢迎来到天天文库
浏览记录
ID:40553704
大小:3.61 MB
页数:33页
时间:2019-08-04
《055 Spark Shuffle Introduction》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、SPARKSHUFFLEINTRODUCTION天火@蘑菇街AboutMeSpark/Hadoop/Hbase/PhoenixcontributorForsparkmainlycontributesin:•Yarn•Shuffle•BlockManager•Scala2.10update•StandaloneHA•Variousotherfixes.tianhuo@mogujie.comWeibo@冷冻蚂蚁blog.csdn.net/colorantWhySparkisfast(er)•Whomdowecompareto?•Whatdowemeanbyfast?•fasttowrite
2、•fasttorunWhySparkisfast(er)cont.•Butthefigureinpreviouspageissomehowmisleading.•Thekeyistheflexibleprogrammingmode.•Whichleadtomorereasonabledataflow.•WhichleadtolessIOoperation.•EspeciallyforiterativeheavyworkloadslikeML.•Whichpotentiallycutoffalotofshuffleoperationsneeded.•But,youwon’talwaysb
3、elucky.•Manyapplogicdidneedtoexchangealotofdata.•Intheend,youwillstillneedtodealwithshuffle•Andwhichusuallyimpactperformancealot.Whatisshuffle7ShuffleoverviewAggregatorAggregatorAggregatorAggregatorAggregatorAggregatorHowdoesshufflecomeintothepicture•Sparkrunjobstagebystage.•StagesarebuildupbyDA
4、GScheduleraccordingtoRDD’sShuffleDependency•e.g.ShuffleRDD/CoGroupedRDDwillhaveaShuffleDependency•ManyoperatorwillcreateShuffleRDD/CoGroupedRDDunderthehook.•Repartition/CombineByKey/GroupBy/ReduceByKey/cogroup•manyotheroperatorwillfurthercallintotheaboveoperators•e.g.variousjoinoperatorwillcallc
5、ogroup.•EachShuffleDependencymapstoonestageinSparkJobandthenwillleadtoashuffle.SoeveryoneshouldhaveseenthisbeforeA: B: G: Stage 1 groupByC: D: F: map E: join Stage 2 union Stage 3 whyshuffleisexpensive•Whendoingshuffle,datanolongerstayinmemoryonly•Forspark,shuffleprocessmightinvolve•datapartitio
6、n:whichmightinvolveveryexpensivedatasortingworksetc.•dataser/deser:toenabledatabeentransferthroughnetworkoracrossprocesses.•datacompression:toreduceIObandwidthetc.•DISKIO:probablymultipletimesononesingledatablock•E.g.ShuffleSpill,MergecombineHistory•Spark0.6-0.7,samecodepathwithRDD’spersistentme
7、thod,canchooseMEMORY_ONLYandDISK_ONLY(default).•Spark0.8-0.9:•separateshufflecodepathfromBMandcreateShuffleBlockManagerandBlockObjectWriteronlyforshuffle,nowshuffledatacanonlybewrittentodisk.•Shuffleoptimization:Consolidates
此文档下载收益归作者所有