dtcc14-spark-runtime-internals(001)

《dtcc14-spark-runtime-internals(001)》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、RuntimeInternals连城lian@databricks.comcheng.lian@ciilabs.orgWhatis•Afastandgeneralengineforlarge-scaledataprocessing•AnopensourceimplementationofResilientDistributedDatasets(RDD)•HasanadvancedDAGexecutionenginethatsupportscyclicdataflowandin-memorycomputingWhy•Fast–Runmachinelearninglikeiterativeprog

2、ramsupto100xfasterthanHadoopMapReduceinmemory,or10xfasterondisk–RunHiveQLcompatiblequeries100xfasterthanHive(withShark/SparkSQL)Why•Easytouse–FluentScala/Java/PythonAPI–Interactiveshell–2-5xlesscode(thanHadoopMapReduce)Why•Easytouse–FluentScala/Java/PythonAPI–Interactiveshell–2-5xlesscode(thanHadoop

3、MapReduce)sc.textFile("hdfs://...").flatMap(_.split("")).map(_->1).reduceByKey(_+_).collectAsMap()Why•Easytouse–FluentScala/Java/PythonAPI–Interactiveshell–2-5xlesscode(thanHadoopMapReduce)sc.textFile("hdfs://...").flatMap(_.split(""))Canyouwritedown.map(_->1)WordCount.reduceByKey(_+_)in30secondswit

4、h.collectAsMap()HadoopMapReduce?Why•Unifiedbigdatapipelinefor:–Batch/Interactive(SparkCorevsMR/Tez)–SQL(Shark/SparkSQLvsHive)–Streaming(SparkStreamingvsStorm)–Machinelearning(MLlibvsMahout)–Graph(GraphXvsGiraph)ABiggerPicture...AnEvenBiggerPicture...•ComponentsoftheSparkstackfocusonbigdataanalysisan

5、darecompatiblewithexistingHadoopstoragesystems•Usersdon’tneedtosufferexpensiveETLcosttousetheSparkstack“OneStackToRuleThemAll”•Well,mostly:-)•Anddon'tforgetShark/SparkSQLvsHiveResilientDistributedDatasetsResilientDistributedDatasets•Conceptually,RDDscanberoughlyviewedaspartitioned,localityawaredistr

6、ibutedvectors•AnRDD…–eitherpointstoadirectdatasource–orappliessometransformationtoitsparentRDD(s)togeneratenewdataelements–ComputationcanberepresentedbylazyevaluatedlineageDAGscomposedbyconnectedRDDsResilientDistributedDatasets•FrequentlyusedRDDscanbematerializedandcachedin-memorytoacceleratecomputa

7、tion•SparkschedulertakesdatalocalityintoaccountTheIn-MemoryMagic•“Infact,onestudy[1]analyzedtheaccesspatternsintheHivewarehousesatFacebookanddiscoveredthatforthevastmajority(96%)ofjobs,theentireinputs