1、SuperVessel:EnablingSparkasaServicewithOpenStackandDockerGuanCheng(G.C.)ChenIBMResearch-Chinaweibo.com/parallellabsSuperVesselCloud•Public cloud built on the POWER7/POWER8 servers with OpenStack •It provides freeaccess for students, researchers, developers ac
2、ross the world, and helps grow OpenPOWER ecosystem (used in 30+ universiNes now) •It provides advanced technology services such as Spark as a Service, Docker Services, CogniNve CompuNng Service, IoT Service, Accelerator as a Service (FPGA and GPU) 4/18/15IBM
3、Research ‐ China2 SparkasaServiceTry it at: www.ptopenlab.com3stepstolaunchaSparkcluster,easy!Step 1: LoginStep 2: CreateStep 3: Ready!4/18/15IBM Research ‐ China3 WhyOpenStack?•Most popular IaaS soXware •Supports Docker* •Heat can orchestra docker containers
4、 easily – good for provision a Spark cluster Picture source: h-p://www.qyjohn.net/?p=38014/18/15IBM Research ‐ China4 WhyDocker?•Less resource consumpNon than KVM –We can provision more Spark clusters! •Boot faster than KVM –Users like fast provision! •Increm
5、entally build, revert and reuse your container –We love Git and AUFS! •However, Docker is not officially supported in OpenStack yet –Nova Docker is an external component of OpenStack•Port Docker to POWER Architecture (ppc64 and ppc64le)–Ubuntu 15.04 includes Do
6、cker for Picture source: h-p://www.zdnet.com/ar@cle/what‐is‐docker‐and‐why‐is‐it‐so‐darn‐popular/POWER8 ppc64le4/18/15IBM Research ‐ China5 WhySpark?•Fast •Unified •Ecosystem PorNng to POWER –Bugfix submieed to the community –Spark 1.3 works smoothly!4/18/15IB
7、M Research ‐ China6 WhynotSahara?SaharaisacomponentforHadoop/SparkasaServiceinOpenStack•We started from OpenStack Icehouse … •DockerizaNon –Beeer service deployment and isolaNon for Big Data dashboard server •CustomizaNon –WaiNng for Sahara’s improvements is
8、somehow *SLOW* •Docker, user analyNcs, Spark 1.4, Spark IDE, scheduling, data visualizaNon etc. 4/18/15IBM Research ‐ China7 ArchitectureDesignSparkMasterNameNodeBilling&AuthGlanceSparkDr
