Spark实验报告材料.doc

《Spark实验报告材料.doc》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、Spark报告金航1510122526Spark实验报告一、环境搭建1、下载scala2.11.4版本下载地址为：.scala-lang.org/download/2.11.4.html2、解压和安装：解压：tar-xvfscala-2.11.4.tgz 安装：mvscala-2.11.4~/opt/3、编辑~/.bash_profile文件增加SCALA_HOME环境变量配置，exportJAVA_HOME=/home/spark/opt/java/jdk1.6.0_37exportCLASSPATH=.:$JAVA_HOME/jre/li

2、b:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jarexportSCALA_HOME=/home/spark/opt/scala-2.11.4exportHADOOP_HOME=/home/spark/opt/hadoop-2.6.0PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:${SCALA_HOME}/bin立即生效source~/.bash_profile4、验证scala：scala-version5、copy到slave机器scp~/.bash_profilespark10..

3、45.56:~/.bash_profile6、下载spark，wgetd3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-hadoop2.4.tgz7、在master主机配置spark：将下载的spark-1.2.0-bin-hadoop2.4.tgz解压到~/opt/即~/opt/spark-1.2.0-bin-hadoop2.4，配置环境变量SPARK_HOME#set javaenvexportJAVA_HOME=/home/spark/opt/java/jdk1.6.0_37exportCLAS

4、SPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jarexportSCALA_HOME=/home/spark/opt/scala-2.11.4exportHADOOP_HOME=/home/spark/opt/hadoop-2.6.0exportSPARK_HOME=/home/spark/opt/spark-1.2.0-bin-hadoop2.4PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:${SCALA_HOME}/bin:${SPA

5、RK_HOME}/bin:${HADOOP_HOME}/bin配置完成后使用source命令使配置生效进入sparkconf目录：[sparkS1PA11opt]$cdspark-1.2.0-bin-hadoop2.4/[sparkS1PA11spark-1.2.0-bin-hadoop2.4]$lsbin conf data ec2 examples lib LICENSE logs NOTICE python README.md RELEASE sbin work[sparkS1PA11spark-1.2.0-bin-hadoop2.4]

6、$cdconf/[sparkS1PA11conf]$lsfairscheduler.xml.template metrics.properties.template slaves.template       spark-env.shlog4j.properties.template slaves           spark-defaults.conf.template spark-env.sh.templatefirst：修改slaves文件，增加两个slave节点S1PA11、S1PA222[sparkS1PA11conf]$visl

7、avesS1PA11S1PA222second：配置spark-env.sh首先把spark-env.sh.templatecopy spark-env.shvi spark-env.sh文件在最下面增加：exportJAVA_HOME=/home/spark/opt/java/jdk1.6.0_37exportSCALA_HOME=/home/spark/opt/scala-2.11.4exportSPARK_MASTER_IP=10.58.44.47exportSPARK_WORKER_MEMORY=2gexportHADOOP_CONF

8、_DIR=/home/spark/opt/hadoop-2.6.0/etc/hadoopHADOOP_CONF_DIR是Hadoop配置文件目录，SPARK_MAS