Spark-yarn环境部署

    基本环境: ubuntu 12.04 32位

一. 安装yarn伪分布式集群

     1. 创建新用户

         (1) 添加用户: sudo useradd -m hadoop -s /bin/bash
         (2) 修改密码: sudo passwd hadoop
         (3) 添加sudo权限: sudo adduser hadoop sudo
         (4) 注销选择hadoop用户登录

     2. 系统环境配置

         (1) sudo apt-get update
         (2) 安装vim: sudo apt-get install vim
         (3) 安装ssh: sudo apt-get install openssh-server

(4) 设置ssh免登录
cd ~/.ssh
rm ./id_rsa*
ssh-keygen -t rsa  (一路回车即可)
cat ./id_rsa.pub >> ./authorized_keys

     3. 安装java环境

         (1) 安装jre和jdk: sudo apt-get install openjdk-7-jre openjdk-7-jdk
         (2) 设置环境变量JAVA_HOME: dpkg -L openjdk-7-jdk | grep '/bin/javac'
              该命令会输出一个路径，除去路径末尾的 “/bin/javac”
         (3) vim ~/.bashrc
              添加一行export JAVA_HOME=...
         (4) source ~/.bashrc
         (5) check java 版本: java -version

     4. 安装hadoop2

         (1) 从http://mirror.bit.edu.cn/apache/hadoop/common/下载最新的稳定版本的hadoop，例如hadoop-2.7.3/hadoop-2.7.3.tar.gz
         (2) 安装hadoop:
               sudo tar -zxf hadoop-2.7.3.tar.gz -C /usr/local
               cd /usr/local
               sudo mv hadoop-2.7.3 hadoop
               sudo chown -R hadoop hadoop

     5. 配置hadoop

       cd /usr/local/hadoop/etc/hadoop进入hadoop配置目录，如果没有hadoop-env.sh或yarn-env.sh需要从后缀名为hadoop-env.sh.template复制一份
       1). 在hadoop-env.sh中配置JAVA_HOME
       2).在yarn-env.sh中配置JAVA_HOME

3). 修改core-site.xml
<configuration>
        <property>
             <name>hadoop.tmp.dir</name>
             <value>file:/usr/local/hadoop/tmp</value>
             <description>Abase for other temporary directories.</description>
        </property>
        <property>
             <name>fs.defaultFS</name>
             <value>hdfs://localhost:9000</value>
        </property>
</configuration>

4). hdfs-site.xml
<configuration>
        <property>
             <name>dfs.replication</name>
             <value>1</value>
        </property>
        <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
             <name>dfs.datanode.data.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
</configuration>

5). mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

6). yarn-site.xml
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

       6. 启动hadoop

(1) namenode格式化
    ./bin/hdfs namenode -format
(2) 修改~/.bashrc添加以下两行，并执行source ~/.bashrc
    export HADOOP_HOME=/usr/local/hadoop
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
(3) ./libexec/hadoop-config.sh 找到JAVA_HOME，在前面添加
    export JAVA_HOME=...
(4) 开启 NameNode 和 DataNode 守护进程
    ./sbin/start-dfs.sh
(5) jps命令查看是否成功启动
    若成功启动则会列出如下进程: “NameNode”、”DataNode” 和 “SecondaryNameNode”（如果 SecondaryNameNode 没有启动，请运行 sbin/stop-dfs.sh 关闭进程，然后再次尝试启动尝试）。如果没有 NameNode 或 DataNode ，那就是配置不成功，请仔细检查之前步骤，或通过查看启动日志排查原因。
(6) 启动成功后可以通过 http://localhost:50070/查看nameNode和dataNode相关信息
(7) 关闭hadoop
    ./sbin/stop-dfs.sh
    第二次之后启动 hadoop，无需进行 NameNode 的初始化，只需要运行 ./sbin/start-dfs.sh 即可

      7. 启动YARN

 (1) 启动yarn
      ./sbin/start-yarn.sh
      ./sbin/mr-jobhistory-daemon.sh start historyserver  #开启历史服务器，才能在Web中查看任务运行情况
 (2) jps查看
      开启后通过 jps 查看，可以看到多了 NodeManager 和 ResourceManager 两个后台进程
 (3) 启动成功后可以通过页面http://localhost:8088/cluster查看集群任务的运行情况
 (4) 关闭yarn
       ./sbin/stop-yarn.sh
       ./sbin/mr-jobhistory-daemon.sh stop historyserver

二. 安装Spark

1. 下载Spark
   wget "http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz"
2. 解压到/usr/local
   sudo tar -xvzf spark-2.0.0-bin-hadoop2.7.tgz -C /usr/local
   cd /usr/local
   sudo mv spark-2.0.0-bin-hadoop2.7 spark
   sudo chown -R hadoop spark
3. 设置环境变化PATH
   vim ~/.bashrc
   export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin
   source ~/.bashrc

4). 配置Spark
cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
配置内容如下
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SPARK_MASTER_IP=master
SPARK_LOCAL_DIRS=/usr/local/spark
SPARK_DRIVER_MEMORY=1G

5). 启动spark
sh /usr/local/sbin/start-all.sh
启动成功后可以使用
pyspark或spark-submit
同时也可以访问以下链接查看spark任务
http://master:8080

三.  使用Yarn运行Spark程序

     为了方便，以后我们都采用Yarn来运行Spark任务，不会再单独启动Spark

     1.  机器配置

sudo chown hadoop:root -R /usr/local/hadoop
sudo chown hadoop:root -R /usr/local/spark
sudo chmod 775 -R /usr/local/hadoop
sudo chmod 775 -R /usr/local/spark

//bashrc配置
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin
export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native:$LD_LIBRARY_PATH

     2. 启动yarn

cd /usr/local/hadoop
./sbin/start-dfs.sh
./sbin/start-yarn.sh
./sbin/mr-jobhistory-daemon.sh start historyserver

     jps查看进程，应该有以下几个

16891 NodeManager
16951 JobHistoryServer
16502 SecondaryNameNode
16028 NameNode
17729 Jps
16683 ResourceManager
16228 DataNode

     3. 停止yarn

cd /usr/local/hadoop
./sbin/stop-dfs.sh
./sbin/stop-yarn.sh
./sbin/mr-jobhistory-daemon.sh stop historyserver

      4. web界面查看

查看nameNode和dataNode: http://localhost:50070/
查看yarn集群任务: http://localhost:8088/cluster

四.  问题汇总

     1. 问题
        Hadoop 2.x.x - warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard.
        解决方案
        1. vi ~/.bashrc
        2. 添加以下2行
           export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
           export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib