在MacOs上配置hadoop和spark環境

Setting up Hadoop with Spark on MacOs

Instructions

1. 準備環境
   如果沒有brew，先google怎樣安裝brew
   先uninstall老版本的Hadoop

   brew cleanup hadoop

   然後更新homebrew formulae

   brew update
   brew upgrade
   brew cleanup

   檢查版本信息

   brew info hadoop
   brew info apache-spark
   brew info sbt
   brew info scala

   如果以上程式沒有安裝，需要使用brew install app 進行安裝。

2. 安裝環境 安裝hadoop

   brew install hadoop

   安裝spark

   brew install apache-spark scala sbt

3. 設置環境變數
   使用vim編輯~/.bash_profile，將以下內容貼到最後

    # set environment variables   
    export JAVA_HOME=$(/usr/libexec/java_home)     
    export HADOOP_HOME=/usr/local/Cellar/hadoop/2.5.1    
    export HADOOP_CONF_DIR=$HADOOP_HOME/libexec/etc/hadoop   
    export SCALA_HOME=/usr/local/Cellar/apache-spark/1.1.0   
   
    # set path variables   
    export PATH=$PATH:$HADOOP_HOME/bin:$SCALA_HOME/bin     
   
    # set alias start & stop scripts   
    alias hstart=$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh   
    alias hstop=$HADOOP_HOME/sbin/stop-dfs.sh;$HADOOP_HOME/sbin/stop-yarn.sh
   

4. Hadoop必須要使ssh生效，設置ssh

   • 配置文件路徑：

     /etc/sshd_config

   • 生成秘鑰：

     sh-3.2# sudo ssh-keygen -t rsa

       Generating public/private rsa key pair.
       Enter file in which to save the key (/var/root/.ssh/id_rsa):  輸入/var/root/.ssh/id_rsa
       Enter passphrase (empty for no passphrase): ［直接回車］
       Enter same passphrase again: ［直接回車］
       Your identification has been saved in /var/root/.ssh/id_rsa.
       Your public key has been saved in /var/root/.ssh/id_rsa.pub.
       key fingerprint is:
       97:e9:5a:5e:91:52:30:63:9e:34:1a:6f:24:64:75:af [email protected]
       The key's randomart image is:
       +--[ RSA 2048]----+
       |       .=.X .    |
       |       . X B .   |
       |        . = . .  |
       |         . + o   |
       |        S = E    |
       |         o . .   |
       |          o .    |
       |         + .     |
       |        . .      |
       +-----------------+
     

   • 修改配置文

     sudo vim /etc/ssh/sshd_config

       Port 22
       #AddressFamily any
       #ListenAddress 0.0.0.0
       #ListenAddress ::
       # The default requires explicit activation of protocol 1
       Protocol 2
       # HostKey for protocol version 1
       #HostKey /etc/ssh/ssh_host_key
       # HostKeys for protocol version 2
       #HostKey /etc/ssh/ssh_host_rsa_key
       #HostKey /etc/ssh/ssh_host_dsa_key
       #HostKey /etc/ssh/ssh_host_ecdsa_key
       HostKey /var/root/.ssh/id_rsa
     
       # Lifetime and size of ephemeral version 1 server key
       KeyRegenerationInterval 1h
       ServerKeyBits 1024
     
       # Logging
       # obsoletes QuietMode and FascistLogging
       SyslogFacility AUTHPRIV
       #LogLevel INFO
     
       # Authentication:
       LoginGraceTime 2m
       PermitRootLogin yes
       StrictModes yes
       #MaxAuthTries 6
       #MaxSessions 10
     
       RSAAuthentication yes
     
       PubkeyAuthentication yes
     

   • 啟動ssh服務

     which sshd //查找sshd的位置。

     Mac 上sshd的位置在 /usr/sbin/sshd

     在終端輸入sudo /usr/sbin/sshd即可啟動sshd服務。

     ssh-keygen -t rsa
     cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

5. 配置Hadoop
   到hadoop的安裝路徑

   cd usr/local/Cellar/hadoop/2.5.1/libexec/

   編輯etc/hadoop/hadoop-env.sh

    # this fixes the "scdynamicstore" warning   
    export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" 
   

   編輯etc/hadoop/core-site.xml

    <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
    </configuration>
   

   編輯etc/hadoop/hdfs-site.xml

    <configuration> 
        <property> 
            <name>dfs.replication</name> 
            <value>1</value> 
        </property> 
    </configuration>
   

   編輯etc/hadoop/mapred-site.xml

    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>
   

   編輯etc/hadoop/yarn-site.xml

    <configuration> 
        <property> 
            <name>yarn.nodemanager.aux-services</name> 
            <value>mapreduce_shuffle</value> 
        </property> 
    </configuration>
   

6. 開始啟用Hadoop
   移動到Hadoop的root directory

   cd /usr/local/Cellar/hadoop/2.5.1

   格式化Hadoop HDFS

   ./bin/hdfs namenode -format

   啟動NameNode和DataNode daemon

   ./sbin/start-dfs.sh

   從網頁中查看

   http://localhost:50070/

   啟動ResourceManager和NodeManager daemon

   ./sbin/start-yarn.sh

   檢查所有的守護線程是不是已經在運行

   jps

   從網頁中查看ResourceManager

   http://localhost:8088/

   創建HDFS目錄

   ./bin/hdfs dfs -mkdir -p /user/{username}

   啟動一個MapReduce的例子

    \#calculate pi  
    ./bin/hadoop jar libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar pi 10 100
   

7. 啟動spark

   到Spark的安裝目錄

   cd /usr/local/Cellar/apache-spark/1.1.0

   啟動Spark的例子

   ./bin/run-example SparkPi

   在網頁中查看Spark任務

   http://localhost:4040/

   也可以使用Spark-submit來提交任務

    # pattern to launch an application in yarn-cluster mode
    ./bin/spark-submit --class <path.to.class> --master yarn-cluster [options] <app.jar> [options]
   
    # run example application (calculate pi)
    ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster libexec/lib/spark-examples-*.jar
   

8. 結束