Hadoop High Availability multinode cluster setup

Hadoop version – 2.7.1

Zookeeper version – 3.4.6

Virtual machine	IP address	Host name
Active name node	10.131.10.125	namenode.hadoop.com
Standby namenode	10.131.10.155	slavenode5.hadoop.com
Data node	10.131.10.50	slavenode2.hadoop.com
Data node	10.131.10.156	slavenode3.hadoop.com

Download zookeeper

Download hadoop

Commands-

tar –xvf zookeeper-3.4.6.tar.gz

tar –xvf hadoop-2.7.1.tar.gz

Open the .bashrc file.

Command: sudo gedit ~/.bashrc

export JAVA_HOME=/usr/java

export HADOOP_PREFIX=/usr/local/hadoop

export HADOOP_HOME=/usr/local/hadoop

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop

export HADOOP_COMMON_HOME=/usr/local/hadoop

export HADOOP_HDFS_HOME=/usr/local/hadoop

export HADOOP_MAPRED_HOME=/usr/local/hadoop

export YARN_HOME=/usr/local/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/hadoop/lib/native/

export ZOOKEEPER_INSTALL=/usr/local/zookeeper

export PATH=$JAVA_HOME/bin:$HADOOP_PREFIX/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_INSTALL/bin:.

Enable the SSH in all the node.

Generate the SSH key in all the nodes.
Command: ssh-keygen –t rsa (This Step in all the Nodes)

Generate the ssh key process in all the nodes.

Once ssh key is generated, you will get the public key and private key.
The .ssh key Directory should contain the Permission 700 and all the keys inside the .ssh directory should contain the permissions 600.

command-

sudo chmod 700 .ssh/

Change the directory to .ssh and change the permission of files to 600

command-

cd .ssh/

chmod 600 *

In Active Namenode, copy the id_rsa.pub using cat command.

Command: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Copy the NameNode public key to all the nodes using ssh-copy-id command.

Command: ssh-copy-id –i .ssh/id_rsa.pub hduser@slavenode5.hadoop.com

Copy NameNode public key to data node.

Command: ssh-copy-id –i .ssh/id_rsa.pub hduser@slavenode3.hadoop.com

Copy NameNode public key to data node.

Command: ssh-copy-id –i .ssh/id_rsa.pub hduser@slavenode2.hadoop.com

Restart the sshd service in all the nodes.

Command: sudo service sshd restart (Do in all the nodes)

Open the core-site.xml file from the Active Name node and add the below properties.

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://ha-cluster</value>

</property>

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/usr/local/data/jn</value>

</property>

</configuration>

Open hdfs-site.xml file in Active Namenode. Add the below Properties.

<configuration>

<property>

<name>dfs.namenode.name.dir</name>

<value>/usr/local/data/namenode</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<property>

<name>dfs.nameservices</name>

<value>ha-cluster</value>

</property>

<property>

<name>dfs.ha.namenodes.ha-cluster</name>

<value>namenode,slavenode5</value>

</property>

<property>

<name>dfs.namenode.rpc-address.ha-cluster.namenode</name>

<value>namenode:8020</value>

</property>

<property>

<name>dfs.namenode.rpc-address.ha-cluster.slavenode5</name>

<value>slavenode5:8020</value>

</property>

<property>

<name>dfs.namenode.http-address.ha-cluster.namenode</name>

<value>namenode:50070</value>

</property>

<property>

<name>dfs.namenode.http-address.ha-cluster.slavenode5</name>

<value>slavenode5:50070</value>

</property>

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://slavenode5:8485;slavenode3:8485;slavenode2:8485/ha-cluster</value>

</property>

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://slavenode5:8485;slavenode3:8485;slavenode2:8485/ha-cluster</value>

</property>

<property>

<name>dfs.client.failover.proxy.provider.ha-cluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

<property>

<name>ha.zookeeper.quorum</name>

<value>namenode:2181,slavenode3:2181,slavenode5:2181</value>

</property>

<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hduser/.ssh/id_rsa</value>

</property>

<property>

<name>dfs.client.failover.proxy.provider.ha-cluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

<property>

<name>ha.zookeeper.quorum</name>

<value>namenode:2181,slavenode3:2181,slavenode5:2181</value>

</property>

<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hduser/.ssh/id_rsa</value>

</property>

</configuration>

Change the directory to zookeeper’s conf directory.

Command: cd zookeeper/conf



In a conf directory you have zoo_sample.cfg file, create the zoo.cfg using zoo_sample.cfg file.
Command: cp zoo_sample.cfg zoo.cfg
Create the directory in any location and use this directory to store the zookeeper data.
Command: mkdir <path, where you want to store the zookeeper files>

Open the zoo.cfg file.

Command: gedit zoo.cfg
Add the directory path that is created in above step to the dataDir property and add the below details regarding remaining node, in the zoo.cfg file.

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/usr/local/data/zookeeper

clientPort=2181

# HA Hadoop multinode cluster configs

server.1=slavenode5:2888:3888

server.2=namenode:2888:3888

server.3=slavenode3:2888:3888

#server.4=slavenode2:2888:3888

Now copy the Java and Hadoop, zookeeper directories, and .bashrc file to all the nodes (Standby name node, Data node) using scp command.

Command: scp –r <path of directory> hduser@<ip address>:<path where you need to copy>

Similarly, copy the .bashrc file and zookeeper directory to all the nodes and change the environment variables in each according to the respective node.

In a data node create any directory where you need to store the HDFS blocks.

In a data node you have to add the dfs.datanode.data.dir properties.

In my case I created datanode directory to store the blocks.



$ mkdir datanode
Change the permission to data node directory.
$ chmod 755 datanode

Open the HDFS-site.xml file, add this Datanode directory path in dfs.datanode.data.dir property.
Note: Keep all the properties that are copied from the Active namenode; add dfs.datanode.data.dir  one extract property in namenode.

<property>

<name> dfs.datanode.data.dir</name>

<value>/usr/local/data/datanode</value>

</property>

In Active namenode, change the directory where you want to store the zookeeper configuration file (dataDir property path).

Create the myid file inside the directory and add numeric 1 to the file and save the file.
Command: vi myid

In a standby namenode change the directory where you want to store the zookeeper configuration file (dataDir property path).

Create the myid file inside the directory and add numeric 2 to the file and save the file.
In a data node, change the directory where you want to store the zookeeper configuration file (dataDir property path).
Create the myid file inside the directory and add numeric 3 to the file and save the file.
Start the Journalnode in all the three nodes.
Command: hadoop-daemon.sh start journalnode

When you enter jps command, you will see the JournalNode daemon in all the nodes.

Format the Active namenode 
Command: hdfs namenode -format

Start the Namenode daemon in Active namenode.

Command: hadoop-daemon.sh start namenode

Copy the HDFS Meta data from active name node to standby namenode.

Command: HDFS namenode -bootstrapStandby

Once you run this command, you will get the information from which node and location the meta data is copying and whether it is copying successfully or not.

Start the namenode daemon in Standby namenode machine.

Command: hadoop-daemon.sh start namenode
Now start the Zookeeper service in all the three nodes.
Command: zkServer.sh start (Run this command in all the nodes)
In Active Namenode:

In Standby Namenode:

In Data node:

After running the Zookeeper server, enter JPS command. In all the nodes you will see the QuorumPeerMain service.

You can check the status of zookeeper in all the nodes run in the command below. It will give the follower or Leader message.
[hduser@slavenode5 bin]$ zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader

[hduser@slavenode3 bin]$ zkServer.sh status

JMX enabled by default

Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg

Mode: follower

Start the Data node daemon in Data node machine.

Command: hadoop-daemon.sh start datanode
Start the Zookeeper fail over controller in Active name node and standby name node.
Format the zookeeper fail over controller in Active namenode.
Command: hdfs zkfc –formatZK
Start the ZKFC in Active namenode.
Command: hadoop-daemon.sh start zkfc
Enter jps command to check the DFSZkFailoverController daemons.

Format the zookeeper fail over controller in Standby namenode.

Command: hdfs zkfc –formatZK
Start the ZKFC in Standby namenode.
Command: hadoop-daemon.sh start zkfc
Enter jps command to check the DFSZkFailoverController daemons.
Now check the status of each Namenode, which node is Active or which node is on Standby by using the below command.
Command: hdfs haadmin –getServiceState namenode

Now Check the status of each Namenode using the web browser.

Open the Web browser and enter the below URL.
<IP Address of Active Namenode>:50070

Open another name node details using the web browser.

In the Active namenode, kill the namenode daemon to change the Standby name node to active namenode.

Enter jps in Active namenode and kill the daemon.
Command: sudo kill -9 <namenode process ID>

The Name node process ID is 7606, kill the namenode.

Command : Sudo kill -9 7606

Open the two nodes through web browser and check the status.