Configuring HDFS High availability using Cloudera Manager
One of the main advantages of Hadoop 2 is its High availability capability through adding a standby NameNode and Resource Manager.
Cloudera Hadoop default installation is not Highly available and this should be configured after the installation. I used CDH5 with Cloudera manager which automates most of the hard work and makes configuring HA very easy. In the next post we will discuss YARN high availability.
We start with HDFS. Go to the clusters menu and select the HDFS service.
At this point you must have the server on which the standby nameNode will run ready and added to the cluster.
Click on “Enable high availability” button above the list.
When using HA, the servers communicate with a new entity called name service instead of communicating directly with the physical NameNode host. The name service points to the active nameNode at all times. In the next screen we will pick a name for the name service (default is nameservice1):
Now we choose which machines will host the two NameNodes and which will host the journalNodes (choose at least 3):
In the “review changes” screen you will have to specify the directory in which every journal node will keep its copy of the edits file.
There are also some extra options at the bottom:
Now cloudera manager will configure the cluster with the new services and definitions, and will restart HDFS. You can watch all the steps and the overall progress:
In the end, you may receive a message about some additional actions that should be done manually to complete the HA configuration (In my case it’s Configuring httpFS and HUE for high availability and update Hive MetaStore to work with the name service instead of the NameNode server. If you run Impala or HBase on your cluster they will have to be configured manually too, but I did not have them running in my cluster), We will look at how to do that next :
Do not worry too much about fencing, Cloudera manager automatically configures the NameNodes with the shell fencing method.
Now we will configure httpFS and HUE:
- Go to the HDFS service.
- Click the Instances
- Click Add Role Instances.
- Click the text box below the HttpFS The Select Hosts dialog displays.
- Select the host on which to run the role and click OK.
- Click Continue.
- Back in Instances, Check the check box next to the HttpFS role and select Actions for Selected > Start.
- Now go to HUE service.
Click Configuration (It is probably shown in Red. that’s ok for now).
- Locate the HDFS Web Interface Role property or search for it by typing its name in the Search box.
- Select the HttpFS role you just created instead of the NameNode role, and save your changes.
- Restart the Hue service.
Configuring HIVE MetaStore for High Availability:
This updates the Hive Metastore to point to a service name instead of a physical server name, so it can point to the active server at all times.
Stop HUE, Impala and Hive services in this order.
- Back up the Hive metastore database.
- Go to Hive service.
- Select Actions> Update Hive Metastore NameNodes and confirm the command.
- Start Hive, Impala and Hue services in this order.
That’s it, HDFS is now highly available.
Now let’s test it:
If we go to HDFS service, you can see that cloudera5 is the active NameNode. From the actions menu we select manual failover. After a while you can see the two NameNodes change their roles.
Another, more aggressive way to check this is to just pull the plug of the active NameNode. If you do that for cloudera5 you can see that now cloudera1 became the active NameNode and cloudera5 is unavailable:
I tried running HDFS commands during the failover. Right after I killed cloudera5 I got exceptions, but few seconds later, when cloudera1 took over, I was able to run HDFS commands again: