Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.
Storage & Clustering Community Blog

Symantec™ High Availability Solution for Hadoop Namenode

Created: 15 May 2012 • Updated: 11 Jun 2014 • 2 comments
Sharad Srivastava's picture
+4 4 Votes
Login to vote

This blog describes the procedure to configure Hadoop Namenode for high availability under Veritas Cluster Server environment using the Agent Builder tool.

Installing and Configuring Hadoop Namenode for High Availability
Below are the tasks that needs to be performed for installing the Hadoop Namenode for clustering purposes:

  • Allocating shared disk resources : Symantec recommends installing Namenode metadata on a separate and dedicated shared disk resource.
  • Creating disk group, volume, and file system : Create a disk group, volume, and file system on a shared disk resource that is allocated for Namenode metadata.
  • Obtaining dedicated virtual IP address and DNS name : Obtain unique virtual addresses and DNS names for Namenode instance. This address and name is required to support the Namenode IP network configuration. Namenode uses this virtual IP address and DNS name irrespective of the cluster system hosting the node.
  • Obtaining dedicated user accounts : If the Namenode instance does not run using a root account, you must assign a dedicated User (eg. hdfs) account for the instance. Additionally you must ensure that the Hadoop installation directory and log permissions should be owned by this dedicated user.
  • Creating service group and supporting resources : Perform the following tasks to create service group and resources.
    • Create a service group that can contain all required resources to support the Namenode instance.
    • Create appropriate cluster resources and create the dependencies to the shared disk resource and network resources that were previously under the cluster control.
    • Bring the service group online to test the configuration.
  • Modify Hadoop Namenode files for clustering purpose : Ensure that the shared disk and network resources are online. You can now install Hadoop on each local node or on the shared disk as per the desired configuration. After installation modify the $HADOOP_HOME/conf/hdfs-site.xml and set the below properties on each Namenode instance:
dfs.name.dir             Path on the Sharad disk filesystem where the 
                         NameNode stores the name table

dfs.http.address         <Virtual IP/Virtual Address>:<Port>

dfs.https.address        <Virtual IP/Virtual Address>:<Port>
      Modify the $HADOOP_HOME/conf/core-site.xml and set the below property on each Namenode instance:
fs.default.name          hdfs://<Virtual IP/Virtual Address>:<Port>
      For more information on Configuring Hadoop for Clustering purpose, please visit Hadoop Wiki Page.

Configuring the agent for Hadoop Namenode with Agent Builder

  • Prerequisites for configuring agent for Hadoop Namenode : Ensure that you meet the following prerequisites before you configure the agent for Hadoop Namenode.
    • Hadoop Namenode must be installed and configured for high availability.
    • Install and configure Veritas Cluster Server (VCS). For more information about installing and configuring VCS, refer to Veritas Cluster Server Installation Guide.
    • ACC Library must be installed on all the cluster nodes. For more information about Downloading and Installing ACC Library please refer https://sort.symantec.com/agents
    • Agent Builder must be installed on one of the cluster node. For more information about Downloading and Installing Agent Builder please refer https://sort.symantec.com/agents

      Now, perform the following steps on any one of the node in the cluster to configure the agent for Hadoop Namenode with Agent Builder

# cd /opt/VRTSagents/ha/bin/AgentBuilder

# ./agentbuilder Hadoop –base vcs50 –platform linux –ssh –system  vcslinux1 –system vcslinux2

Hadoop Namenode agent attributes

Following are the list of VCS Agent attributes that are mandatory to be filled while configuring the Hadoop Namenode resource.

Table 1-1 lists all the required agent attributes.

Attribute                     Description
User                          OS user running the Hadoop Namenode
StartProgram                  Command to start Hadoop Namenode
StopProgram                   Command to stop Hadoop Namenode
CleanProgram                  Command to stop Hadoop Namenode
EnvFile                       EnvFile to source before running Start/Stop/Clean
                              commands
MonitorProcessPatterns        List of process patterns to monitor for the
                              Hadoop Namenode instance
ListenAddressPort             Listen Address and Port for the Hadoop Namenode.
MonitorProgram                The complete path name and command-line arguments
                              for an externally provided monitor program
Configuring the MonitorProcessPatterns attribute

You can configure the MonitorProcessPatterns attribute of the Hadoop Namenode resource by identifying Hadoop Namenode service processes.

# ps -aef | grep "java.*-Dproc​_namenode.*NameNode"

For example:

# ps -aef | grep  "java.*-Dproc_namenode.*NameNode"

hdfs      6359     1  1 13:54 ?        00:00:06 /opt/jdk1.6.0_21/bin/java -Dproc_namenode -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Dhadoop.log.dir=/opt/hadoop/libexec/../logs -Dhadoop.log.file=hadoop-hdfs-namenode-vcslx202.log -Dhadoop.home.dir=/opt/hadoop/libexec/.. -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,DRFA -Dhadoop.security.logger=INFO,DRFAS -Djava.library.path=/opt/hadoop/libexec/../lib/native/Linux-i386-32 -Dhadoop.policy.file=hadoop-policy.xml -classpath /opt/hadoop/libexec/../conf:/opt/jdk1.6.0_21/lib/tools.jar:/opt/hadoop/libexec/..:/opt/hadoop/libexec/../hadoop-core-1.0.2.jar:/opt/hadoop/libexec/../lib/asm-3.2.jar:/opt/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/opt/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/opt/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/libexec/../lib/commons-cli-1.2.jar:/opt/hadoop/libexec/../lib/commons-codec-1.4.jar:/opt/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/opt/hadoop/libexec/../lib/commons-configuration-1.6.jar:/opt/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/libexec/../lib/commons-digester-1.8.jar:/opt/hadoop/libexec/../lib/commons-el-1.0.jar:/opt/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/libexec/../lib/commons-lang-2.4.jar:/opt/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/opt/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/libexec/../lib/commons-math-2.1.jar:/opt/hadoop/libexec/../lib/commons-net-1.4.1.jar:/opt/hadoop/libexec/../lib/core-3.1.1.jar:/opt/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.2.jar:/opt/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.2.jar:/opt/hadoop/libexec/../lib/hadoop-thriftfs-1.0.2.jar:/opt/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/libexec/../lib/jdeb-0.8.jar:/opt/hadoop/libexec/../lib/jersey-core-1.8.jar:/opt/hadoop/libexec/../lib/jersey-json-1.8.jar:/opt/hadoop/libexec/../lib/jersey-server-1.8.jar:/opt/hadoop/libexec/../lib/jets3t-0.6.1.jar:/opt/hadoop/libexec/../lib/jetty-6.1.26.jar:/opt/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/opt/hadoop/libexec/../lib/jsch-0.1.42.jar:/opt/hadoop/libexec/../lib/junit-4.5.jar:/opt/hadoop/libexec/../lib/kfs-0.2.2.jar:/opt/hadoop/libexec/../lib/log4j-1.2.15.jar:/opt/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/opt/hadoop/libexec/../lib/oro-2.0.8.jar:/opt/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/opt/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/opt/hadoop/libexec/../lib/xmlenc-0.52.jar:/opt/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar org.apache.hadoop.hdfs.server.namenode.NameNode

The processes matching this pattern(regex) can be used to configure the MonitorProcessPatterns attribute of the Hadoop Namenode VCS resource.

Removing the agent for Hadoop Namenode : Perform the following steps to remove the agent for Hadoop Namenode created using Agent Builder.

To remove the agent for Hadoop Namenode

  1. Log in as superuser.
  2. Remove all the resources of the resource type to be deleted.
  3. Delete the resource type that was created using Agent Builder.
  4. Remove the agent directory from all nodes in which the agent is deployed.
# rm -rf /opt/VRTSagents/ha/bin/Hadoop

Sample Hadoop Namenode resource attributes : In the following example configuration, Hadoop Namenode is installed in /opt/hadoop directory.

Table 1-2 lists the Hadoop Namenode resource attributes.

 

Attribute                   Value
ResLogLevel                 INFO
User                        hdfs
StartProgram                /opt/hadoop/bin/hadoop-daemon.sh  \
                            --config $HADOOP_CONF_DIR start namenode
StopProgram                 /opt/hadoop/bin/hadoop-daemon.sh  \
                            --config $HADOOP_CONF_DIR stop namenode
CleanProgram                /opt/hadoop/bin/hadoop-daemon.sh  \
                            --config $HADOOP_CONF_DIR stop namenode
EnvFile                     /opt/hadoop/libexec/hadoop-config.sh
MonitorProcessPatterns      java.*-Dproc_namenode.*NameNode  
ListenAddressPort           namenode.vxindia.veritas.com:8020
MonitorSequence             MonitorProcessPatterns ListenAddressPort \
                            PidFilesPatterns MonitorProgram

Sample Hadoop resource configuration : A sample excerpt from the main.cf file is shown below.

group Hadoop_SG (

   SystemList = { vcslx202 = 0, vcslx203 = 1 }
   )

   DiskGroup Hadoop_dg (
       Critical = 0
       DiskGroup = hadoopdg
       )

   Hadoop NameNode (
       Critical = 1
       ResLogLevel = TRACE
       StopProgram = "/opt/hadoop/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop namenode"
       CleanProgram = "/opt/hadoop/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop namenode"
       User = hdfs
       EnvFile = "/opt/hadoop/libexec/hadoop-config.sh"
       ListenAddressPort = "10.209.73.93:8020"
       StartProgram = "/opt/hadoop/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode"
       MonitorProcessPatterns = { "java.*-Dproc_namenode.*NameNode" }
       MonitorSequence = "MonitorProcessPatterns ListenAddressPort PidFilesPatterns MonitorProgram"
       )

   IP Hadoop_Ip (
       Critical = 0
       Device = eth0
       Address = "10.209.73.93"
       NetMask = "255.255.252.0"
       )

   Mount Hadoop_mnt (
       Critical = 1
       MountPoint = "/hadoopdata"
       BlockDevice = "/dev/vx/dsk/hadoopdg/hadoopvol"
       FSType = vxfs
       FsckOpt = "-y"
       )

   Hadoop_mnt requires Hadoop_dg
   NameNode requires Hadoop_mnt
   NameNode requires Hadoop_Ip
            
   // resource dependency tree
   //  
   //  group Hadoop
   //  {
   //  Hadoop NameNode
   //      {
   //      Mount Hadoop_mnt
   //          {
   //          DiskGroup Hadoop_dg
   //          }
   //      IP Hadoop_Ip
   //      }
   //  }

Sample Service Group configuration

Figure 1-1 shows a sample service group configuration for Hadoop Namenode instance.

 

Figure 1-2 shows a sample hadoop namenode resource properties

 

Comments 2 CommentsJump to latest comment

Harsh J's picture

This isn't exactly HA. Your HDFS clients will not survive the switch (especially continually open clients such as HBase, those will break right up as your switch happens).

For a proper, automatic and hot (client-lease-persisting) Apache HDFS HA solution present in the new Apache HDFS 2, read up at http://hadoop.apache.org/common/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailability.html or a version of deployment-specific info at https://ccp.cloudera.com/display/CDH4DOC/CDH4+High+Availability+Guide - Should save you lots of operational maintenance nightmares :)

0
Login to vote
bruceparker's picture

Is client-persistence supported in 1.0?  If not, is it on the roadmap?

 

0
Login to vote