Hadoop

Hadoop jobtracker is dead and pid file exists              [FAILED]

Problem Description:

The Hadoop JobTracker service won’t start even though the ‘start’ option shows ‘OK’.

                

$ sudo service hadoop-0.20-mapreduce-jobtracker status
Hadoop jobtracker is dead and pid file exists              [FAILED]

Solution:

Try to start the service

$ sudo service hadoop-0.20-mapreduce-jobtracker start

Starting Hadoop jobtracker:                                [  OK  ]

starting jobtracker, logging to /var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-elephant.out

Note that even though it says [OK], the service hasn’t started. From above check the logfile for errors

$ view /var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-elephant.log . . .         at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:297)         at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4538) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=mapred, access=WRITE, inode="/var/lib/hadoop-hdfs/cache/mapred/mapred":hdfs:supergroup:drwxr-xr-x         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)         at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204) . . .

In my case I had the ownership of the directory wrong.

$ sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred

Once the permission was corrected, the JobTracker service could be restarted.

$ sudo service hadoop-0.20-mapreduce-jobtracker start Starting Hadoop jobtracker:                                [  OK  ] starting jobtracker, logging to /var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-elephant.out $ sudo service hadoop-0.20-mapreduce-jobtracker status Hadoop jobtracker is running                               [  OK  ] $ sudo jps 2202 TaskTracker 2385 NameNode 3450 Jps 3375 JobTracker 2293 DataNode 2496 SecondaryNameNode