Cygwin中运行hadoop mapreduce任务必须设置中间暂存目录

一个人的世界 2012-04-20

cygwin上运行hadoop mr任务,一直报错:

2012-04-20 00:47:28,937 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.FileNotFoundException: File C:/home/Administrator/hadoop-0.20.2/hadooptmp/mapred/local/taskTracker/jobcache/job_201204200043_0002/attempt_201204200043_0002_m_000001_0/work/tmp does not exist.
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
	at org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
	at org.apache.hadoop.mapred.Child.main(Child.java:155)
2012-04-20 00:47:28,937 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the task
2012-04-20 00:47:28,937 INFO org.apache.hadoop.mapred.TaskTracker: Error cleaning upjava.lang.NullPointerException

最终解决:

在mapred-site.xml中添加:

<property>
  <name>mapred.child.tmp</name>
  <value>/home/Administrator/hadoop-0.20.2/child/temp</value>
  <description> To set the value of tmp directory for map and reduce tasks.
  If the value is an absolute path, it is directly assigned. Otherwise, it is
  prepended with task's working directory. The java tasks are executed with
  option -Djava.io.tmpdir='the absolute path of the tmp dir'. Pipes and
  streaming are set with environment variable,
   TMPDIR='the absolute path of the tmp dir'
  </description>
</property>
 

相关推荐