用FUSE挂载Hadoop的HDFS文件系统
camper00 2012-08-09
用户空间文件系统(Filesystem in Userspace，FUSE)允许一些文件系统整合为一个Unix文件系统在用户空间中执行。通过使用Hadoop的Fuse-DFS分类模块，任意一个Hadoop文件系统(不过一般为HDFS)都可以作为一个标准文件系统进行挂载。我们随后便可以使用Unix的工具(如ls和cat)与这个文件系统交互，还可以通过任意一种编程语言使用POSIX库来访问文件系统。
Fuse-DFS是用C语言实现的，使用libhdfs作为与HDFS的接口。要想了解如何编译和运行Fuse-DFS，可参见src/contrib./fuse-dfs中的Hadoop分布目录。
环境：
CentOS5.5
hadoop-0.20.2
gcc自带
Jdk：1.6.0.21
fuse-2.8.4.tar.gz
Automake
Autoconf
m4
Libtool
apache-ant-1.8.4-bin.tar.bz2
软件包下载：
Ant下载地址http://mirror.bjtu.edu.cn/apache//ant/binaries/
Fuse下载地址sourceforge.net/projects/fuse/files/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz/download
安装：(如果安装过程报错提示某某command no found 请注意PATH，并且用rpm-qa | grep CMD (这个工具)查看是否缺少某些工具，我在安装过程中少automake，果断yum装上，你懂得！！)
1、ant安装
Tar jxvf apache-ant-1.8.4-bin.tar.bz2
Mv apache-ant-1.8.4 /usr/local/ant
2、fuse安装
# tar zxvf fuse-2.8.4.tar.gz
# cd /usr/local/fuse-2.8.4/
# ./configure
# make
# make install
# modprobe fuse
3、环境变量设置(请结合自己的实际情况设置如下项目)
Vi /etc/profile
export OS_ARCH=i386
##i386/amd64
export OS_BIT=32
##32/64
export JAVA_HOME=/usr/java/jdk1.6.0_21
ANT_HOME=/usr/local/ant
export HADOOP_HOME=/home/hadoop/hadoop-append/hadoop-0.20.2
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:${HADOOP_HOME}/c++/Linux-$OS_ARCH-$OS_BIT/lib:/usr/local/lib:/usr/lib:
PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATH:$MAVEN_HOME/bin:$IVY_HOME:$ERLANG_HOME
保存退出
Source !$
编译libhdfs
Cd $HADOOP_HOME
/usr/local/ant/bin/ant compile-c++-libhdfs -Dlibhdfs=1 -Dcompile.c++=1
ln -s c++/Linux-$OS_ARCH-$OS_BIT/lib build/libhdfs
/usr/local/ant/bin/ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
vi /home/hadoop/hadoop-append/hadoop-0.20.2/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh
if [ "$HADOOP_HOME" = "" ]; then
export HADOOP_HOME=/home/hadoop/hadoop-append/hadoop-0.20.2
fi
export PATH=$HADOOP_HOME/build/contrib/fuse_dfs:$PATH
for f in ls $HADOOP_HOME/lib/*.jar $HADOOP_HOME/*.jar ; do
export CLASSPATH=$CLASSPATH:$f
done
if [ "$OS_ARCH" = "" ]; then
export OS_ARCH=i386
fi
if [ "$JAVA_HOME" = "" ]; then
export JAVA_HOME=/usr/java/jdk1.6.0_21
fi
if [ "$LD_LIBRARY_PATH" = "" ]; then
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:/usr/local/share/hdfs/libhdfs/:/usr/local/lib
fi
./fuse_dfs $@
保存退出
挂载hdfs
$chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh
$chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs
$ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh /usr/local/bin
$ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs /usr/local/bin/
$mkdir /tmp/dfs
$cd $HADOOP_HOME/build/contrib/fuse-dfs
$./fuse_dfs_wrapper.sh dfs://192.168.64.99:9000 /tmp/dfs
卸载hdfs
Umount /tmp/dfs
开机自动挂载hdfs文件系统
vi /etc/fstab
fuse_dfs_wrapper.sh dfs:192.168.64.99:9000 /tmp/dfs fuse rw,auto 0 0
