用FUSE挂载Hadoop的HDFS文件系统

camper00 2012-08-09

用户空间文件系统(Filesystem in Userspace,FUSE)允许一些文件系统整合为一个Unix文件系统在用户空间中执行。通过使用Hadoop的Fuse-DFS分类模块,任意一个Hadoop文件系统(不过一般为HDFS)都可以作为一个标准文件系统进行挂载。我们随后便可以使用Unix的工具(如ls和cat)与这个文件系统交互,还可以通过任意一种编程语言使用POSIX库来访问文件系统。

Fuse-DFS是用C语言实现的,使用libhdfs作为与HDFS的接口。要想了解如何编译和运行Fuse-DFS,可参见src/contrib./fuse-dfs中的Hadoop分布目录。

环境:

CentOS5.5

hadoop-0.20.2

gcc自带

Jdk:1.6.0.21

fuse-2.8.4.tar.gz

Automake

Autoconf

m4

Libtool

apache-ant-1.8.4-bin.tar.bz2

软件包下载:

Ant下载地址http://mirror.bjtu.edu.cn/apache//ant/binaries/

Fuse下载地址sourceforge.net/projects/fuse/files/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz/download

安装:(如果安装过程报错提示某某command no found 请注意PATH,并且用rpm-qa | grep CMD (这个工具)查看是否缺少某些工具,我在安装过程中少automake,果断yum装上,你懂得!!)

1、ant安装

Tar jxvf  apache-ant-1.8.4-bin.tar.bz2

Mv apache-ant-1.8.4 /usr/local/ant

2、fuse安装

# tar zxvf fuse-2.8.4.tar.gz
# cd /usr/local/fuse-2.8.4/
# ./configure
# make
# make install
# modprobe fuse

3、环境变量设置(请结合自己的实际情况设置如下项目)

Vi /etc/profile

export OS_ARCH=i386

##i386/amd64

export OS_BIT=32

##32/64

export JAVA_HOME=/usr/java/jdk1.6.0_21

ANT_HOME=/usr/local/ant

export HADOOP_HOME=/home/hadoop/hadoop-append/hadoop-0.20.2

export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:${HADOOP_HOME}/c++/Linux-$OS_ARCH-$OS_BIT/lib:/usr/local/lib:/usr/lib:

PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATH:$MAVEN_HOME/bin:$IVY_HOME:$ERLANG_HOME

保存退出

Source !$

编译libhdfs

Cd $HADOOP_HOME

/usr/local/ant/bin/ant compile-c++-libhdfs -Dlibhdfs=1 -Dcompile.c++=1

ln -s c++/Linux-$OS_ARCH-$OS_BIT/lib build/libhdfs

/usr/local/ant/bin/ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1

vi /home/hadoop/hadoop-append/hadoop-0.20.2/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh

if [ "$HADOOP_HOME" = "" ]; then
export HADOOP_HOME=/home/hadoop/hadoop-append/hadoop-0.20.2
fi

export PATH=$HADOOP_HOME/build/contrib/fuse_dfs:$PATH

for f in ls $HADOOP_HOME/lib/*.jar $HADOOP_HOME/*.jar ; do
export    CLASSPATH=$CLASSPATH:$f
done

if [ "$OS_ARCH" = "" ]; then
export OS_ARCH=i386
fi

if [ "$JAVA_HOME" = "" ]; then
export    JAVA_HOME=/usr/java/jdk1.6.0_21
fi

if [ "$LD_LIBRARY_PATH" = "" ]; then
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:/usr/local/share/hdfs/libhdfs/:/usr/local/lib
fi

./fuse_dfs $@

保存退出

挂载hdfs

$chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh
$chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs
$ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh /usr/local/bin
$ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs /usr/local/bin/

$mkdir /tmp/dfs
$cd $HADOOP_HOME/build/contrib/fuse-dfs
$./fuse_dfs_wrapper.sh dfs://192.168.64.99:9000 /tmp/dfs

卸载hdfs

Umount /tmp/dfs

开机自动挂载hdfs文件系统

vi /etc/fstab
fuse_dfs_wrapper.sh dfs:192.168.64.99:9000 /tmp/dfs    fuse rw,auto 0 0

相关推荐

硅步至千里 / 0评论 2020-04-19