mesos下载、准备安装环境、编译以及测试(转)

86530991 2017-05-06

参考网址:http://mesos.apache.org/gettingstarted/

实验环境:mesos版本:0.21.0、操作系统:RedHat6

为了了解Spark的三种部署standalone、mesos和YARN。所以想尝试部署下mesos集群。我根据自己的需求,参考了官方文档的如下步骤:

DownloadingMesos

TherearedifferentwaysyoucangetMesos:

DownloadthelateststablereleasefromApache(Recommended)

$ wget http://www.apache.org/dist/mesos/0.22.1/mesos-0.22.1.tar.gz
 $ tar -zxf mesos-0.22.1.tar.gz

CentOS6.5

FollowingaretheinstructionsforstockCentOS6.5.IfyouareusingadifferentOS,pleaseinstallthepackagesaccordingly.

Mesos0.21.0+requiressubversion1.8+develpackagewhichisnotavailablebydefaultbyyum.

Addoneoftherepothathassubversion-devel1.8available,i.e:

Addnewrepo/etc/yum.repos.d/wandisco-svn.repo,with:

[WandiscoSVN]

name=WandiscoSVNRepo

baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/

enabled=1

gpgcheck=0

 
$ sudo yum groupinstall -y "Development Tools"

  $ sudo yum install -y python-devel java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel

  # Install maven.
  $ wget http://mirror.nexcess.net/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
  $ sudo tar -zxf apache-maven-3.0.5-bin.tar.gz -C /opt/
  $ sudo ln -s /opt/apache-maven-3.0.5/bin/mvn /usr/bin/mvn
Building Mesos
    # Change working directory.
    $ cd mesos

    # Bootstrap (***Skip this if you are not building from git repo***).
    $ ./bootstrap

    # Configure and build.
    $ mkdir build
    $ cd build
    $ ../configure
    $ make

    # Run test suite.
    $ make check

    # Install (***Optional***).
    $ make install

Examples

MesoscomesbundledwithexampleframeworkswritteninC++,JavaandPython.

# Change into build directory.
 $ cd build

 # Start mesos master (***Ensure work directory exists and has proper permissions***).
 $ ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos

 # Start mesos slave.
 $ ./bin/mesos-slave.sh --master=127.0.0.1:5050

 # Visit the mesos web page.
 $ http://127.0.0.1:5050

 # Run C++ framework (***Exits after successfully running some tasks.***).
 $ ./src/test-framework --master=127.0.0.1:5050

 # Run Java framework (***Exits after successfully running some tasks.***).
 $ ./src/examples/java/test-framework 127.0.0.1:5050

 # Run Python framework (***Exits after successfully running some tasks.***).
 $ ./src/examples/python/test-framework 127.0.0.1:5050
NOTE: To build the example frameworks, make sure you build the test suite by doingmake check.

但是mesos的构建远比我想象的要难一些。它花了我大概一天半的时间对它进行下载、准备安装环境以及简单的测试。其中几乎每一个步骤都会遇到问题,所以特意记录下来,方便将来参考:

下载Mesos:

ApacheMesos官方文档上有两种方式来获取Mesos:

1、从Apache官网上下载(推荐的方式)。

$ wget http://www.apache.org/dist/mesos/0.21.0/mesos-0.21.0.tar.gz
 $ tar -zxf mesos-0.21.0.tar.gz
2、从 Mesos git repository上获取 (只适用于大神们的方式)
 $ git clone https://git-wip-us.apache.org/repos/asf/mesos.git

因为Spark1.3.1支持的是mesos0.21.0,所以我没有使用最新的mesos版本,所以使用了0.21.0版本。我采用的是第一种方式下载Mesos。下载下来的Mesos只有是源码,而没有二进制包。所以需要我们按照自己的操作系统来编译。在编译之前,首先确认编译mesos所需的环境。

对系统的要求:

Mesos运行在Linux(64Bit)和MacOSX(64Bit)上:

CentOS6.5

1、确保/etc/yum.repos.d下的源是CentOS的,并且版本也和Linux一致。

我的RedHat6采用的是CentOS6的源。如果Ubuntu,可以参考官方文档来做。

我之前RedHat6采用了CentOS5的源,导致后面环境更新时出现了问题,所以把CentOS6的源也记录下:

在/etc/yum.repos.d这个目录下,文件名可以自己取,我用的是CentOS6.repo这个文件名,其内容如下:

[base]

name=CentOS-6 - Base - 163.com
baseurl=http://mirrors.163.com/centos/6/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#gpgcheck=1
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##released updates
#[updates]
#name=CentOS-6 - Updates - 163.com
#baseurl=http://mirrors.163.com/centos/6/updates/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#gpgcheck=1
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##additional packages that may be useful
#[extras]
#name=CentOS-6 - Extras - 163.com
#baseurl=http://mirrors.163.com/centos/6/extras/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#gpgcheck=1
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##additional packages that extend functionality of existing packages
#[centosplus]
#name=CentOS-6 .2- Plus - 163.com
#baseurl=http://mirrors.163.com/centos/6/centosplus/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#gpgcheck=1
#enabled=0
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##contrib - packages by Centos Users
#[contrib]
#name=CentOS-6 - Contrib - 163.com
#baseurl=http://mirrors.163.com/centos/6/contrib/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#gpgcheck=1
#enabled=0
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
然后执行:
yum clean all && yum clean metadata && yum clean dbcache && yum makecache && yum update

2、Mesos0.21.0+需要subversion1.8+deve这个包,它在默认的yum是获取不到的。可以加入一个新的repo,/etc/yum.repos.d/wandisco-svn.repo它的内容是:

[WandiscoSVN]

name=WandiscoSVNRepo

baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/

enabled=1

gpgcheck=0

我不确定/etc/yum.repos.d/wandisco-svn.repo这个文件是否起到了作用,将wandisco-svn.repo这个文件添加到/etc/yum.repos.d。

3、执行sudoyumgroupinstall-y"DevelopmentTools",遇到如下问题:

publickeyforsystemtap-client-2.5-5.el6.x86_64.rpmisnotinstalled

原因是缺少systemtap-client-2.5-5.el6.x86_64.rpm。只要下载了这个rpm后执行rpm-ihvsystemtap-client-2.5-5.el6.x86_64.rpm安装即可,如果缺少其他依赖,就先安装其他依赖。

4、执行sudoyuminstall-ypython-develjava-1.7.0-openjdk-develzlib-devellibcurl-developenssl-develcyrus-sasl-develcyrus-sasl-md5apr-develsubversion-develapr-util-devel

5、安装maven

构建Mesos:

# Change working directory.
$ cd mesos
# Configure and build.
$ mkdir build
$ cd build

$../configure

在执行到../configure时遇到了如下问题:

1)configure:error:cannotfindlibz.libzisrequiredformesostobuild.

原因是缺少zlib-devel-1.2.3-25.el6.x86_64

需要下载相应的rpm,并安装:

rpm-ihvzlib-devel-1.2.3-25.el6.x86_64.rpm

2)configure:error:cannotfindlibcurl.libcurlisrequiredformesostobuild.

原因是缺少libcurl-devel-7.19.7-16.el6.x86_64

需要下载相应的rpm,并安装:

rpm-ihvlibidn-devel-1.18-2.el6.x86_64.rpm

rpm-ihvlibcurl-devel-7.19.7-16.el6.x86_64.rpm

3)configure:error:cannotfindlibapr-1.libapr-1isrequiredformesostobuild.

原因是缺少apr-devel-1.3.9-3.el6.x86_64

需要下载相应的rpm,并安装:

rpm -ihv apr-1.3.9-3.el6.x86_64.rpm 
rpm -ihv apr-devel-1.3.9-3.el6.x86_64.rpm
rpm -ihv  apr-util-1.3.9-3.el6.x86_64.rpm
4) configure:error:cannot find libsvn_subr-1 headers. libsubversion-1 is required for mesos to build.

从http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/找到下面的包,并安装:

rpm -ihv serf-1.2.0-1.x86_64.rpm
rpm -ihv subversion-debuginfo-1.8.0-1.x86_64.rpm
rpm -ihv subversion-1.8.0-1.x86_64.rpm  
rpm -ihv subversion-javahl-1.8.0-1.x86_64.rpm
rpm -ihv subversion-devel-1.8.0-1.x86_64.rpm

5)configure:error:cannotfindlibsasl2.Weneedlibsasl2forauthentication!

安装rpm-ihvcyrus-sasl-devel-2.1.23-8.el6.x86_64.rpm

6)如下图的错误:

libcurlisrequiredformesostobuild.

安装libcurl-devel

libapr-1isrequiredformesostobuild

安装apr-devel-1.3.9-3.el6.x86_64

libsubversion-1isrequiredformesostobuild.

#处理方法

yuminstallsubversion

yuminstallsubversion-javahl

yuminstallsubversion-devel

WeneedCRAM-MD5supportforSASLauthentication.

yuminstallcyrus-sasl-md5

configure:error:in`/data/mesos/mesos-0.28.0/build':

configure:error:

CouldnotlinktestprogramtoPython.MaybethemainPythonlibraryhasbeen

installedinsomenon-standardlibrarypath.Ifso,passittoconfigure,

viatheLDFLAGSenvironmentvariable.

Example:./configureLDFLAGS="-L/usr/non-standard-path/python/lib"

============================================================================

ERROR!

YouprobablyhavetoinstallthedevelopmentversionofthePythonpackage

foryourdistribution.Theexactnameofthispackagevariesamongthem.

============================================================================

See`config.log'formoredetails

yuminstallPython-devel

$make

遇到如下问题:

为了解决上面的问题,需要安装下列包:

rpm -ihv apr-util-1.3.9-3.el6.x86_64.rpm 
rpm -ihv apr-util-ldap-1.3.9-3.el6.x86_64.rpm 
rpm -ihv db4-cxx-4.7.25-16.el6.x86_64.rpm  
rpm -ihv db4-devel-4.7.25-16.el6.x86_64.rpm
rpm -ihv expat-devel-2.0.1-9.1.el6.x86_64.rpm 
rpm -ihv openldap-devel-2.4.19-15.el6.x86_64.rpm
rpm -ihv apr-util-devel-1.3.9-3.el6.x86_64.rpm

#Runtestsuite.

$makecheck

遇到如下问题:

意思是Linux的内核太旧了,这个错误我没鸟它,直接跳到下一步了。

#Install(***Optional***).

$makeinstall

WeneedCRAM-MD5supportforSASLauthentication.

yuminstallcyrus-sasl-md5

测试:

#Changeintobuilddirectory.

$ cd build
 # Start mesos master (***Ensure work directory exists and has proper permissions***).
 $ ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos
 # Start mesos slave.
 $ ./bin/mesos-slave.sh --master=127.0.0.1:5050   (可以在最后加个&)
 # Visit the mesos web page.
 $ http://127.0.0.1:5050
 # Run Java framework (***Exits after successfully running some tasks.***).
 $ ./src/examples/java/test-framework 127.0.0.1:5050

测试步骤没什么大问题,可以访问页面,需要注意的是,使用jps并不能查到mesos的进程.

相关推荐