LVS双机热备和负载均衡一体化安装方法

heartbeat  以心跳机制监控实体主机是否存活，实现ipvsadm和ldirectord服务的故障转移。
ldirectord 可用来監控实体主機的服務进程是否正常，以决定在ipvsadm配置中删除或加入。

在本案例中，将使用CentOS6系统来安装传统的heartbeat+ldirector以实现双机热备的功能,因为RHEL均已经移除相关套件，所以安装比较麻烦一点，但只要一步一步做下来，其实也并不是太困难，而且还能够学习到很多东西。

另外请注意，因为本案例中将DR主机和RS主机合并在一起以节省IT资源，因此本案例只适合部署于同一子网内的主机(因为虚拟的VIP不能改变)，也因此之故，ipvsadm也就可选择使用Director Routing 模式。

                        ________
                       |        |
                       | client |
                       |________|
                           |
             CIP->VIP |    |   ^ 
                      v    |   | VIP->CIP
                           |
          ------------------------------------
          |                                  |
          |                                  |
  RIP1=192.168.57.101             RIP2=192.168.57.102 (eth0)
  VIP =192.168.57.200             VIP =192.168.57.200 (eth0:0, arps)
  VIP =192.168.57.200             VIP =192.168.57.200 (all lo:0,non-arping)
    _____________                      _____________
   |             |                    |             |
   | Director1 + |                    | Director2 + |
   | RealServer1 |                    | RealServer2 |
   |_____________|                    |_____________|

nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
调试应注意的情况:
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
另外，调试中要注意一个情况，ipvsadm所添加的规则应用于内核的路由，优先于其他网络
服务。举个例子来说，比如某IP的端口80被ipsvadm设置为转发，而同时本机又有HTTPD服务
使用此IP的80端口，那么ipvsadm规则将应用在先，而其他IP的端口80的服务则不受影响。

nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
注意ARP地址的更新：
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
因为我使用虚拟主机进行测试，所以ARP地址需要手动更新，否则测试会停留在旧地址上：
# arping -I vboxnet1 192.168.57.200;

一般LINUX系统预设的ARP缓存timeout时间是60秒，如有需要，您可以调整此内核参数：
# cat /proc/sys/net/ipv4/neigh/vboxnet1/gc_stale_time
# echo 5 > /proc/sys/net/ipv4/neigh/vboxnet1/gc_stale_time

如果您使用硬件设备作网关，如有需要，则请相应调整的硬件参数，例如，CISCO路由器的
预设ARP缓存时间是14400秒（4小时），如果您不想修改此参数，那就肯定需要手动刷新了。
CISCO 路由器用于操作 arp cache 的相关命令可参考如下：
router# show arp
router# clear arp-cache

────────────────────────────────────────────────────────────────────────────────
LVS
────────────────────────────────────────────────────────────────────────────────
Official Site:
http://www.linuxvirtualserver.org/
http://zh.linuxvirtualserver.org/

Ref:
http://www.ultramonkey.org/3/lvs.html
http://www.ultramonkey.org/3/linux-ha.html
http://www.ultramonkey.org/3/ldirectord.html

# yum install ipvsadm;

注意： 在本案例中，因为双机热备需要两边node完全相同的配置，因此所有命令都同时
适用于两台node主机，除了 heartbeat配置配置文档/etc/ha.d/ha.cf有点稍微不同之外，
其他配置也完全相同。

────────────────────────────────────────────────────────────────────────────────
Linux-HA:注意最新版本(heartbeat 3)作了相当大的改动
────────────────────────────────────────────────────────────────────────────────
http://www.linux-ha.org/wiki/Main_Page

Heartbeat 3 的组成
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
Heartbeat 3 按模块拆分为三个子项目，提供了一个名为 cluster-glue 的组件， 专用于
Local Resource Manager 的管理。即heartbeat + cluster-glue + resouce-agent三部分：
(1) hearbeat集群心跳, 负责维护集群各节点的信息以及它们之前通信;
(2) cluster-glue相当于一个中间层，可将heartbeat和crm(pacemaker)联系起来，包括LRM和STONITH;
(3) resource-agent各种资源的ocf脚本，这些脚本将被LRM调用从而实现各种资源启动、停止、监控等;

http://www.linux-ha.org/wiki/Download
In post-2.1.4 releases, Linux-HA code is packaged in three different sub-projects:
  * Heartbeat:http://www.linux-ha.org/wiki/Heartbeat
  * Cluster Glue:http://www.linux-ha.org/wiki/Cluster_Glue
  * Resource Agents:http://www.linux-ha.org/wiki/Resource_Agents

http://www.linux-ha.org/wiki/Resource_Agents
Pacemaker supports three types of Resource Agents:
  * LSB Resource Agents:http://www.linux-ha.org/wiki/LSB_Resource_Agents
  * OCF Resource Agents:http://www.linux-ha.org/wiki/OCF_Resource_Agents
  * legacy Heartbeat Resource Agents: http://www.linux-ha.org/wiki/Heartbeat_Resource_Agents

Mercurial Repositories:
http://hg.linux-ha.org/agents/

################################################################################
RHEL6/CentOS6 (Heartbeat3: heartbeat + cluster-glue + resouce-agent)
################################################################################
REF:
http://ftp.jaist.ac.jp/pub/Linux/Fedora/
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/i386/
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/x86_64/
# rpm -Uvh http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/i386/epel-release-6-7.noarch.rpm
# yum search heartbeat;
# yum install heartbeat;

────────────────────────────────────────────────────────────────────────────────
Install ldirectord:
────────────────────────────────────────────────────────────────────────────────
nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
Note: RHEL6 only added ldirectord to RHN Satellite server in a separate channel.
uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu
http://horms.net/projects/ldirectord/
http://en.wikipedia.org/wiki/Ldirectord

ldirector (Linux Director Daemon) is a background process computer program used
to monitor and administer real servers in the Linux Virtual Server (LVS) cluster.

********************************************************************************
Download: http://www.linbit.com/support/hb/rhel6/ (Need non-free support login )
********************************************************************************
wget http://www.linbit.com/support/hb/rhel6/ldirectord-3.9.3-1.i686.rpm
wget http://www.linbit.com/support/hb/rhel6/ldirectord-3.9.3-1.x86_64.rpm
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

# vi /etc/yum.repos.d/slinux.repo;#(same as centos6, not include ldirectord)
--------------------------------------------------------------------------------
[scientific-linux]
name=Scientific Linux
#baseurl=http://ftp.scientificlinux.org/linux/scientific/6/x86_64/os/
baseurl=http://ftp.scientificlinux.org/linux/scientific/6/i386/os/
enabled=0
--------------------------------------------------------------------------------
Note: this is not working, here is only for recoding the RedHat dirty things !

http://hg.linux-ha.org/agents/
https://github.com/ClusterLabs/resource-agents

# yum install gcc autoconf automake glib2-devel;
# yum install heartbeat resource-agents cluster-glue-libs-devel;
# yum install perl-Socket6 perl-libwww-perl perl-IO-Socket-INET6 perl-MailTools;
# yum install docbook-*;

Download the resource-agents:
# wget https://github.com/ClusterLabs/resource-agents/zipball/master \
   -O ClusterLabs-resource-agents-v3.9.3-47-g0bf8af1.zip
# unzip ClusterLabs-resource-agents-v3.9.3-47-g0bf8af1.zip
# cd ClusterLabs-resource-agents-v3.9.3-47-g0bf8af1/;

OR you can download with git tools:
# yum install git;
# git clone git://github.com/ClusterLabs/resource-agents
# cd resource-agents/

# ./autogen.sh
# ./configure --hlep
# ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
--------------------------------------------------------------------------------
resource-agents configuration:
  Version                  = 3.9.3.47-0bf8a
  Build Version            = :%H$
  Features                 =
  Prefix                   = /usr
  Executables              = /usr/sbin
  Man pages                = /usr/share/man
  Libraries                = /usr/lib
  Header files             = /usr/include
  Arch-independent files   = /usr/share
  Documentation            = /usr/share/doc/resource-agents
  State information        = /var
  System configuration     = /etc
  RA state files           = /var/run/resource-agents
  AIS Plugins              =
  Libraries                =  -lglib-2.0
  Stack Libraries          =
--------------------------------------------------------------------------------

# make

# cp ldirectord/ldirectord /usr/sbin/
# cp ldirectord/init.d/ldirectord /etc/init.d/
# chmod 755 /usr/sbin/ldirectord
# chmod 755 /etc/init.d/ldirectord

# cp ldirectord/ldirectord.cf /etc/ha.d/

# vi /etc/ha.d/ldirectord.cf;
--------------------------------------------------------------------------------
checktimeout=3
checkinterval=1
logfile = "/var/log/ldirectord.log"
autoreload=yes
#quiescent=yes
quiescent=no
virtual=192.168.57.200:80
real=192.168.57.100:80 gate
real=192.168.58.100:80 gate
#real=192.168.57.100:80 masq
#real=192.168.58.100:80 masq
#real=192.168.57.100:80 ipip
#real=192.168.58.100:80 ipip
 fallback=127.0.0.1:80 gate
service=http
request="healthcheck.html"
receive="OK"
virtualhost=www.testing.net
scheduler=rr
#scheduler=wrr
#scheduler=wlc
persistent=600
#netmask=255.255.255.255
protocol=tcp
  checktype=negotiate
  checkport=80
--------------------------------------------------------------------------------

参数说明：
gatesame as ipvsadm-gLVS/DR
ipipsame as ipvsadm-iLVS/TUN
masqsame as ipvsadm-mLVS/NAT

# vi /etc/rc.local;
--------------------------------------------------------------------------------
/sbin/ifconfig lo:0 192.168.57.200 netmask 255.255.255.255 broadcast 192.168.57.255 up
/sbin/route add -host 192.168.57.200 dev lo:0
echo 0 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
--------------------------------------------------------------------------------

────────────────────────────────────────────────────────────────────────────────
Configurate heartbeat(for DR fail-over)
────────────────────────────────────────────────────────────────────────────────
# cp /usr/share/doc/heartbeat-3.0.4/ha.cf /etc/ha.d/
# cp /usr/share/doc/heartbeat-3.0.4/authkeys /etc/ha.d/

# dd if=/dev/urandom bs=512 count=1 | openssl md5

# vi /etc/ha.d/authkeys;#(Same on both DR servers);
--------------------------------------------------------------------------------
auth 1
1 sha1 46b61a7f417d395c712d9ecf30665aaf
--------------------------------------------------------------------------------
# chmod 600 /etc/ha.d//authkeys;

# vi /etc/init.d/startvip;#(Same on both DR servers);
--------------------------------------------------------------------------------
#!/bin/bash
VIP=192.168.57.200
case $1 in
start)
    IP=`/sbin/ifconfig lo:0 | grep $VIP`
    if [ ! -z "$IP" ] ; then
      /sbin/ifconfig lo:0 down
    fi
    /sbin/ifconfig eth0:0 $VIP netmask 255.255.255.255 broadcast 192.168.57.255 up
    /sbin/arping -q -c 1 -w 1 -I eth0 $VIP ;  echo ""
    ;;
stop)
    IP=`/sbin/ifconfig eth0:0 | grep $VIP`
    if [ ! -z "$IP" ] ; then
      /sbin/ifconfig eth0:0 down
    fi
    echo 0 > /proc/sys/net/ipv4/ip_forward
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast 192.168.57.255 up
    /sbin/route add -host $VIP dev lo:0
    ;;
status)
    /sbin/ifconfig eth0:0
    ;;
*)
    echo "Syntax incorrect. You need one of {start|stop|status}"
    ;;
esac
--------------------------------------------------------------------------------
chmod 755 /etc/init.d/startvip;

关于 apring 命令的说明：

/sbin/arping -q -c 1 -w 1 -I eth0 $VIP ;  echo ""

如前所述，这行命令用于通知本地网络更新arp地址，因为在shell环境这个命令总是返回
大于0的结果，导致heartbeat误认为服务失败，所以我在此命令后跟了个echo命令以消除其
返回数值。请注意，使用 arping 命令不保证一定能够更新您的网络环境中的所有arp缓存
记录，在实用中请对照您的网络设备（路由器）的手册调整相关设置。

********************************************************************************
注意：在此案例中请不要使用Heartbeat资源所自带的IPAddr脚本来启动虚拟IP地址，因为
本案例设计架构所使用的本地 lo:0 含有与 eth0:0 相同的VIP地址，因此 IPAddr 脚本可
能会报错并会拒绝启动相关资源。
********************************************************************************

# vi /etc/ha.d/haresources;#(Same on both DR servers);
--------------------------------------------------------------------------------
centos6-ocfs2-1 startvip ldirectord
--------------------------------------------------------------------------------
Note: 这里不需要加入ipvsadm服务，因为现在ipvsadm服务改由ldirectord服务负责启动。

DR1# vi /etc/ha.d/ha.cf;#(on DR centos6-ocfs2-1);
--------------------------------------------------------------------------------
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth0
ucast eth0 192.168.57.102
keepalive 2
auto_failback on
node centos6-ocfs2-1
node centos6-ocfs2-2
--------------------------------------------------------------------------------
Note: 这里的IP地址是对端node主机的IP地址。

DR2# vi /etc/ha.d/ha.cf;#(on DR centos6-ocfs2-2);
--------------------------------------------------------------------------------
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth0
ucast eth0 192.168.57.101
keepalive 2
auto_failback on
node centos6-ocfs2-1
node centos6-ocfs2-2
--------------------------------------------------------------------------------
Note: 这里的IP地址是对端node主机的IP地址。

设置并启动服务：

# chkconfig --list ipvsadm;
# chkconfig ipvsadm off;
# chkconfig ldirectord off;
# chkconfig heartbeat on;

注意：在这里，Centos 只需要启动heartbeat,然后由heartbeat负责启动ldirectord服务，
再由ldirectord服务启动并管理ipvsadm服务。

最后，分别在 RS1 和 RS2 主机启动 heartbeat 服务：

RS1# service heartbeat start;
RS2# service heartbeat start;

观察启动结果：
# service ldirectord status;
# service ipvsadm status;
# ip addr show;