yevvzi 2020-05-04
? 前面简单介绍了Kubernetes的基础理论原理与核心组件,本文将给出单节点以二进制部署Kubernetes集群的第一个步骤——搭建etcd集群实验流程。
1、三个节点上部署etcd集群
2、两个node节点上部署docker环境、部署flannel(容器内部之间的通信,依赖于vxlan技术)
3、master节点上部署kube-apiserver、kube-controller-manager、kube-scheduler
4、node节点上部署kubelet、kube-proxy
服务器ip地址规划
master01地址:192.168.0.128
node01地址:192.168.0.129
node02地址:192.168.0.130
本文将给出的是etcd集群的搭建配置流程。
? 三台服务器建议设置主机名称、绑定静态ip和关闭网络管理服务,其次需要关闭防火墙和核心防护,清空iptables
这里以master01为例,其他节点按照此节点类似设置即可
[ ~]# hostnamectl set-hostname master01 [ ~]# su [ ~]# systemctl stop firewalld [ ~]# setenforce 0 [ ~]# iptables -F
设置静态ip地址
[ ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [ ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="static" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="7c933cbb-b29c-4a36-bb12-d1ac1c505524" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.0.128" NETMASK="255.255.255.0" GATEWAY="192.168.0.2" DNS1=192.168.0.2 [ ~]# systemctl restart network [ ~]# ip add 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:06:97:04 brd ff:ff:ff:ff:ff:ff inet 192.168.0.128/24 brd 192.168.0.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::5bd9:44c7:cf2c:ef20/64 scope link valid_lft forever preferred_lft forever [ ~]# ping www.baidu.com #这里测试是否可以连外网 PING www.a.shifen.com (180.101.49.12) 56(84) bytes of data. 64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=1 ttl=128 time=13.4 ms 64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=2 ttl=128 time=11.2 ms 64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=3 ttl=128 time=11.1 ms ^C --- www.a.shifen.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2008ms rtt min/avg/max/mdev = 11.135/11.967/13.476/1.068 ms #关闭网络管理 [ ~]# systemctl stop NetworkManager [ ~]# systemctl disable NetworkManager Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service. Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service. Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
首先、在master节点上,安装制作查看证书的相关工具,如:cfssl、cfssljson、cfssl-certinfo
//先自己创建并且进入工作目录 [ ~]# mkdir k8s [ ~]# cd k8s/ #编写脚本下载上述三个命令工具到指定的目录下 [ k8s]# vim cfssl.sh #脚本在下一个代码块中另外给出 [ k8s]# bash cfssl.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9.8M 100 9.8M 0 0 131k 0 0:01:17 0:01:17 --:--:-- 47667 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2224k 100 2224k 0 0 823k 0 0:00:02 0:00:02 --:--:-- 823k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6440k 100 6440k 0 0 426k 0 0:00:15 0:00:15 --:--:-- 523k
下载命令的shell脚本如下:
[ k8s]# cat cfssl.sh curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo #查看命令是否成功安装到指定目录 [ k8s]# ls /usr/local/bin/ cfssl cfssl-certinfo cfssljson
用于创建生成ca证书的工具已经有啦,现在就需要材料(各种文件)来创建证书了
创建证书生成目录,从而生成各种需要的证书
[ k8s]# mkdir etcd-cert [ k8s]# cd etcd-cert/ [ etcd-cert]# ls #编写证书创建需要的材料——各种文件,可以依赖shell脚本一键执行,下一个代码块将解释脚本的主要内容及相关参数 [ etcd-cert]# vim etcd-cert.sh #执行脚本 [ etcd-cert]# bash etcd-cert.sh 2020/05/03 20:05:57 [INFO] generating a new CA key and certificate from CSR 2020/05/03 20:05:57 [INFO] generate received request 2020/05/03 20:05:57 [INFO] received CSR 2020/05/03 20:05:57 [INFO] generating key: rsa-2048 2020/05/03 20:05:57 [INFO] encoded CSR 2020/05/03 20:05:57 [INFO] signed certificate with serial number 594719677485784071979843988457153533430072455164 2020/05/03 20:05:57 [INFO] generate received request 2020/05/03 20:05:57 [INFO] received CSR 2020/05/03 20:05:57 [INFO] generating key: rsa-2048 2020/05/03 20:05:58 [INFO] encoded CSR 2020/05/03 20:05:58 [INFO] signed certificate with serial number 603393813825663113730743133623713339445920555574 2020/05/03 20:05:58 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). #查看通过脚本执行产生的文件,下面将剖析这些文件的由来和相关内容解释 [ etcd-cert]# ls ca-config.json ca-csr.json ca.pem server.csr server-key.pem ca.csr ca-key.pem etcd-cert.sh server-csr.json server.pem
shell脚本文件和相关解释(复制脚本后先取消我的注释,否则会报错)
#编写ca证书配置文件,第一个json文件 cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" #过期时间为10年 }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", #服务和客户端的授权验证 表示client可以用该 CA 对server提供的证书进行验证; "client auth" #表示server可以用该CA对client提供的证书进行验证; ] } } } } EOF #编写ca证书签名文件,第二个json文件 cat > ca-csr.json <<EOF { "CN": "etcd CA", "key": { "algo": "rsa",#非对称密钥格式 "size": 2048 #字节长度 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF #使用下载的命令依赖编写的json文件来生成ca证书,从而生成两个pem证书文件:ca.pem和ca-key.pem cfssl gencert -initca ca-csr.json | cfssljson -bare ca - #----------------------- #指定etcd集群中三个节点之间的通信验证,需要编写server端的验证签名文件第三个json文件 cat > server-csr.json <<EOF { "CN": "etcd", "hosts": [ "192.168.0.128", "192.168.0.129", "192.168.0.130" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF #使用cfssl命令依赖ca证书和配置文件,-profiles是指定特定的使用场景,将会生成: # server-key.pem 和 server.pem证书(私钥) cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
梳理一下,脚本为一个.sh文件,脚本中编写了三个json文件,通过命令生成4个证书(私钥)文件(一对为ca的、另一对为server端的)以及对应的签名文件(ca.csr和server.csr文件)
以上我们需要的证书就创建完毕了,下面开始etcd集群的搭建了
首先我们需要下载etcd软件包,这边我使用第三方工具从官方上下载下来的,可以给大家使用,软件包链接
链接:https://pan.baidu.com/s/1jMeYk2oAYNE9woDBlgnZYA
提取码:wwz4
etcd软件包的网站链接也贴出来:ETCD 二进制包地址
https://github.com/etcd-io/etcd/releases
资源给出来了下面开始干活吧!^_^
首先解压该软件包
[ k8s]# ls cfssl.sh etcd-cert etcd-v3.3.10-linux-amd64.tar.gz [ k8s]# tar zxf etcd-v3.3.10-linux-amd64.tar.gz [ k8s]# cd etcd-v3.3.10-linux-amd64/ #查看一下软件包中的文件 [ro etcd-v3.3.10-linux-amd64]# ls Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
#我们需要的是etcd和etcdctl两个命令工具待会我们需要将它们移动到我们创建的集群搭建目录中,现在我们创建各种命令,然后将需要的文件存入或创建写入对应的目录中
首先创建集群搭建的各种目录
[ ~]# mkdir /opt/etcd/{cfg,bin,ssl} -p #分别为配置文件目录,命令目录和证书目录 [ ~]# ls -R /opt/etcd/ /opt/etcd/: bin cfg ssl /opt/etcd/bin: /opt/etcd/cfg: /opt/etcd/ssl: #目前目录中没有任何文件
移动相关文件
[ etcd-v3.3.10-linux-amd64]# mv etcd etcdctl /opt/etcd/bin/ [ etcd-v3.3.10-linux-amd64]# cd /root/k8s/etcd-cert/ [ etcd-cert]# ls ca-config.json ca-csr.json ca.pem server.csr server-key.pem ca.csr ca-key.pem etcd-cert.sh server-csr.json server.pem [ etcd-cert]# cp *.pem /opt/etcd/ssl [ etcd-cert]# ls -R /opt/etcd/ /opt/etcd/: bin cfg ssl /opt/etcd/bin: etcd etcdctl /opt/etcd/cfg: /opt/etcd/ssl: ca-key.pem ca.pem server-key.pem server.pem
现在就差配置文件(包括配置文件和服务启动文件)了,这里依旧可以使用shell脚本(真正体会到shell脚本的魅力了吧)来编写,务必注意将其中的注释去除后再执行该脚本
[ etcd-cert]# cd /opt/etcd/cfg/ [ cfg]# ls [ cfg]# vim etcd.sh #!/bin/bash # example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380 #位置变量 ETCD_NAME=$1 ETCD_IP=$2 ETCD_CLUSTER=$3 #工作路径 WORK_DIR=/opt/etcd #重定向写入文件 cat <<EOF >$WORK_DIR/cfg/etcd #[Member] ETCD_NAME="${ETCD_NAME}" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380" #etcd群集服务器之间通信端口 ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"#外部访问节点的端口 #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380" ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379" ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #令牌名称集群节点服务器需要保持一致 ETCD_INITIAL_CLUSTER_STATE="new" EOF cat <<EOF >/usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=${WORK_DIR}/cfg/etcd ExecStart=${WORK_DIR}/bin/etcd --name=\${ETCD_NAME} --data-dir=\${ETCD_DATA_DIR} --listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} --listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 --advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} --initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} --initial-cluster=\${ETCD_INITIAL_CLUSTER} --initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} --initial-cluster-state=new --cert-file=${WORK_DIR}/ssl/server.pem --key-file=${WORK_DIR}/ssl/server-key.pem --peer-cert-file=${WORK_DIR}/ssl/server.pem --peer-key-file=${WORK_DIR}/ssl/server-key.pem --trusted-ca-file=${WORK_DIR}/ssl/ca.pem --peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable etcd systemctl restart etcd
此时可以先执行该脚本,上述脚本中给出了执行该脚本的example,会进入阻塞等待节点加入状态,如果在一定时间内没有节点加入则会由于timeout(超时)而终止阻塞状态
[ cfg]# ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380
可以重新开一个终端查看etcd服务状态
[ ~]# ps -ef | grep etcd root 50829 15209 0 20:46 pts/1 00:00:00 /bin/bash ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380 root 50874 50829 0 20:46 pts/1 00:00:00 systemctl restart etcd root 50880 1 4 20:46 ? 00:00:01 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.0.128:2380 --listen-client-urls=https://192.168.0.128:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.0.128:2379 --initial-advertise-peer-urls=https://192.168.0.128:2380 --initial-clusteretcd01=https://192.168.0.128:2380,etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem root 50937 50896 0 20:46 pts/2 00:00:00 grep --color=auto etcd
#一定时间内没有节点加入集群就会推出阻塞状态,由于目前其他节点还没有进行任何设置因此一定会退出的 [ cfg]# ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380 Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
下面我们将master节点的etcd目录中的文件以及服务启动文件远程拷贝到两个节点服务器的对应目录中,这里以node1节点为例
[ cfg]# scp -r /opt/etcd :/opt The authenticity of host ‘192.168.0.129 (192.168.0.129)‘ can‘t be established. ECDSA key fingerprint is SHA256:bkzGRcdP2iJrSTerWtyuqSDENF2mKLWUZHMRkzJZBFI. ECDSA key fingerprint is MD5:b0:9b:9f:31:de:da:51:8a:d3:ff:87:86:fa:19:63:2c. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ‘192.168.0.129‘ (ECDSA) to the list of known hosts. ‘s password: etcd.sh 100% 1812 1.5MB/s 00:00 etcd 100% 509 583.4KB/s 00:00 etcd 100% 18MB 109.8MB/s 00:00 etcdctl 100% 15MB 104.3MB/s 00:00 ca-key.pem 100% 1679 689.5KB/s 00:00 ca.pem 100% 1265 1.1MB/s 00:00 server-key.pem 100% 1675 1.1MB/s 00:00 server.pem 100% 1338 2.2MB/s 00:00 [ cfg]# scp /usr/lib/systemd/system/etcd.service :/usr/lib/systemd/system/ ‘s password: etcd.service 100% 923 430.4KB/s 00:00
下面再到各个节点是进行配置修改(主要是配置文件中的etcd集群节点名称和ip地址)
[ ~]# cd /opt/etcd/cfg/ [ cfg]# ls etcd etcd.sh #主要修改配置文件etcd [ cfg]# vim etcd #修改ip地址和节点名称 #[Member] ETCD_NAME="etcd02" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.129:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.129:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.129:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.129:2379" ETCD_INITIAL_CLUSTER="etcd01=https://192.168.0.128:2380,etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
此时我们再在master上执行集群搭建命令,然后在两个节点服务器上快速启动etcd服务,服务启动好后会自动退出阻塞状态
[ cfg]# ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380
#两个节点启动etcd服务 [ cfg]# systemctl start etcd [ cfg]# systemctl start etcd
最后我们来检查一下etcd集群状态,如果是health状态则说明etcd集群搭建成功
[ cfg]# cd ../ssl/ #需要依赖证书来检查该集群状态 [ ssl]# ls ca-key.pem ca.pem server-key.pem server.pem [ ssl]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.0.128:2379,https://192.168.0.129:2379,https://192.168.0.130:2379" cluster-health member a25c294d3a391c7c is healthy: got healthy result from https://192.168.0.128:2379 member b2db359ffad36ee5 is healthy: got healthy result from https://192.168.0.129:2379 member eddae83baed564ba is healthy: got healthy result from https://192.168.0.130:2379 cluster is healthy
好了,etcd到此就搭建成功了,后面我们将继续部署其他组件,因此建议如果是实验环境下将虚拟机挂起为最优选择,负责当越来越多的服务来的时候,关机后再启动可能导致各种问题。