岳英豪 2019-06-27
主要参考每天5分钟玩转docker容器技术博客。 记录下自己的实验过程。 如果想学习,请直接参考该博客http://www.cnblogs.com/CloudMan6/
flannel为每个host分配一个subnet,容器从此subnet中分配ip。这些ip可以在host间路由,容器间无需nat和port mapping就可以跨主机通信。
每个subnet都是从一个更大的ip池中划分的,flannel会在每个主机上运行一个叫flanneld的agent,职责是从池中分配subnet。为了在各个主机间共享信息,flannel用etcd存放网络配置、已分配的subnet、host的ip等信息。
数据包如何在主机间转发是由 backend 实现的。flannel 提供了多种 backend,最常用的有 vxlan 和 host-gw
在192.168.19.120上运行如下脚本:
ETCD_VER=v2.3.7 DOWNLOAD_URL=https://github.com/coreos/etcd/releases/download curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz mkdir -p /tmp/test-etcd && tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/test-etcd --strip-components=1 cp /tmp/test-etcd/etcd* /usr/local/bin/
运行etcd
etcd -listen-client-urls http://192.168.19.120:2379 -advertise-client-urls http://192.168.19.120:2379
docker pull cloudman6/kube-cross:v1.6.2-2
docker tag cloudman6/kube-cross:v1.6.2-2 gcr.io/google_containers/kube-cross:v1.6.2-2
git clone https://github.com/coreos/fla...
cd flannel
make dist/flanneld-amd64
scp dist/flanneld-amd64 192.168.19.121:/usr/local/bin/flanneld
scp dist/flanneld-amd64 192.168.19.122:/usr/local/bin/flanneld
配置文件如下:
{ "Network":"10.2.0.0/16", "SubnetLen":24, "Backend":{ "Type":"vxlan" } }
解释:
将配置存入etcd
etcdctl --endpoint=192.168.19.120:2379 set /docker-test/network/config < flannel_config.json
在两台机器上运行如下命令:
flanneld -etcd-endpoints=http://192.168.19.120:2379 -iface=enp0s3 -etcd-prefix=/docker-test/network
-etcd-endpoints 指定 etcd url
-iface 指定主机间数据传输使用的interface
-etcd-prefix 指定etcd存放flannel网络配置文件的key
输出如下:
root@host3:~# flanneld -etcd-endpoints=http://192.168.19.120:2379 -iface=enp0s3 -etcd-prefix=/docker-test/network I0520 02:24:57.594377 2463 main.go:499] Using interface with name enp0s3 and address 192.168.19.122 I0520 02:24:57.594766 2463 main.go:516] Defaulting external address to interface address (192.168.19.122) I0520 02:24:57.595348 2463 main.go:240] Created subnet manager: Etcd Local Manager with Previous Subnet: None I0520 02:24:57.595643 2463 main.go:243] Installing signal handlers I0520 02:24:57.598847 2463 main.go:358] Found network config - Backend type: vxlan I0520 02:24:57.599288 2463 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false I0520 02:24:57.649392 2463 local_manager.go:234] Picking subnet in range 10.2.1.0 ... 10.2.255.0 I0520 02:24:57.651626 2463 local_manager.go:220] Allocated lease (10.2.99.0/24) to current node (192.168.19.122) I0520 02:24:57.653626 2463 main.go:305] Wrote subnet file to /run/flannel/subnet.env I0520 02:24:57.654905 2463 main.go:309] Running backend. I0520 02:24:57.660772 2463 vxlan_network.go:60] watching for new subnet leases I0520 02:24:57.665879 2463 main.go:401] Waiting for 23h0m0.025221044s to renew lease I0520 02:24:57.672707 2463 iptables.go:115] Some iptables rules are missing; deleting and recreating rules I0520 02:24:57.674333 2463 iptables.go:137] Deleting iptables rule: -s 10.2.0.0/16 -j ACCEPT I0520 02:24:57.678549 2463 iptables.go:137] Deleting iptables rule: -d 10.2.0.0/16 -j ACCEPT I0520 02:24:57.688935 2463 iptables.go:125] Adding iptables rule: -s 10.2.0.0/16 -j ACCEPT I0520 02:24:57.696815 2463 iptables.go:125] Adding iptables rule: -d 10.2.0.0/16 -j ACCEPT
root@host2:~# ip addr show flannel.1 8: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 6e:9f:1d:bf:5c:b9 brd ff:ff:ff:ff:ff:ff inet 10.2.21.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::6c9f:1dff:febf:5cb9/64 scope link valid_lft forever preferred_lft forever
root@host2:~# ip r default via 192.168.19.2 dev enp0s3 onlink 10.2.99.0/24 via 10.2.99.0 dev flannel.1 onlink 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 linkdown 192.168.19.0/24 dev enp0s3 proto kernel scope link src 192.168.19.121 192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.101
*两台机器都做此操作
首先查看/run/flannel/subnet.env
root@host2:~# cat /run/flannel/subnet.env FLANNEL_NETWORK=10.2.0.0/16 FLANNEL_SUBNET=10.2.21.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false
之后,编辑docker的配置文件vim /etc/systemd/system/docker.service.d/10-machine.conf,给ExecStart命令的参数中添加--bip和--mtu,分别对应subnet.env的 FLANNEL_SUBNET 与FLANNEL_MTU
root@host2:~# cat /etc/systemd/system/docker.service.d/10-machine.conf [Service] ExecStart= ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic --bip=10.2.21.1/24 --mtu=1450 Environment=
systemctl daemon-reload systemctl restart docker.service
对比路由表
# docker重启前 root@host3:~# ip r default via 192.168.19.2 dev enp0s3 onlink 10.2.21.0/24 via 10.2.21.0 dev flannel.1 onlink 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 linkdown 192.168.19.0/24 dev enp0s3 proto kernel scope link src 192.168.19.122 192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.102 root@host3:~# systemctl daemon-reload root@host3:~# systemctl restart docker.service # docker重启后 root@host3:~# ip r default via 192.168.19.2 dev enp0s3 onlink 10.2.21.0/24 via 10.2.21.0 dev flannel.1 onlink 10.2.99.0/24 dev docker0 proto kernel scope link src 10.2.99.1 linkdown 172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 linkdown 192.168.19.0/24 dev enp0s3 proto kernel scope link src 192.168.19.122 192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.102
flannel 没有创建新的 docker 网络,而是直接使用默认的 bridge 网络。同一主机的容器通过 docker0 连接,跨主机流量通过 flannel.1 转发。
root@host2:~# docker run -itd --name bbox1 busybox root@host2:~# docker exec bbox1 ip r default via 10.2.21.1 dev eth0 10.2.21.0/24 dev eth0 scope link src 10.2.21.2 root@host3:~# docker run -itd --name bbox2 busybox root@host3:~# docker exec bbox2 ip r default via 10.2.99.1 dev eth0 10.2.99.0/24 dev eth0 scope link src 10.2.99.2
# 10.2.99.2 为bbox2的ip root@host2:~# docker exec bbox1 ping -c 1 10.2.99.2 PING 10.2.99.2 (10.2.99.2): 56 data bytes 64 bytes from 10.2.99.2: seq=0 ttl=62 time=0.604 ms
root@host2:~# docker exec bbox1 traceroute 10.2.99.2 traceroute to 10.2.99.2 (10.2.99.2), 30 hops max, 46 byte packets 1 10.2.21.1 (10.2.21.1) 0.010 ms 0.216 ms 0.020 ms 2 10.2.99.0 (10.2.99.0) 0.938 ms 2.731 ms 1.663 ms 3 10.2.99.2 (10.2.99.2) 2.016 ms 0.647 ms 0.596 ms
### dns不可用
无法通过主机名ping通机器。
###host字段指定授权使用该证书的etcd节点IP或子网列表,需要将etcd集群的3个节点都添加其中。cp etcd-v3.3.13-linux-amd64/etcd* /opt/k8s/bin/