kubernetes入门到实战(八)Pod健康检查机制

JustHaveTry 2020-06-12

1. 健康检查

1.1 健康检查概述

kubernetes入门到实战(八)Pod健康检查机制

应用在运行过程中难免会出现错误,如程序异常,软件异常,硬件故障,网络故障等,kubernetes提供Health Check健康检查机制,当发现应用异常时会自动重启容器,将应用从service服务中剔除,保障应用的高可用性。k8s定义了三种探针Probe:

  • readiness probes 准备就绪检查,通过readiness是否准备接受流量,准备完毕加入到endpoint,否则剔除
  • liveness probes 在线检查机制,检查应用是否可用,如死锁,无法响应,异常时会自动重启容器
  • startup probes 启动检查机制,应用一些启动缓慢的业务,避免业务长时间启动而被前面的探针kill掉

每种探测机制支持三种健康检查方法,分别是命令行exec,httpGet和tcpSocket,其中exec通用性最强,适用与大部分场景,tcpSocket适用于TCP业务,httpGet适用于web业务。

  • exec 提供命令或shell的检测,在容器中执行命令检查,返回码为0健康,非0异常
  • httpGet http协议探测,在容器中发送http请求,根据http返回码判断业务健康情况
  • tcpSocket tcp协议探测,向容器发送tcp建立连接,能建立则说明正常

每种探测方法能支持几个相同的检查参数,用于设置控制检查时间:

  • initialDelaySeconds 初始第一次探测间隔,用于应用启动的时间,防止应用还没启动而健康检查失败
  • periodSeconds 检查间隔,多久执行probe检查,默认为10s
  • timeoutSeconds 检查超时时长,探测应用timeout后为失败
  • successThreshold 成功探测阈值,表示探测多少次为健康正常,默认探测1次

1.2 exec命令行健康检查

许多应用程序运行过程中无法检测到内部故障,如死锁,出现故障时通过重启业务可以恢复,kubernetes提供liveness在线健康检查机制,我们以exec为例,创建一个容器启动过程中创建一个文件/tmp/liveness-probe.log,10s后将其删除,定义liveness健康检查机制在容器中执行命令ls -l /tmp/liveness-probe.log,通过文件的返回码判断健康状态,如果返回码非0,暂停20s后kubelet会自动将该容器重启。

1、定义一个容器,启动时创建一个文件,健康检查时ls -l /tmp/liveness-probe.log返回码为0,健康检查正常,10s后将其删除,返回码为非0,健康检查异常

[ demo]# cat centos-exec-liveness-probe.yaml
apiVersion: v1
kind: Pod
metadata:
  name: exec-liveness-probe
  annotations:
    kubernetes.io/description: "exec-liveness-probe"
spec:
  containers:
  - name: exec-liveness-probe
    image: centos:latest
    imagePullPolicy: IfNotPresent
    args:    #容器启动命令,生命周期为30s
    - /bin/sh
    - -c
    - touch /tmp/liveness-probe.log && sleep 10 && rm -f /tmp/liveness-probe.log && sleep 20
    livenessProbe:
      exec:  #健康检查机制,通过ls -l /tmp/liveness-probe.log返回码判断容器的健康状态
        command:
        - ls 
        - l 
        - /tmp/liveness-probe.log
      initialDelaySeconds: 1
      periodSeconds: 5
      timeoutSeconds: 1

2、应用配置生成容器

[ demo]# kubectl apply -f centos-exec-liveness-probe.yaml 
pod/exec-liveness-probe created

3、查看容器的event日志,容器启动后,10s以内容器状态正常,11s开始执行liveness健康检查,检查异常,触发容器重启

[ demo]# kubectl describe pods exec-liveness-probe | tail
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  28s                default-scheduler  Successfully assigned default/exec-liveness-probe to node-3
  Normal   Pulled     27s                kubelet, node-3    Container image "centos:latest" already present on machine
  Normal   Created    27s                kubelet, node-3    Created container exec-liveness-probe
  Normal   Started    27s                kubelet, node-3    Started container exec-liveness-probe
  #容器已启动
  Warning  Unhealthy  20s (x2 over 25s)  kubelet, node-3    Liveness probe failed: /tmp/liveness-probe.log
ls: cannot access l: No such file or directory #执行健康检查,检查异常
  Warning  Unhealthy  15s  kubelet, node-3  Liveness probe failed: ls: cannot access l: No such file or directory
ls: cannot access /tmp/liveness-probe.log: No such file or directory
  Normal  Killing  15s  kubelet, node-3  Container exec-liveness-probe failed liveness probe, will be restarted
  #重启容器

4、查看容器重启次数,容器不停的执行,重启次数会响应增加,可以看到RESTARTS的次数在持续增加

[ demo]# kubectl get pods exec-liveness-probe 
NAME                  READY   STATUS    RESTARTS   AGE
exec-liveness-probe   1/1     Running   6          5m19s

1.3 httpGet健康检查

1、httpGet probe主要主要用于web场景,通过向容器发送http请求,根据返回码判断容器的健康状态,返回码小于4xx即表示健康,如下定义一个nginx应用,通过探测http://<container>:port/index.html的方式判断健康状态

[ demo]# cat nginx-httpGet-liveness-readiness.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-httpget-livess-readiness-probe 
  annotations:
    kubernetes.io/description: "nginx-httpGet-livess-readiness-probe"
spec:
  containers:
  - name: nginx-httpget-livess-readiness-probe
    image: nginx:latest
    ports:
    - name: http-80-port
      protocol: TCP
      containerPort: 80
    livenessProbe:   #健康检查机制,通过httpGet实现实现检查
      httpGet:
        port: 80
        scheme: HTTP
        path: /index.html
      initialDelaySeconds: 3
      periodSeconds: 10
      timeoutSeconds: 3

2、生成pod并查看健康状态

[ demo]# kubectl apply -f nginx-httpGet-liveness-readiness.yaml 
pod/nginx-httpget-livess-readiness-probe created
[ demo]# kubectl get pods nginx-httpget-livess-readiness-probe 
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-httpget-livess-readiness-probe   1/1     Running   0          6s

3、模拟故障,将pod中的path文件所属文件删除,此时发送http请求时会健康检查异常,会触发容器自动重启

查询pod所属的节点
[ demo]# kubectl get pods nginx-httpget-livess-readiness-probe -o wide 
NAME                                   READY   STATUS    RESTARTS   AGE    IP            NODE     NOMINATED NODE   READINESS GATES
nginx-httpget-livess-readiness-probe   1/1     Running   1          3m9s   10.244.2.19   node-3   <none>           <none>

登录到pod中将文件删除
[ demo]# kubectl exec -it nginx-httpget-livess-readiness-probe /bin/bash
:/# ls -l /usr/share/nginx/html/index.html 
-rw-r--r-- 1 root root 612 Sep 24 14:49 /usr/share/nginx/html/index.html
:/# rm -f /usr/share/nginx/html/index.html

4、再次查看pod的列表,此时会RESTART的次数会增加1,表示重启重启过一次,AGE则多久前重启的时间

[ demo]# kubectl get pods nginx-httpget-livess-readiness-probe 
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-httpget-livess-readiness-probe   1/1     Running   1          4m22s

5、查看pod的详情,观察容器重启的情况,通过Liveness 检查容器出现404错误,触发重启。

[ demo]# kubectl describe pods nginx-httpget-livess-readiness-probe | tail
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m45s                  default-scheduler  Successfully assigned default/nginx-httpget-livess-readiness-probe to node-3
  Normal   Pulling    3m29s (x2 over 5m45s)  kubelet, node-3    Pulling image "nginx:latest"
  Warning  Unhealthy  3m29s (x3 over 3m49s)  kubelet, node-3    Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    3m29s                  kubelet, node-3    Container nginx-httpget-livess-readiness-probe failed liveness probe, will be restarted
  Normal   Pulled     3m25s (x2 over 5m41s)  kubelet, node-3    Successfully pulled image "nginx:latest"
  Normal   Created    3m25s (x2 over 5m40s)  kubelet, node-3    Created container nginx-httpget-livess-readiness-probe
  Normal   Started    3m25s (x2 over 5m40s)  kubelet, node-3    Started container nginx-httpget-livess-readiness-probe

1.4 tcpSocket健康检查

1、tcpsocket健康检查适用于TCP业务,通过向指定容器建立一个tcp连接,可以建立连接则健康检查正常,否则健康检查异常,依旧以nignx为例使用tcp健康检查机制,探测80端口的连通性

[ demo]# cat nginx-tcp-liveness.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-tcp-liveness-probe
  annotations:
    kubernetes.io/description: "nginx-tcp-liveness-probe"
spec:
  containers:
  - name: nginx-tcp-liveness-probe 
    image: nginx:latest
    ports:
    - name: http-80-port
      protocol: TCP
      containerPort: 80
    livenessProbe:  #健康检查为tcpSocket,探测TCP 80端口
      tcpSocket:
       port: 80
      initialDelaySeconds: 3
      periodSeconds: 10
      timeoutSeconds: 3

2、应用配置创建容器

[ demo]# kubectl apply -f nginx-tcp-liveness.yaml 
pod/nginx-tcp-liveness-probe created

[ demo]# kubectl get pods nginx-tcp-liveness-probe 
NAME                       READY   STATUS    RESTARTS   AGE
nginx-tcp-liveness-probe   1/1     Running   0          6s

3、模拟故障,获取pod所属节点,登录到pod中,安装查看进程工具htop

获取pod所在node
[ demo]# kubectl get pods nginx-tcp-liveness-probe -o wide 
NAME                       READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
nginx-tcp-liveness-probe   1/1     Running   0          99s   10.244.2.20   node-3   <none>           <none>

登录到pod中
[ demo]# kubectl exec -it nginx-httpget-livess-readiness-probe /bin/bash

#执行apt-get update更新和apt-get install htop安装工具
:/# apt-get update      
Get:1 http://cdn-fastly.deb.debian.org/debian buster InRelease [122 kB]             
Get:2 http://security-cdn.debian.org/debian-security buster/updates InRelease [39.1 kB]     
Get:3 http://cdn-fastly.deb.debian.org/debian buster-updates InRelease [49.3 kB]            
Get:4 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [95.7 kB]
Get:5 http://cdn-fastly.deb.debian.org/debian buster/main amd64 Packages [7899 kB]
Get:6 http://cdn-fastly.deb.debian.org/debian buster-updates/main amd64 Packages [5792 B]
Fetched 8210 kB in 3s (3094 kB/s)
Reading package lists... Done
:/# apt-get install htop
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  lsof strace
The following NEW packages will be installed:
  htop
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 92.8 kB of archives.
After this operation, 230 kB of additional disk space will be used.
Get:1 http://cdn-fastly.deb.debian.org/debian buster/main amd64 htop amd64 2.2.0-1+b1 [92.8 kB]
Fetched 92.8 kB in 0s (221 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package htop.
(Reading database ... 7203 files and directories currently installed.)
Preparing to unpack .../htop_2.2.0-1+b1_amd64.deb ...
Unpacking htop (2.2.0-1+b1) ...
Setting up htop (2.2.0-1+b1) ...

4、运行htop查看进程,容器进程通常为1,kill掉进程观察容器状态,观察RESTART次数重启次数增加

:/# kill 1
:/# command terminated with exit code 137

查看pod情况
[ demo]# kubectl get pods nginx-tcp-liveness-probe 
NAME                       READY   STATUS    RESTARTS   AGE
nginx-tcp-liveness-probe   1/1     Running   1          13m

5、查看容器详情,发现容器有重启的记录

[ demo]# kubectl describe pods nginx-tcp-liveness-probe | tail
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age                From               Message
  ----    ------     ----               ----               -------
  Normal  Scheduled  14m                default-scheduler  Successfully assigned default/nginx-tcp-liveness-probe to node-3
  Normal  Pulling    44s (x2 over 14m)  kubelet, node-3    Pulling image "nginx:latest"
  Normal  Pulled     40s (x2 over 14m)  kubelet, node-3    Successfully pulled image "nginx:latest"
  Normal  Created    40s (x2 over 14m)  kubelet, node-3    Created container nginx-tcp-liveness-probe
  Normal  Started    40s (x2 over 14m)  kubelet, node-3    Started container nginx-tcp-liveness-probe

1.5 readiness健康就绪

就绪检查用于应用接入到service的场景,用于判断应用是否已经就绪完毕,即是否可以接受外部转发的流量,健康检查正常则将pod加入到service的endpoints中,健康检查异常则从service的endpoints中删除,避免影响业务的访问。

1、创建一个pod,使用httpGet的健康检查机制,定义readiness就绪检查探针检查路径/test.html

[ demo]# cat httpget-liveness-readiness-probe.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-tcp-liveness-probe
  annotations:
    kubernetes.io/description: "nginx-tcp-liveness-probe"
  labels:  #需要定义labels,后面定义的service需要调用
    app: nginx
spec:
  containers:
  - name: nginx-tcp-liveness-probe 
    image: nginx:latest
    ports:
    - name: http-80-port
      protocol: TCP
      containerPort: 80
    livenessProbe:  #存活检查探针
      httpGet:
        port: 80
        path: /index.html
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 10
      timeoutSeconds: 3
    readinessProbe:  #就绪检查探针
      httpGet:
        port: 80
        path: /test.html
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 10
      timeoutSeconds: 3

2、定义一个service,将上述的pod加入到service中,注意使用上述定义的labels,app=nginx

[ demo]# cat nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx-service 
spec:
  ports:
  - name: http
    port: 80 
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx 
  type: ClusterIP

3、生成配置

[ demo]# kubectl apply -f httpget-liveness-readiness-probe.yaml 
pod/nginx-tcp-liveness-probe created
[ demo]# kubectl apply -f nginx-service.yaml 
service/nginx-service created

4、此时pod状态正常,此时readiness健康检查异常

[ ~]# kubectl get pods nginx-httpget-livess-readiness-probe 
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-httpget-livess-readiness-probe   1/1     Running   2          153m

#readiness健康检查异常,404报错(最后一行)
[ demo]# kubectl describe pods nginx-tcp-liveness-probe | tail
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  2m6s                default-scheduler  Successfully assigned default/nginx-tcp-liveness-probe to node-3
  Normal   Pulling    2m5s                kubelet, node-3    Pulling image "nginx:latest"
  Normal   Pulled     2m1s                kubelet, node-3    Successfully pulled image "nginx:latest"
  Normal   Created    2m1s                kubelet, node-3    Created container nginx-tcp-liveness-probe
  Normal   Started    2m1s                kubelet, node-3    Started container nginx-tcp-liveness-probe
  Warning  Unhealthy  2s (x12 over 112s)  kubelet, node-3    Readiness probe failed: HTTP probe failed with statuscode: 404

5、查看services的endpoints,发现此时endpoints为空,因为readiness就绪检查异常,kubelet认为此时pod并未就绪,因此并未将其加入到endpoints中。

[ ~]# kubectl describe services nginx-service 
Name:              nginx-service
Namespace:         default
Labels:            app=nginx
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-service","namespace":"default"},"s...
Selector:          app=nginx
Type:              ClusterIP
IP:                10.110.54.40
Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         <none> #Endpoints对象为空
Session Affinity:  None
Events:            <none>

#endpoints状态
[ demo]# kubectl describe endpoints nginx-service 
Name:         nginx-service
Namespace:    default
Labels:       app=nginx
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2019-09-30T14:27:37Z
Subsets:
  Addresses:          <none>
  NotReadyAddresses:  10.244.2.22  #pod处于NotReady状态
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    http  80    TCP

Events:  <none>

6、进入到pod中手动创建网站文件,使readiness健康检查正常

[ ~]# kubectl exec -it nginx-httpget-livess-readiness-probe /bin/bash
:/# echo "readiness probe demo" >/usr/share/nginx/html/test.html

7、此时readiness健康检查正常,kubelet检测到pod就绪会将其加入到endpoints中

健康检查正常
[ demo]# curl http://10.244.2.22/test.html

查看endpoints情况
readines[ demo]# kubectl describe endpoints nginx-service 
Name:         nginx-service
Namespace:    default
Labels:       app=nginx
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2019-09-30T14:33:01Z
Subsets:
  Addresses:          10.244.2.22 #就绪地址,已从NotReady中提出,加入到正常的Address列表中
  NotReadyAddresses:  <none>
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    http  80    TCP

查看service状态
[ demo]# kubectl describe services nginx-service 
Name:              nginx-service
Namespace:         default
Labels:            app=nginx
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-service","namespace":"default"},"s...
Selector:          app=nginx
Type:              ClusterIP
IP:                10.110.54.40
Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.22:80 #已和endpoints关联
Session Affinity:  None
Events:            <none>

8、同理,如果此时容器的健康检查异常,kubelet会自动将其动endpoint中

删除站点信息,使健康检查异常
[ demo]# kubectl exec -it nginx-tcp-liveness-probe  /bin/bash
:/# rm -f /usr/share/nginx/html/test.html 

查看pod健康检查event日志
[ demo]# kubectl get pods nginx-tcp-liveness-probe 
NAME                       READY   STATUS    RESTARTS   AGE
nginx-tcp-liveness-probe   0/1     Running   0          11m
[ demo]# kubectl describe pods nginx-tcp-liveness-probe | tail
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  12m                  default-scheduler  Successfully assigned default/nginx-tcp-liveness-probe to node-3
  Normal   Pulling    12m                  kubelet, node-3    Pulling image "nginx:latest"
  Normal   Pulled     11m                  kubelet, node-3    Successfully pulled image "nginx:latest"
  Normal   Created    11m                  kubelet, node-3    Created container nginx-tcp-liveness-probe
  Normal   Started    11m                  kubelet, node-3    Started container nginx-tcp-liveness-probe
  Warning  Unhealthy  119s (x32 over 11m)  kubelet, node-3    Readiness probe failed: HTTP probe failed with statuscode: 404

查看endpoints
[ demo]# kubectl describe endpoints nginx-service 
Name:         nginx-service
Namespace:    default
Labels:       app=nginx
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2019-09-30T14:38:01Z
Subsets:
  Addresses:          <none>
  NotReadyAddresses:  10.244.2.22 #健康检查异常,此时加入到NotReady状态
  Ports:
    Name  Port  Protocol
    ----  ----  --------
    http  80    TCP

Events:  <none>

查看service状态,此时endpoints为空
[ demo]# kubectl describe services nginx-service 
Name:              nginx-service
Namespace:         default
Labels:            app=nginx
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx-service","namespace":"default"},"s...
Selector:          app=nginx
Type:              ClusterIP
IP:                10.110.54.40
Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         #为空
Session Affinity:  None
Events:            <none>

写在最后

本章介绍kubernetes中健康检查两种Probe:livenessProbe和readinessProbe,livenessProbe主要用于存活检查,检查容器内部运行状态,readiness主要用于就绪检查,是否可以接受流量,通常需要和service的endpoints结合,当就绪准备妥当时加入到endpoints中,当就绪异常时从endpoints中删除,从而实现了services的健康检查和服务探测机制。对于Probe机制提供了三种检测的方法,分别适用于不同的场景:1. exec命令行,通过命令或shell实现健康检查,2. tcpSocket通过TCP协议探测端口,建立tcp连接,3. httpGet通过建立http请求探测,读者可多实操掌握其用法。

附录

健康检查:https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

TKE健康检查设置方法:https://cloud.tencent.com/document/product/457/32815

相关推荐