一、kubernetes集群环境搭建

1.1.1 decker环境安装

1. decker环境安装

#在hdss-201 hdss-21 hdss-22 执行以下命令
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i ‘s+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+‘ /etc/yum.repos.d/docker-ce.repo
yum makecache fast
yum install docker-ce -y
yum install -y yum-utils device-mapper-persistent-data lvm2 (非必须)

1. 配置daemon.json加速(所有docker环境都要配置)

#hdss-201
[ /opt/certs]# mkdir -p /etc/docker
[ /opt/certs]# mkdir -p /data/docker
[ /opt/certs]# cat >>/etc/docker/daemon.json<<EOF
> {
>   "graph": "/date/docker",
>   "storage-driver": "overlay2",
>   "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
>   "registry-mirrors": ["https://2giwyi75.mirror.aliyuncs.com"],
>   "bip":"10.0.201.1/24",
>   "exec-opts":["native.cgroupdriver=systemd"],
>    "live-restore":true
> }
> EOF

[ /opt/certs]# cat /etc/docker/daemon.json
{
  "graph": "/date/docker",
  "storage-driver": "overlay2",
  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
  "registry-mirrors": ["https://2giwyi75.mirror.aliyuncs.com"],
  "bip":"10.0.201.1/24",
  "exec-opts":["native.cgroupdriver=systemd"],
   "live-restore":true
}

#hdss-22
[ ~]# mkdir -p /etc/docker
[ ~]# mkdir -p /data/docker
[ ~]# cat >>/etc/docker/daemon.json<<EOF
> {
>   "graph": "/date/docker",
>   "storage-driver": "overlay2",
>   "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
>   "registry-mirrors": ["https://2giwyi75.mirror.aliyuncs.com"],
>   "bip":"10.0.22.1/24",
>   "exec-opts":["native.cgroupdriver=systemd"],
>    "live-restore":true
> }
> EOF
[ ~]# cat /etc/docker/daemon.json 
{
  "graph": "/date/docker",
  "storage-driver": "overlay2",
  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
  "registry-mirrors": ["https://2giwyi75.mirror.aliyuncs.com"],
  "bip":"10.0.22.1/24",
  "exec-opts":["native.cgroupdriver=systemd"],
   "live-restore":true
}

#hdss-21
[ ~]# mkdir -p /etc/docker
[ ~]# mkdir -p /data/docker
[ ~]# cat >>/etc/docker/daemon.json<<EOF
> {
>   "graph": "/date/docker",
>   "storage-driver": "overlay2",
>   "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
>   "registry-mirrors": ["https://2giwyi75.mirror.aliyuncs.com"],
>   "bip":"10.0.21.1/24",
>   "exec-opts":["native.cgroupdriver=systemd"],
>    "live-restore":true
> }
> EOF
[ ~]# cat /etc/docker/daemon.json
{
  "graph": "/date/docker",
  "storage-driver": "overlay2",
  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
  "registry-mirrors": ["https://2giwyi75.mirror.aliyuncs.com"],
  "bip":"10.0.21.1/24",
  "exec-opts":["native.cgroupdriver=systemd"],
   "live-restore":true
}

1. 启动docker

#hdss-21
[ ~]# systemctl start docker
[ ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.  
[ ~]# docker info


#hdsss-22
[ ~]# systemctl start docker
[ ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[ ~]# docker info

#hdss-201
[ /opt/certs]# systemctl start docker
[ro /opt/certs]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[ /opt/certs]# docker info

二、部署docker镜像私有仓库harbor

2.1.1 部署docker镜像私有仓库harbor

1. 部署docker镜像私有仓库harbor （在hdss-201上部署）

harbor下载地址：https://github.com/goharbor/harbor/releases （下载版本要在1.7.6以上，因为不支持wget下载，手动下载后上传。 ）
[ ~]# mkdir /opt/src
[ ~]# cd /opt/src/
[ /opt/src]# ll
total 566428
-rw-r--r-- 1 root root 580021898 Apr 27 14:44 harbor-offline-installer-v1.8.3.tgz

2.解压harbor

[ /opt/src]# tar -zxf harbor-offline-installer-v1.8.3.tgz -C /opt
[ /opt/src]# ls /opt
certs  containerd  harbor  src

#重命令加版本做link方便版本升级
[ /opt/src]# ln -s /opt/harbor-v1.8.3/ /opt/harbor
[ /opt/src]# ll /opt/harbor
lrwxrwxrwx 1 root root 19 Jun 11 00:13 /opt/harbor -> /opt/harbor-v1.8.3/

1. 配置harbor.yml配置文件

[ /opt/src]# cd /opt/harbor
[ /opt/harbor]# ll
total 569632
-rw-r--r-- 1 root root 583269670 Sep 16  2019 harbor.v1.8.3.tar.gz
-rw-r--r-- 1 root root      4519 Sep 16  2019 harbor.yml
-rwxr-xr-x 1 root root      5088 Sep 16  2019 install.sh
-rw-r--r-- 1 root root     11347 Sep 16  2019 LICENSE
-rwxr-xr-x 1 root root      1654 Sep 16  2019 prepare

#修改如下
hostname: harbor.od.com  #主机域名
port: 180                #端口
harbor_admin_password: Harbor12345 #harbor密码（生产环境请自行更改）
data_volume: /data/harbor   
location: /var/log/harbor   #log日志存放地 默认是 /var/log/harbor 

[ /opt/harbor]# mkdir -p /data/harbor/logs

1. 安装单机编排工具 docker-compose

[ /opt/harbor]# yum install docker-compose -y
[ /opt/harbor]# rpm -qa docker-compose 
docker-compose-1.18.0-4.el7.noarch

1. 执行install.sh安装harbor

[ /opt/harbor]# ll
total 569632
-rw-r--r-- 1 root root 583269670 Sep 16  2019 harbor.v1.8.3.tar.gz
-rw-r--r-- 1 root root      4519 Jun 11 00:26 harbor.yml
-rwxr-xr-x 1 root root      5088 Sep 16  2019 install.sh
-rw-r--r-- 1 root root     11347 Sep 16  2019 LICENSE
-rwxr-xr-x 1 root root      1654 Sep 16  2019 prepare
[ /opt/harbor]# ./install.sh 
[Step 0]: checking installation environment ...

Note: docker version: 19.03.11

Note: docker-compose version: 1.18.0

[Step 1]: loading Harbor images ...
b80136ee24a4: Loading layer [==================================================>]  34.25MB/34.25MB
88c6d2bb8c65: Loading layer [==================================================>]  63.49MB/63.49MB
2d2b1a3edc7d: Loading layer [==================================================>]  53.06MB/53.06MB
8f9195383bd6: Loading layer [==================================================>]  6.656kB/6.656kB
5ca164c40e26: Loading layer [==================================================>]  2.048kB/2.048kB
.......。
? ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at http://harbor.od.com. 
For more details, please visit https://github.com/goharbor/harbor .


[ /opt/harbor]# docker-compose ps
      Name                     Command               State             Ports          
--------------------------------------------------------------------------------------
harbor-core         /harbor/start.sh                 Up                               
harbor-db           /entrypoint.sh postgres          Up      5432/tcp                 
harbor-jobservice   /harbor/start.sh                 Up                               
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up      80/tcp                   
nginx               nginx -g daemon off;             Up      0.0.0.0:180->80/tcp      
redis               docker-entrypoint.sh redis ...   Up      6379/tcp                 
registry            /entrypoint.sh /etc/regist ...   Up      5000/tcp                 
registryctl         /harbor/start.sh                 Up

5.安装nginx做反向代理(不要直接映射80端口)

[ /opt/harbor]# vim /etc/nginx/conf.d/harbor.od.com.conf
[ /opt/harbor]# cat /etc/nginx/conf.d/harbor.od.com.conf
server {
    listen       80;
    server_name  harbor.od.com;

    client_max_body_size 1000m;

    location / {
        proxy_pass http://127.0.0.1:180;
    }
}

[ /opt/harbor]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[ /opt/harbor]# ss -luntp|grep nginx
tcp    LISTEN     0      511       *:80                    *:*                   users:(("nginx",pid=10175,fd=6),("nginx",pid=10174,fd=6))
tcp    LISTEN     0      511    [::]:80                 [::]:*                   users:(("nginx",pid=10175,fd=7),("nginx",pid=10174,fd=7))

1. 添加harbor.od.com到DNS解析

[ ~]# vim /var/named/od.com.zone 
修改如下：
[ ~]# cat /var/named/od.com.zone 
$ORIGIN od.com.
$TTL 600	; 10 minutes
@   		IN SOA	dns.od.com. dnsadmin.od.com. (
				202061002 ; serial               #注意修改一次序列号+1
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
				NS   dns.od.com.
$TTL 60	; 1 minute
dns                A    10.0.0.11
harbor             A    10.0.0.201   #添加A记录

[ ~]# systemctl restart named
[ ~]# systemctl status named

[ ~]# dig -t A harbor.od.com +short
10.0.0.201

1. web访问
   
   登录创建项目
   
   
2. pull 一个nginx 镜像

[ /opt/harbor]# docker pull nginx:1.7.9
1.7.9: Pulling from library/nginx
Image docker.io/library/nginx:1.7.9 uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
a3ed95caeb02: Pull complete 
6f5424ebd796: Pull complete 
d15444df170a: Pull complete 
e83f073daa67: Pull complete 
a4d93e421023: Pull complete 
084adbca2647: Pull complete 
c9cec474c523: Pull complete 
Digest: sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451
Status: Downloaded newer image for nginx:1.7.9
docker.io/library/nginx:1.7.9

#打标签上传镜像
[ /opt/harbor]# docker tag nginx:1.7.9 harbor.od.com/public/nginx:v1.7.9
[ /opt/harbor]# docker images
REPOSITORY                      TAG                        IMAGE ID            CREATED             SIZE
goharbor/chartmuseum-photon     v0.9.0-v1.8.3              ec654bcf3624        8 months ago        131MB
goharbor/harbor-migrator        v1.8.3                     6f945bb96ea3        8 months ago        362MB
goharbor/redis-photon           v1.8.3                     cda8fa1932ec        8 months ago        109MB
goharbor/clair-photon           v2.0.8-v1.8.3              5630fa937f6d        8 months ago        165MB
goharbor/notary-server-photon   v0.6.1-v1.8.3              e0a54affd0c8        8 months ago        136MB
goharbor/notary-signer-photon   v0.6.1-v1.8.3              72708cdfb905        8 months ago        133MB
goharbor/harbor-registryctl     v1.8.3                     9dc783842a19        8 months ago        97.2MB
goharbor/registry-photon        v2.7.1-patch-2819-v1.8.3   a05e085842f5        8 months ago        82.3MB
goharbor/nginx-photon           v1.8.3                     3a016e0dc7de        8 months ago        37MB
goharbor/harbor-log             v1.8.3                     b92621c47043        8 months ago        82.6MB
goharbor/harbor-jobservice      v1.8.3                     53bc2359083f        8 months ago        120MB
goharbor/harbor-core            v1.8.3                     a3ccc3897bc0        8 months ago        136MB
goharbor/harbor-portal          v1.8.3                     514f2fb70e90        8 months ago        43.9MB
goharbor/harbor-db              v1.8.3                     d1b8adbed58f        8 months ago        147MB
goharbor/prepare                v1.8.3                     a37e777b7fe7        8 months ago        147MB
nginx                           1.7.9                      84581e99d807        5 years ago         91.7MB
harbor.od.com/public/nginx      v1.7.9                     84581e99d807        5 years ago         91.7MB

#上传镜像
[ ~]# docker login harbor.od.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[ ~]# docker push harbor.od.com/public/nginx:v1.7.9
The push refers to repository [harbor.od.com/public/nginx]
5f70bf18a086: Pushed 
4b26ab29a475: Pushed 
ccb1d68e3fb7: Pushed 
e387107e2065: Pushed 
63bf84221cce: Pushed 
e02dce553481: Pushed 
dea2e4984e29: Pushed 
v1.7.9: digest: sha256:b1f5935eb2e9e2ae89c0b3e2e148c19068d91ca502e857052f14db230443e4c2 size: 3012