K8S-证书过期问题

发表于 更新于 分类于 ，

K8S-证书过期问题

最近在开发环境中，在master使用kubectl get pods命令，发现报The connection to the server 172.24.2.69:6443 was refused - did you specify the right host or port?经过不断的百度和google终于把问题解决了，但网上的文章都很片面，没有具体说明处理的整个过程，下面说明一下我在整个解决问题的过程，我使用的k8s版本是1.14.1。

找到问题

1、执行systemctl status kubelet，看一下kuberntes是否启动正常，这个时候如果显示Unit kubelet.service entered failed state.句话，说明服务没有启动，有错误。

2、执行journalctl -xefu kubelet，查看kubernets中具体报错原因，在我这里显示的是Part of the existing bootstrap client certificate is expired: 2020-06-16 05:56:48 +0000 UTC，证书已经过期了。

3、使用以下命令，查看所有证书过期时间

1
for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;do openssl x509 -in $item -text -noout| grep Not;echo ======================$item===============;done

or 也可以单独查看某一个证书过期时间

1
openssl x509 -noout -dates -in /etc/kubernetes/pki/apiserver.crt

注意巨坑：Kuberntes里面有很多证书，其中有些证书失效时间是10年，有些证书是1年，我安装的正好到了时间，所以爆出了证书过期，现在看一下如何解决

解决证书问题

修改源码，调整过期时间

1、克隆源码，切换到对应的分支，并修改源码

1
2
3
[root@k8s-master ~/source]# git clone https://github.com/kubernetes/kubernetes.git
[root@k8s-master ~/source]# git checkout -b remotes/origin/release-1.14.1  v1.14.1
[root@k8s-master ~/source]# vi kubernetes/cmd/kubeadm/app/util/pkiutil/pki_helpers.go

将NotAfter: time.Now().Add(duration365d).UTC(),修改为NotAfter: time.Now().Add(duration365d * 10).UTC(),

2、下载go语言编译环境，并编译

1
2
3
4
[root@k8s-master ~/source]# docker run --rm -v {源码路径}:/go/src/k8s.io/kubernetes -it icyboy/k8s_build:v1.14.1 bash
root@8a61c0029fbf:/go# cd /go/src/k8s.io/kubernetes
root@8a61c0029fbf:/go/src/k8s.io/kubernetes# make all WHAT=cmd/kubeadm GOFLAGS=-v
root@8a61c0029fbf:/go/src/k8s.io/kubernetes# exit

3、替换本地的kubeadm文件

1
2
3
[root@k8s-master ~/source]# mv /usr/bin/kubeadm /usr/bin/kubeadm-bak
[root@k8s-master ~/source]# ln -s kubernetes/_output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm
[root@k8s-master ~/source]# ln -s kubernetes/_output/local/bin/linux/amd64/kubeadm /usr/local/bin/kubeadm

4、查看是否替换成功

1
2
3
4
5
6
7
[root@k8s-master ~/source]# kubeadm version

新的
kubeadm version: &version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.1-dirty", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"dirty", BuildDate:"2020-06-17T08:02:51Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

旧的
kubeadm version: &version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"dirty", BuildDate:"2020-06-17T08:02:51Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

创建新证书，重新载入

1、创建kubeadm.conf文件

1
2
3
4
5
6
[root@k8s-master /usr/local/bin]# touch kubeadm.conf

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.3			## 注意：这个是你k8s的版本号
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

2、生成证书

1
2
3
[root@k8s-master /usr/local/bin]# cp -r /etc/kubernetes /etc/kubernetes-bak
[root@k8s-master /usr/local/bin]# rm -rf /etc/kubernetes/pki
[root@k8s-master /usr/local/bin]# kubeadm alpha certs renew all --config=./kubeadm.conf

3、查看证书过期时间

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[root@k8s-master /usr/local/bin]# for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;do openssl x509 -in $item -text -noout| grep Not;echo ======================$item===============;done

===== /etc/kubernetes/pki/apiserver.crt =====
        Validity
            Not Before: Sep 22 04:24:14 2021 GMT
            Not After : Oct  5 02:34:45 2033 GMT
        Subject: CN=kube-apiserver
===== /etc/kubernetes/pki/apiserver-etcd-client.crt =====
        Validity
            Not Before: Sep 22 04:24:14 2021 GMT
            Not After : Oct  5 02:34:45 2033 GMT
        Subject: O=system:masters, CN=kube-apiserver-etcd-client
===== /etc/kubernetes/pki/apiserver-kubelet-client.crt =====
        Validity
            Not Before: Sep 22 04:24:14 2021 GMT
            Not After : Oct  5 02:34:45 2033 GMT
        Subject: O=system:masters, CN=kube-apiserver-kubelet-client
===== /etc/kubernetes/pki/ca.crt =====
        Validity
            Not Before: Sep 22 04:24:14 2021 GMT
            Not After : Sep 20 04:24:14 2031 GMT
        Subject: CN=kubernetes
===== /etc/kubernetes/pki/front-proxy-ca.crt =====
        Validity
            Not Before: Sep 22 04:24:15 2021 GMT
            Not After : Sep 20 04:24:15 2031 GMT
        Subject: CN=front-proxy-ca
===== /etc/kubernetes/pki/front-proxy-client.crt =====
        Validity
            Not Before: Sep 22 04:24:15 2021 GMT
            Not After : Oct  5 02:34:46 2033 GMT
        Subject: CN=front-proxy-client

4、重新载入

1
2
3
4
5
6
7
8
9
10
11
# 下面这几个文件，必须删了（后面会自动生成），不然会显示已存在，无法载入
[root@k8s-master /usr/local/bin]# rm -rf /etc/kubernetes/admin.conf
[root@k8s-master /usr/local/bin]# rm -rf /etc/kubernetes/kubelet.conf
[root@k8s-master /usr/local/bin]# rm -rf /etc/kubernetes/controller-manager.conf
[root@k8s-master /usr/local/bin]# rm -rf /etc/kubernetes/scheduler.conf

[root@k8s-master /usr/local/bin]# ./kubeadm init phase kubeconfig all --config=./kubeadm.conf
[root@k8s-master /usr/local/bin]# cd ~/.kube/
[root@k8s-master ~/.kube]# mv config config-back
[root@k8s-master ~/.kube]# cp /etc/kubernetes/admin.conf ./config
[root@k8s-master ~/.kube]# chown $(id -u):$(id -g) $HOME/.kube/config

5、重启kuberntes

1
2
systemctl restart docker
systemctl restart kubelet

每一个版本解决思路不太一样，其他版本可以参考文献：http://team.jiunile.com/blog/2018/12/k8s-kubeadm-ca-upgdate.html