概要
久しぶり(1年以上ぶり？)に、おうち k8s クラスターにアクセスしようとしたら、アクセスできませんでした。
$ kubectl get pod E1229 16:06:04.610736 7674 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.1.100:6443/api?timeout=32s\": EOF" E1229 16:06:14.861213 7674 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.1.100:6443/api?timeout=32s\": EOF" E1229 16:06:25.056394 7674 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.1.100:6443/api?timeout=32s\": EOF" E1229 16:06:36.059596 7674 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.1.100:6443/api?timeout=32s\": EOF"
対処します。
確認及び対処内容
全部のnode(system x3,user x3)は立ち上がっており、sshでログインできました。
ロードバランサーも正常なように見えます。
$ nc -v 192.168.1.100 6443 Connection to 192.168.1.100 6443 port [tcp/*] succeeded!
とすると、証明書関係のような気がします。
wurly@k8s-ctrl1:~$ sudo kubeadm certs check-expiration
[sudo] password for wurly:
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jun 22, 2025 02:35 UTC <invalid> ca no
apiserver Jun 22, 2025 02:35 UTC <invalid> ca no
apiserver-etcd-client Jun 22, 2025 02:35 UTC <invalid> etcd-ca no
apiserver-kubelet-client Jun 22, 2025 02:35 UTC <invalid> ca no
controller-manager.conf Jun 22, 2025 02:35 UTC <invalid> ca no
etcd-healthcheck-client Jun 22, 2025 02:35 UTC <invalid> etcd-ca no
etcd-peer Jun 22, 2025 02:35 UTC <invalid> etcd-ca no
etcd-server Jun 22, 2025 02:35 UTC <invalid> etcd-ca no
front-proxy-client Jun 22, 2025 02:35 UTC <invalid> front-proxy-ca no
scheduler.conf Jun 22, 2025 02:35 UTC <invalid> ca no
super-admin.conf Jun 22, 2025 02:35 UTC <invalid> ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jun 20, 2034 02:35 UTC 8y no
etcd-ca Jun 20, 2034 02:35 UTC 8y no
front-proxy-ca Jun 20, 2034 02:35 UTC 8y no
コントロールプレーンの証明書を更新します。
sudo kubeadm certs renew all
下記の通り、更新されました。
wurly@k8s-ctrl1:~$ sudo kubeadm certs renew all [sudo] password for wurly: [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [renew] Error reading configuration from the Cluster. Falling back to default configuration certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed certificate embedded in the kubeconfig file for the super-admin renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
wurly@k8s-ctrl1:~$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 29, 2026 07:56 UTC 364d ca no
apiserver Dec 29, 2026 07:56 UTC 364d ca no
apiserver-etcd-client Dec 29, 2026 07:56 UTC 364d etcd-ca no
apiserver-kubelet-client Dec 29, 2026 07:56 UTC 364d ca no
controller-manager.conf Dec 29, 2026 07:56 UTC 364d ca no
etcd-healthcheck-client Dec 29, 2026 07:56 UTC 364d etcd-ca no
etcd-peer Dec 29, 2026 07:56 UTC 364d etcd-ca no
etcd-server Dec 29, 2026 07:56 UTC 364d etcd-ca no
front-proxy-client Dec 29, 2026 07:56 UTC 364d front-proxy-ca no
scheduler.conf Dec 29, 2026 07:56 UTC 364d ca no
super-admin.conf Dec 29, 2026 07:56 UTC 364d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jun 20, 2034 02:35 UTC 8y no
etcd-ca Jun 20, 2034 02:35 UTC 8y no
front-proxy-ca Jun 20, 2034 02:35 UTC 8y no
sudo systemctl restart kubelet
更新された etc/kubernetes/admin.conf を作業PCの ~.kube/config にコピーする。 これで、kubectl は接続できるようになりました。
