Jenkins Kubernetes Pod Template
Last friday we faced with a non very common issue with a customer that use jenkins and kubernetes for their CICD pipelines…
Some of their Jenkins pipelines freezed because the Kubernetes nodes which runs those executions still pending/offline for some reason and every console output shows us the following message:
[Pipeline] node Still waiting to schedule task All nodes of label ‘docker-build-xxxxxx’ are offline
Here I attach you a plugin link with very usefull information and configurations about the Kubernetes plugin (we strongly recommend you)
The Problem
We decided to check the Kubernetes cluster in order to inspect the pods execution inside the cicd namespaces to understand what is happening… then we notice that we are not able to talk with the Kubernates API thourgh the kubectl client, all the time we try to do some request to the API we get the following message:
Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Bingo! That’s the issue, our Kubernetes certificates we use to connect to the api (and used for etcd as well) has expired. This means a little problem because in order to renew a Kubernetes certificate you must to use a client to connect to the K8S API, but as I told you, the client doesn’t work because the certificates already expired… sooooo
What we gonna do?
Geko to the rescue!
There is a workaround to solve it, basically we must to fake our Kubernetes by destroying (moving) the current certificates manually and then force re-create it using init config into the whole cluster.
We strongly suggest you to previously move all the old certificates in a temporary folder, then force init config and finally reboot the K8S to startup with the new certificates we will use in our client side.
$ cd /etc/kubernetes/pki/
$ mkdir -p /tmp/oldcerts/etcd
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} /tmp/oldcerts
$ mv etcd/* /tmp/oldcerts/etcd
$ kubeadm init phase certs all --apiserver-advertise-address
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/oldcerts
$ kubeadm init phase kubeconfig all
$ reboot
After reboot , our cluster will startup using the new certificates and will create a new .kubeconfig file which we will copy locally to use through our kubectl client.
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
Once we copy the new kubeconfig file , we are fully available to work with our K8S cluster again.
In our case we delete all the orphan pods previously generated inside the “cicd” namespace (we love awk 🙂 )
for docker in `kubectl get pods -n cicd | awk {'print $1'}`;do kubectl delete pod $docker -ncicd;done
We hope this post could help you with your Kubernetes expired certificates in order to restore the API connectivity with your clients.
If we can help you somehow don’t hesitate to contact us here.