If the above story is familiar to you continue reading as this article will cover from how to detect it, to the solution that will make your cluster logs being received again.<\/p>\n
1. Symptoms<\/h2>\n\n- \n
\n- – The very first symptom you may notice is the sudden, complete absence of logs in Stackdriver<\/a>, which should be coming -as usual- from your cluster’s pods.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
<\/p>\n
\n- – Beside the previous fact, you will additionally find out you have no Fluentd<\/a> agents anymore in your cluster.\n\n
$ kubectl get daemonset -n kube-system | grep -i fluentd\r\n\r\n<\/pre>\n<\/div>\n<\/li>\n
- – On the other hand, all the nodes of your cluster have the label beta.kubernetes.io\/fluentd-ds-ready=true<\/strong>, which is required for the Fluentd agents to know to which nodes should be deployed.\n\n
$ kubectl get nodes\r\nNAME STATUS ROLES AGE VERSION\r\ngke-cluster-node-93c020-7rjq Ready none 14h v1.15.11-gke.15\r\ngke-cluster-node-93c020-9w22 Ready none 23h v1.15.11-gke.15\r\ngke-cluster-node-93c020-jdbt Ready none 47h v1.15.11-gke.15\r\ngke-cluster-node-93c020-jvcl Ready none 3h17m v1.15.11-gke.15\r\n\r\n$ kubectl describe node gke-cluster-node-93c020-7rjq | grep -i fluentd-ds-ready\r\n beta.kubernetes.io\/fluentd-ds-ready=true\r\n\r\n$ # Do the same for the remaining nodes in order to ensure all of them are properly labeled<\/pre>\n<\/div>\n<\/li>\n
- – Your GKE cluster is currently at the version v1.15 or greater.<\/li>\n<\/ol>\n
2. Diagnose<\/h2>\n
What is currently happening is GCP forcibly deprecating the legacy monitoring\/logging system. The replacement is called Cloud Operations for GKE<\/a> which (for our use case) does basically the same. Once said that, keep in mind there are a few differences you should take care about when searching (such as metric names changes), and that you will find them all on the migration guide<\/a>.<\/p>\n3. Treatment<\/h2>\nLabel the cluster nodes<\/h3>\n
If you found on previous steps that your cluster nodes were not labeled, do it now!<\/p>\n
\n$ kubectl label node $NODE_NAME beta.kubernetes.io\/fluentd-ds-ready=true<\/pre>\n<\/div>\nGet your cluster’s name<\/h3>\n
List the available clusters.<\/p>\n
\n$ gcloud container clusters list\r\nNAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS\r\nmy-cluster europe-east1-a 1.15.12-gke.2 34.35.36.37 n1-standard-2 1.15.11-gke.15 4 RUNNING\r\n<\/pre>\n<\/div>\nDisable the logging service for your cluster<\/h3>\n
Update your cluster’s config in order to set the logging service to none.<\/em><\/p>\n\n$ gcloud container clusters update my-cluster --logging-service none<\/pre>\n<\/div>\nEnable again the logging service for your cluster<\/h3>\n
Update your cluster’s config in order to set the logging service to the default one.<\/p>\n
\n$ gcloud container clusters update my-cluster --logging-service logging.googleapis.com<\/pre>\n<\/div>\nWARNING:<\/strong> You will get an error telling you that enabling it is not possible due to the deprecation. This is the way we finally realized where the problem was. Just move on, you are on the right way.<\/p>\nEnsure the migration status is the expected one<\/h3>\n
Open the monitoring site at the GCP GUI<\/a> and then go to Settings. You will find a tab named “Kubernetes Migration Status” which should look as follows.<\/p>\n![\"\"](\"https:\/\/geko2.factoryfy.com\/wp-content\/uploads\/kubernetes-migration-dashboard.png\")
<\/p>\n
Fully enable the logging&monitoring service for your cluster<\/h3>\n
Open the GKE clusters list<\/a> and then click on the name of the cluster where you want to setup the new logging system. Once you see the cluster config, click on the EDIT button as is shown below.<\/p>\n![\"\"](\"https:\/\/geko2.factoryfy.com\/wp-content\/uploads\/edit-cluster.png\")
<\/p>\n
Scroll down until finding the Kubernetes Engine Monitoring<\/strong> setting and then select the System and workload logging and monitoring<\/strong> option.<\/span><\/span><\/p>\n![\"\"](\"https:\/\/geko2.factoryfy.com\/wp-content\/uploads\/monitoring-dropdown.png\")
<\/p>\n
IMPORTANT: <\/strong>Don’t forget to save your changes at the bottom.<\/p>\nAnd that’s it! Your cluster will receive logs again as expected!<\/h5>\n
<\/p>\n
Conclusion<\/h2>\n
As you may probably know Google Cloud Platform is still in beta, so you will often find some changes breaking your already-working systems. You have two options at this point: On the one hand you could keep yourself up to date with all the future changes and deprecations. On the other hand you could just limit yourself to fix them as soon as they appear, but keep in mind you will always be blindly fighting against them.<\/p>\n
There is a third option (which should be the first one), consisting on regularly coming to the Geko’s blog<\/a> and check out if we have already dealt with the problem you are encountering. The Geko team will be always glad to see you here \ud83d\ude42<\/p>\n
- \n
- – The very first symptom you may notice is the sudden, complete absence of logs in Stackdriver<\/a>, which should be coming -as usual- from your cluster’s pods.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
<\/p>\n
- \n
- – Beside the previous fact, you will additionally find out you have no Fluentd<\/a> agents anymore in your cluster.\n\n
$ kubectl get daemonset -n kube-system | grep -i fluentd\r\n\r\n<\/pre>\n<\/div>\n<\/li>\n
- – On the other hand, all the nodes of your cluster have the label beta.kubernetes.io\/fluentd-ds-ready=true<\/strong>, which is required for the Fluentd agents to know to which nodes should be deployed.\n
\n$ kubectl get nodes\r\nNAME STATUS ROLES AGE VERSION\r\ngke-cluster-node-93c020-7rjq Ready none 14h v1.15.11-gke.15\r\ngke-cluster-node-93c020-9w22 Ready none 23h v1.15.11-gke.15\r\ngke-cluster-node-93c020-jdbt Ready none 47h v1.15.11-gke.15\r\ngke-cluster-node-93c020-jvcl Ready none 3h17m v1.15.11-gke.15\r\n\r\n$ kubectl describe node gke-cluster-node-93c020-7rjq | grep -i fluentd-ds-ready\r\n beta.kubernetes.io\/fluentd-ds-ready=true\r\n\r\n$ # Do the same for the remaining nodes in order to ensure all of them are properly labeled<\/pre>\n<\/div>\n<\/li>\n
- – Your GKE cluster is currently at the version v1.15 or greater.<\/li>\n<\/ol>\n
2. Diagnose<\/h2>\n
What is currently happening is GCP forcibly deprecating the legacy monitoring\/logging system. The replacement is called Cloud Operations for GKE<\/a> which (for our use case) does basically the same. Once said that, keep in mind there are a few differences you should take care about when searching (such as metric names changes), and that you will find them all on the migration guide<\/a>.<\/p>\n
3. Treatment<\/h2>\n
Label the cluster nodes<\/h3>\n
If you found on previous steps that your cluster nodes were not labeled, do it now!<\/p>\n
\n$ kubectl label node $NODE_NAME beta.kubernetes.io\/fluentd-ds-ready=true<\/pre>\n<\/div>\n
Get your cluster’s name<\/h3>\n
List the available clusters.<\/p>\n
\n$ gcloud container clusters list\r\nNAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS\r\nmy-cluster europe-east1-a 1.15.12-gke.2 34.35.36.37 n1-standard-2 1.15.11-gke.15 4 RUNNING\r\n<\/pre>\n<\/div>\n
Disable the logging service for your cluster<\/h3>\n
Update your cluster’s config in order to set the logging service to none.<\/em><\/p>\n
\n$ gcloud container clusters update my-cluster --logging-service none<\/pre>\n<\/div>\n
Enable again the logging service for your cluster<\/h3>\n
Update your cluster’s config in order to set the logging service to the default one.<\/p>\n
\n$ gcloud container clusters update my-cluster --logging-service logging.googleapis.com<\/pre>\n<\/div>\n
WARNING:<\/strong> You will get an error telling you that enabling it is not possible due to the deprecation. This is the way we finally realized where the problem was. Just move on, you are on the right way.<\/p>\n
Ensure the migration status is the expected one<\/h3>\n
Open the monitoring site at the GCP GUI<\/a> and then go to Settings. You will find a tab named “Kubernetes Migration Status” which should look as follows.<\/p>\n
<\/p>\nFully enable the logging&monitoring service for your cluster<\/h3>\n
Open the GKE clusters list<\/a> and then click on the name of the cluster where you want to setup the new logging system. Once you see the cluster config, click on the EDIT button as is shown below.<\/p>\n
<\/p>\nScroll down until finding the Kubernetes Engine Monitoring<\/strong> setting and then select the System and workload logging and monitoring<\/strong> option.<\/span><\/span><\/p>\n
<\/p>\nIMPORTANT: <\/strong>Don’t forget to save your changes at the bottom.<\/p>\n
And that’s it! Your cluster will receive logs again as expected!<\/h5>\n
<\/p>\n
Conclusion<\/h2>\n
As you may probably know Google Cloud Platform is still in beta, so you will often find some changes breaking your already-working systems. You have two options at this point: On the one hand you could keep yourself up to date with all the future changes and deprecations. On the other hand you could just limit yourself to fix them as soon as they appear, but keep in mind you will always be blindly fighting against them.<\/p>\n
There is a third option (which should be the first one), consisting on regularly coming to the Geko’s blog<\/a> and check out if we have already dealt with the problem you are encountering. The Geko team will be always glad to see you here \ud83d\ude42<\/p>\n
- – On the other hand, all the nodes of your cluster have the label beta.kubernetes.io\/fluentd-ds-ready=true<\/strong>, which is required for the Fluentd agents to know to which nodes should be deployed.\n
- – Beside the previous fact, you will additionally find out you have no Fluentd<\/a> agents anymore in your cluster.\n