It all started with a simple message: “Hello Geko, we are receiving a DB connection timeout“. It took less than 2 minutes to discover what was going on: The database was deleted. Lots of facepalms after, the main task was to recover the data (and also the service). Fortunately, Google enforces the backup policies to always copy the database once per day. At Geko, we all agree with this policy. So the thing is we went to look for the backups and the nightmare began. There were no backups! There was nothing! And then we found out backups are strongly bounded to the database resource, so if the database gets deleted the backups also do. Google states it so clear on their docs for Cloud SQL.

We are not going to go deeper on this post about how the database got deleted. Just tell it was an automated process which detected a disk size increase and when trying to go back to a previous, smaller size an entire database replacement was required. There was no opportunity to accept it nor stop it, so the situation was what it was:

• No database == No data
• No backups

What we did to resolve the situation

Even when Google was telling us the backups were deleted when the database also was, and all the facts were pointing to this hypothesis, we were still stubborn and we didn’t give up. The GCP web UI gave us no chances to recover, so we decided to continue digging by using the CLI (gcloud). And this was finally the key to our success!

We had the theory backups should be still somewhere even when the docs say they shouldn’t, so we checked different kinds of storage places until checking the SQL backups section. It comes that we were quick enough when checked this section, and we also knew the name of the deleted database, so we were able to run the next command.

$ gcloud sql backups list --instance=deleted-db-name --project our-project
ID             WINDOW_START_TIME              ERROR  STATUS
1614876500000  2021-03-04T04:00:00.000+00:00  -      SUCCESSFUL
1614765400000  2021-03-03T04:00:00.000+00:00  -      SUCCESSFUL
1614654300000  2021-03-02T04:00:00.000+00:00  -      SUCCESSFUL
1614543200000  2021-03-01T04:00:00.000+00:00  -      SUCCESSFUL
1614432100000  2021-02-28T04:00:00.000+00:00  -      SUCCESSFUL
1614321000000  2021-02-27T04:00:00.000+00:00  -      SUCCESSFUL
1614210000000  2021-02-26T04:00:00.000+00:00  -      SUCCESSFUL

And there they were still! After catching breath and getting our smiles back, the recovering process started. We still were not fully trusting this was going to work since maybe the backup list was there while no longer the data behind, but we had to give it a chance. We moved fast as we knew time was playing against us, so we immediately created a new database from scratch and just after we started the restoring process.

$ gcloud sql backups restore 1614876500000 --restore-instance=new-db-from-scratch-name --project our-project --backup-instance=deleted-db-name
All current data on the instance will be lost when the backup is 
restored.

Do you want to continue (Y/n)?  

Restoring Cloud SQL instance...done.                                                                                                                                                                                                        
Restored [https://sqladmin.googleapis.com/sql/v1beta4/projects/our-project/instances/new-db-from-scratch-name]

Finally, even when having positive feedback from GCP, we were still not fully believing this had worked. We needed to verify all the data was there, so we did. Fortunately again, everything was recovered. Our next and final step was to perform a SQL-dump in order to ensure we had a recent copy at another location.

Conclusion

Even after deeply searching on Google and not finding any helping results — as all of them say there’s nothing you can do — our knowledge and passion kept us continue digging on the topic until we found a way. We know we were lucky about finding the backups still there, but we know we also were quick, methodical, and obstinate when detecting and looking around to fix it. On the other hand, we have learned we cannot trust database backups performed by the provider, so we’re currently working on procedures to have backups in more places. We have pretty clear this is the first and the last time we are handling this topic.

Moreover, and as opposite to AWS, as we previously mentioned GCP strongly links the database resource to its backups. This has shown up it could be a huge problem when dealing with accidental deletion. Additionally, it’s not possible to copy SQL backups to any other kind of storage. Nevertheless, there are custom solutions consisting of regularly dumping the database and then storing it on a bucket, but it’s not something official.

On the other hand, we strongly advise being careful when having automated processes playing around. As we have seen, the only way to protect a GCP database against termination is to limit the permissions. So the way to proceed is to remove the DELETE permission from the (service) accounts the automated processes use.

Thankfully, you can always count on Geko team —a high-skilled engineering team— who will dig on the topic until getting it is easy and solved for you. Don’t forget to come back to the Geko’s blog and check out what’s new in here!

The Geko team will be always glad to see you back, and also you should contact us for further information!

La entrada GCP Cloud SQL – Recovering an accidentally deleted database se publicó primero en Geko Cloud.

It’s very likely you have been running a GKE cluster at the version v1.15 for many months without issues and then -suddenly- the logs stopped being received. You are using the default logging system the system provides, so you then check the GCP status website but everything is running fine. You also know you have not modified anything on the cluster before the log stop, but at the same time you guess it has to be something on your side (on our case because it’s not happening on some other projects we manage).

If the above story is familiar to you continue reading as this article will cover from how to detect it, to the solution that will make your cluster logs being received again.

1. Symptoms

1. 1. – The very first symptom you may notice is the sudden, complete absence of logs in Stackdriver, which should be coming -as usual- from your cluster’s pods.

1. – Beside the previous fact, you will additionally find out you have no Fluentd agents anymore in your cluster.

   $ kubectl get daemonset -n kube-system | grep -i fluentd
   
   

2. – On the other hand, all the nodes of your cluster have the label beta.kubernetes.io/fluentd-ds-ready=true, which is required for the Fluentd agents to know to which nodes should be deployed.

   $ kubectl get nodes
   NAME                           STATUS   ROLES    AGE     VERSION
   gke-cluster-node-93c020-7rjq   Ready    none     14h     v1.15.11-gke.15
   gke-cluster-node-93c020-9w22   Ready    none     23h     v1.15.11-gke.15
   gke-cluster-node-93c020-jdbt   Ready    none     47h     v1.15.11-gke.15
   gke-cluster-node-93c020-jvcl   Ready    none     3h17m   v1.15.11-gke.15
   
   $ kubectl describe node gke-cluster-node-93c020-7rjq | grep -i fluentd-ds-ready
          beta.kubernetes.io/fluentd-ds-ready=true
   
   $ # Do the same for the remaining nodes in order to ensure all of them are properly labeled

3. – Your GKE cluster is currently at the version v1.15 or greater.

2. Diagnose

What is currently happening is GCP forcibly deprecating the legacy monitoring/logging system. The replacement is called Cloud Operations for GKE which (for our use case) does basically the same. Once said that, keep in mind there are a few differences you should take care about when searching (such as metric names changes), and that you will find them all on the migration guide.

3. Treatment

Label the cluster nodes

If you found on previous steps that your cluster nodes were not labeled, do it now!

$ kubectl label node $NODE_NAME beta.kubernetes.io/fluentd-ds-ready=true

Get your cluster’s name

List the available clusters.

$ gcloud container clusters list
NAME        LOCATION        MASTER_VERSION  MASTER_IP    MACHINE_TYPE   NODE_VERSION    NUM_NODES  STATUS
my-cluster  europe-east1-a  1.15.12-gke.2   34.35.36.37  n1-standard-2  1.15.11-gke.15  4          RUNNING

Disable the logging service for your cluster

Update your cluster’s config in order to set the logging service to none.

$ gcloud container clusters update my-cluster --logging-service none

Enable again the logging service for your cluster

Update your cluster’s config in order to set the logging service to the default one.

$ gcloud container clusters update my-cluster --logging-service logging.googleapis.com

WARNING: You will get an error telling you that enabling it is not possible due to the deprecation. This is the way we finally realized where the problem was. Just move on, you are on the right way.

Ensure the migration status is the expected one

Open the monitoring site at the GCP GUI and then go to Settings. You will find a tab named “Kubernetes Migration Status” which should look as follows.

Fully enable the logging&monitoring service for your cluster

Open the GKE clusters list and then click on the name of the cluster where you want to setup the new logging system. Once you see the cluster config, click on the EDIT button as is shown below.

Scroll down until finding the Kubernetes Engine Monitoring setting and then select the System and workload logging and monitoring option.

IMPORTANT: Don’t forget to save your changes at the bottom.

And that’s it! Your cluster will receive logs again as expected!

Conclusion

As you may probably know Google Cloud Platform is still in beta, so you will often find some changes breaking your already-working systems. You have two options at this point: On the one hand you could keep yourself up to date with all the future changes and deprecations. On the other hand you could just limit yourself to fix them as soon as they appear, but keep in mind you will always be blindly fighting against them.

There is a third option (which should be the first one), consisting on regularly coming to the Geko’s blog and check out if we have already dealt with the problem you are encountering. The Geko team will be always glad to see you here

Contact us for further information!

La entrada GCP/GKE – New logging system se publicó primero en Geko Cloud.

From time to time Google introduces new features and changes that sometimes also force the Terraform modules to upgrade themselves. It was our case at Geko, where we were using the GKE module for public-cluster deployment&management at version 5.x. A few days ago, when we planned to update some parameters it came that Google had removed the support for the Kubernetes dashboard. It was completely deprecated and the module was failing because of it, so we were forced to upgrade the module in order to meet the new conditions. There were up to 3 major version upgrades available, so we decided to go for it and use the latest one. However, it was not a standalone solution as it required to handle Terraform state’s incoherences.

The aim of this lab is to learn how to upgrade the official Terraform module intended to deploy&manage a public GKE cluster. We will specially deal with module’s (kubernetes-engine.beta-public-cluster) breaking changes, and we will manage to obtain the consistent status we previously had before the failure which preceded the upgrade.

Estimated time to finish this lab: ~20 minutes

1. Remove the previous resources

It’s strongly encouraged to perform a tfstate file backup before continue!

It’s especially important to remove all the conflicting resources from the Terraform state as soon as they are bounded between them using dependencies. The goal here is to remove any deprecated binding prior to importing them again from the current “picture” there’s already deployed.

The main components on a Kubernetes cluster are the networks (and subnetworks), the node pool and the cluster itself. Let’s focus on them.

terraform state rm module.gke.google_container_cluster.primary
terraform state rm module.gke.google_container_node_pool.pools[0]
terraform state rm module.vpc.google_compute_network.network
terraform state rm module.vpc.google_compute_subnetwork.subnetwork[0]
terraform state rm module.vpc.google_compute_subnetwork.subnetwork[1]

2. Upgrade versions

Once removed the previous states the next step is to set the version for the required modules to the current latest version. For the GKE module the latest now it’s 8.1.0, but it will be allowed to automatically adopt minor upgrades (“~>”).

Upgrade the GKE cluster module

 module "gke" {
   source  = "terraform-google-modules/kubernetes-engine/google//modules/beta-public-cluster"
-  version = "~> 5.0"
+  version = "~> 8.1"

Upgrade the VPC module

 module "vpc" {
-  source  = "github.com/terraform-google-modules/terraform-google-network?ref=v1.1.0"
+  source  = "github.com/terraform-google-modules/terraform-google-network?ref=v2.3.0"

Check the new resources

In order to find out if the new resources have experienced a name change (due to the modules upgrade), a Terraform plan is strongly encouraged.

On this case it has been found that some module’s internal hierarchy and also list’s indexes have changed.

  module.gke.google_container_cluster.primary
  
- module.gke.google_container_node_pool.pools[0]
+ module.gke.google_container_node_pool.pools["default-node-pool"]

  module.vpc.google_compute_network.network
  
- module.vpc.google_compute_subnetwork.subnetwork[0]
+ module.vpc.module.subnets.google_compute_subnetwork.subnetwork["southamerica-east1/my-cluster-public"]

- module.vpc.google_compute_subnetwork.subnetwork[1]
+ module.vpc.module.subnets.google_compute_subnetwork.subnetwork["southamerica-east1/my-cluster-private"]

3. Import fresh resources

Keep in mind that the zone/region depends on your kind of cluster. If it’s zonal you must use the master zone (e.g. southamerica-east1-a). On the other hand, if it’s a regional cluster you must use the region (e.g. southamerica-east1). The following example assumes a regional cluster located at southamerica-east1, in the project “my-project“, and with a cluster name “my-cluster“. The network names were set accordingly to the cluster’s name, just adding the suffixes “private” and “public” to the subnets to properly differentiate them.

Note also the new module hierarchy and indexing.

# Global vars
REGION="southamerica-east1"
PROJECT="my-project"
CLUSTER="my-cluster"

# Cluster
CLUSTER_LOCAL="module.gke.google_container_cluster.primary"
CLUSTER_REMOTE="${PROJECT}/${REGION}/${CLUSTER}"
terraform import $CLUSTER_LOCAL $CLUSTER_REMOTE

# Node pool
POOL_LOCAL="module.gke.google_container_node_pool.pools["default-node-pool"]"
POOL_REMOTE="${CLUSTER_REMOTE}/default-node-pool"
terraform import $POOL_LOCAL $POOL_REMOTE

# Subnetworks
BASE_SUBNET_LOCAL="module.vpc.module.subnets.google_compute_subnetwork.subnetwork"

## Public
PUBLIC_SUBNET_LOCAL="${BASE_SUBNET_LOCAL}["${REGION}/${CLUSTER}-public"]"
PUBLIC_SUBNET_REMOTE="${CLUSTER_REMOTE}-public"
terraform import $PUBLIC_SUBNET_LOCAL $PUBLIC_SUBNET_REMOTE

## Private
PRIVATE_SUBNET_LOCAL="${BASE_SUBNET_LOCAL}["${REGION}/${CLUSTER}-private"]"
PRIVATE_SUBNET_REMOTE="${CLUSTER_REMOTE}-private"
terraform import $PRIVATE_SUBNET_LOCAL $PRIVATE_SUBNET_REMOTE

# Network
NETWORK_LOCAL="module.vpc.module.vpc.google_compute_network.network"
NETWORK_REMOTE="${PROJECT}/${CLUSTER}"
terraform import $NETWORK_LOCAL $NETWORK_REMOTE

 4. Update parameters

It’s very likely you will encounter that after a Terraform plan the google_container_cluster resource still needs to be updated due to a subnetwork parameter change. The new subnet keys have made the indexes to change their order. Just edit your GKE module to replace the subnetwork parameter as below.

- subnetwork = module.vpc.subnets_names[0]
+ subnetwork = module.vpc.subnets_names[1]

Conclusion

As you may have read above, sometimes -when relying on third parties- could happen that a breaking change is introduced and you get yourself into troubles to get the service back again. Beside this, the solution could introduce collateral damages which will require additional sub-solutions. On this particular case regarding Terraform, dealing with inconsistent states is not really common nor recommended, but it comes that is the only method you have available to solve them on your tool-set.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada Upgrade GKE public-cluster’s Terraform module se publicó primero en Geko Cloud.

In Geko we decided to implement SSO with most of our internal services since we already work with Gsuite accounts, so it was the obvious way to go. Anyway you can keep the default user management functional if you still want to use it.

Using SSO also makes it a lot easier for us when there are new incorporations in our team since they can sign up straight forward and we just need to set roles for them afterwards. Even for our clients we can allow their domains to set up accounts for them, if needed.

Estimated time to finish this lab: 20-30 minutes

Setup Google Cloud account

To set up SSO we need to create some credentials in Google Cloud. If you don’t have a project created in GCP you will have to create one: https://console.cloud.google.com

Oauth consent screen

The consent screen is what users will see when they try to login to your application (Grafana). You can use this link to get to the configuration: https://console.cloud.google.com/apis/credentials/consent/edit?

Here we can set some parameters to customize our consent screen. We are going to set the following values for this lab:

• Application name: Grafana
• Support email: Your email or any mail associated with your account
• Authorized domains: These are the domains against which your users will be able to authenticate. You will usually use your organization domain name but you might be interested in allowing others too.

As you can see the configuration is pretty straight forward. If you want to set any other parameter feel free, it’s up to you.

Creating credentials

We will now set up the credentials that will be used in Grafana to authenticate against our Google Cloud project: https://console.cloud.google.com/apis/credentials?

• Click on “CREATE CREDENTIALS” and then “Oauth Client ID”.
• Select “Web application”, set the name to “Grafana“ and click “Create”

Next, you will see a popup window with the following information:

Warning! Store your Client ID and Client Secret in a secure location.

Setup Grafana

I assume you already have setup docker service in your local machine. If you have not, get started here: https://docs.docker.com/get-docker/

Persistent volume

Even if we do use docker we want to have persistent data so any modification in configuration or dashboards will not get lost even if we kill the container. This makes upgrades to newer versions really easy and less painfully. So go ahead and create a volume for Grafana:

docker volume create grafana-data

Prepare the arguments to setup Grafana

We are going to spin up the Grafana container using some environment variables to setup Grafana. Here we will describe what is each one for:

• GF_SECURITY_ADMIN_PASSWORD: Strong random password
• GF_SERVER_ROOT_URL: Set this if you want to override the server root. Otherwise you can remove this parameter. Useful if you run Grafana behind a reverse proxy (for example nginx) and need to access a specific uri.
• GF_AUTH_GOOGLE_ENABLED: Enable Google SSO
• GF_AUTH_GOOGLE_AUTH_URL: Self explanatory
• GF_AUTH_GOOGLE_TOKEN_URL: Self explanatory
• GF_AUTH_GOOGLE_CLIENT_SECRET: Client Secret that we obtained when we created the credentials in GCP
• GF_AUTH_GOOGLE_CLIENT_ID: Client ID that we obtained when we created the credentials in GCP
• GF_ALLOWED_DOMAINS: Your company domain and every other domain you want to grant access to (for example your clients)

We can of course set all these parameters in grafana.ini instead of using environment vars, it’s your choice which method to use.

Start Grafana container with our custom arguments

We can now spin up our container. Note that we are mapping the Grafana port to our host 8081 port:

docker run -d --rm -p 8081:3000 --name grafana 
    -e "GF_SECURITY_ADMIN_PASSWORD=<some_password>" 
    -e "GF_SERVER_ROOT_URL=http://" 
    -e "GF_AUTH_BASIC_ENABLED=<disable_default_auth>" 
    -e "GF_AUTH_GOOGLE_ENABLED=<enable_google_auth>" 
    -e "GF_AUTH_GOOGLE_AUTH_URL=https://accounts.google.com/o/oauth2/auth" 
    -e "GF_AUTH_GOOGLE_TOKEN_URL=https://accounts.google.com/o/oauth2/token" 
    -e "GF_AUTH_GOOGLE_CLIENT_SECRET=<your_client_secret>" 
    -e "GF_AUTH_GOOGLE_CLIENT_ID=<your_client_id>" 
    -e “GF_ALLOWED_DOMAINS=<your_company_domain>”
    -v grafana-storage:/var/lib/grafana 
grafana/grafana

If everything is fine you should be able to access grafana at http://localhost:8081 and will see a button to authenticate with Google:

If you try to sign in with a gmail account that belongs to an allowed domain you should be able to access Grafana now.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful, such as “What is the kubernetes?“. Do not hesitate to contact us if you would like us to help you with your projects.

See you in the next post!

La entrada How to run Grafana in docker with Google SSO se publicó primero en Geko Cloud.

But sometimes there are some situations that the GCP Load Balancer doesn’t fit your needs or, simply you don’t want to use it… In those cases we have a very simple configuration using two well known pieces of software: HAProxy and Keepalived.

Keepalived uses Virtual Router Redundancy Protocol (VRRP) and floating IP addresses that can ‘move’ from one VM to another in case of one of them becomes unavailable. In a classic and on-prem environment this is something like that:

In case of a server failure, when the other server takes over the floating IP addresses, it adds these addresses to its network interface. The server announces this takeover to other devices using Layer 2 by sending a gratuitous Address Resolution Protocol (ARP) frame. Google Compute Engine uses a virtualized network stack and typical implementation mechanisms don’t work here. The VPC network handles ARP requests based on the configured routing topology, and ignores gratuitous ARP frames.

First of all, you need to install haproxy and keepalived in your server. In this case, we used Ubuntu 18.04. Just run:

sudo apt install haproxy 
sudo apt install keepalived

For this tutorial we will configure an internal load balancer but you can also configure an external load balancer with some small modifications.

We’ll use this /etc/haproxy/haproxy.cfg file in both servers. Both servers (MASTER and BACKUP) need to have exactly the same HAProxy configuration.

global
        log /dev/log    local0 debug
        log /dev/log    local1 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

defaults
        log     global
        mode    http
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend service-1
  bind 192.168.240.140:443
  mode tcp
  option tcplog
  log global
  default_backend service-1-be
backend service-1-be
  mode tcp
  server service-1-server-1 10.152.0.220:443 check id 1
  server service-1-server-2 10.152.0.221:443 check id 2


frontend service-2
  bind 192.168.240.141:443
  mode tcp
  option tcplog
  log global
  default_backend service-2-be
backend service-2-be
  mode tcp
  server service-2-server-1 10.10.0.40:443 check id 1
  server service-2-server-2 10.10.0.41:443 check id 2

In this example, the two floating IPs will be: 192.168.240.140 and 192.168.240.141.

Don’t forget to restart the haproxy service(sudo systemctl restart haproxy) every time you change the config file.

For the /etc/keepalived/keepalived.conf configuration file we will use these two config files in the master and slave VMs:

MASTER

vrrp_instance floating_ip {
    state MASTER
    interface ens4
    unicast_src_ip 192.168.240.10
    unicast_peer {
        192.168.240.11
    }

    virtual_router_id 50
    priority 100
    authentication {
        auth_type PASS
        auth_pass your_passwd
    }
    notify_master /etc/keepalived/takeover.sh root
}

BACKUP

vrrp_instance floating_ip {
    state BACKUP
    interface ens4
    unicast_src_ip 192.168.240.11
    unicast_peer {
        192.168.240.10
    }

    virtual_router_id 50
    priority 50
    authentication {
        auth_type PASS
        auth_pass your_passwd
    }
    notify_master /etc/keepalived/takeover.sh root
}

Don’t forget to restart the keepalived service (sudo systemctl restart keepalived) every time you change the config file.

And now the part that does the ‘magic’: the /etc/keepalived/takeover.sh script. Basically, this script unassigns IP aliases from the peer that is down and assigns them to himself. After that, haproxy service is reloaded to allow haproxy to bind to the floating IPs.

MASTER

# Unassign peer's IP aliases. Try it until it's possible.
until gcloud compute instances network-interfaces update haproxy-prod-backup --zone europe-west1-c --aliases "" > /etc/keepalived/takeover.log 2>&1; do
    echo "Instance not accessible during takeover. Retrying in 5 seconds..."
    sleep 5
done

# Assign IP aliases to me because now I am the MASTER!
gcloud compute instances network-interfaces update haproxy-prod 
    --zone europe-west1-b 
    --aliases "192.168.240.140/32;192.168.240.141/32" >> /etc/keepalived/takeover.log 2>&1
systemctl restart haproxy
echo "I became the MASTER at: $(date)" >> /etc/keepalived/takeover.log

BACKUP

# Unassign peer's IP aliases. Try it until it's possible.
until gcloud compute instances network-interfaces update haproxy-prod --zone europe-west1-b --aliases "" > /etc/keepalived/takeover.log 2>&1; do
    echo "Instance not accessible during takeover. Retrying in 5 seconds..."
    sleep 5
done

# Assign IP aliases to me because now I am the MASTER!
gcloud compute instances network-interfaces update haproxy-prod-backup 
    --zone europe-west1-c 
    --aliases "192.168.240.140/32;192.168.240.141/32" >> /etc/keepalived/takeover.log 2>&1
systemctl restart haproxy
echo "I became the MASTER at: $(date)" >> /etc/keepalived/takeover.log

In fact, all that we do is assign/unassing IP alias depending on what VM is the MASTER.

Easy, isn’t it?

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful, such as “What is the cloud?“. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada How to set up a High-Availability HAProxy in Google Cloud with Keepalived se publicó primero en Geko Cloud.

Introduction

We worked in a project that required a nginx server to be able to whitelist some public ip addresses while denying all other connections. While this can be addressed using GCP firewall rules there were some other reasons why it was necessary to be done through nginx configuration instead of using GCP rules.

The problem is that nginx was not showing the correct real ip address of the request in the REMOTE_ADDR header.

130.211.0.230 - - [23/Jan/2020:09:44:51 +0000] "GET / HTTP/1.1" 404 "108.26.106.168, 36.129.221.25"

Scenario

I’ll be using a very simple scenario:

• Google Compute private instance running two docker containers: nginx and a php based app
• Public GCP Load Balancer targeting the nginx container.

Fake IP addresses used in this post:

• User originating requests: 108.26.106.168
• GCP Load Balancer public IP: 36.129.221.25
• GCP Load Balancer private ranges: 130.211.0.0/22 and 35.191.0.0/16

That’s it.

The problem

Let’s quickly review how a Load Balancer works: When a request is received from a remote client it is terminated and a new request is issued from the LB against the backend, forwarding a set of headers (click on GCP and AWS for specific details) that are then catched by the underlying service.

When you place a nginx server behind the LB you receive the Load Balancer’s private IP as remote address instead of the user’s real public ip.

If we take a look at nginx logs we can see this:

130.211.0.130 - - [23/Jan/2020:09:02:51 +0000] "GET / HTTP/1.1" 200 "108.26.106.168, 36.129.221.25"

Wait! The user IP address appears in that chain at the end of the log. Why the heck is nginx taking the private range of the LB as the origin ip address? Well, this is because the LB is actually doing a new request.

How did we solve the issue

The first IP in the log comes from the REMOTE_ADDR header. We need to replace the value of this header with the real ip address received in the X-Forwarded-For header.

But there’s something else we need to deal with this second header: It actually comes not only with the user real IP but with the Load Balancer public address too.

In order to solve all this we will use the real_ip module. We are going to apply the following configuration in nginx.conf inside the “server” block:

set_real_ip_from 36.129.221.25/32; // LB Public IP address
set_real_ip_from 130.211.0.0/22; // Private IP range for GCP Load Balancers
set_real_ip_from 35.191.0.0/16; //Private IP range for GCP Load Balancers
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Let’s split it down:

• set_real_ip_from: Tell nginx to trust GCP LB public and private IPs.
• real_ip_header: Replace REMOTE_ADDR header with the values from X-Forwarded-For.
• real_ip_recursive on: Filter out the trusted ips from the chain, therefore the last non-trusted address in the chain will be used as the remote address.

Finally test it out:

108.26.106.168 - - [23/Jan/2020:09:44:51 +0000] "GET / HTTP/1.1" 200 "108.26.106.168, 36.129.221.25"

Conclusion

As we can see the solution was pretty straight forward, but still it took some time diving into documentation to understand how the Load Balancer was forwarding the headers and how we can tune nginx to rewrite the headers in order to achieve the expected behaviour.

Nevertheless it was a good learning experience.

Sources: https://nginx.org/en/docs/http/ngx_http_realip_module.html

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful, such as “What is the cloud?“. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada Forward real IP to a NGINX behind a GCP Load Balancer se publicó primero en Geko Cloud.