Modules in Terraform are a collection of resources that can be linked or just created individually. When we call a module, we are calling a file with a bunch of resources with some parameters inside, parameters we need either to specify in the resource itself or in another file which stores our variables.

Why do we use Terraform?

• It is a very useful tool to develop, change and have a versioning of our infrastructure in an efficient and safe way.
• It is a tool that allows you to define infrastructure as code and enables you to change and track the infrastructure with ease.
• It is independent of the platform or ‘cloud agnostic’ and allows working with multiple cloud providers.

What do we need for this lab?

• Terraform installed. In this case we run v.0.12.7.
• In our case, we are using AWS as our Cloud Provider, so we need a user with programmatic access. We need as well to give access to this user with the proper IAM Policy.
• Your preferred code editor.

Let’s get started!

Usually, first we should either set up a Terraform module with our required parameters or just get some official module from the network. In our case, I did myself a module with an S3 Bucket and its own policy and therefore, I specify what options I’d like the module to have.

It’d look something like this:

resource "aws_s3_bucket" "this" {
  bucket            = var.bucket_name
  acl               = var.type_acl
  force_destroy     = var.destroy

  tags = {
    Name            = var.tag_name
  }
  versioning {
    enabled         = var.versioning
  } 
}

resource "aws_s3_bucket_policy" "this" {
  bucket = var.bucket_name
  policy = templatefile("${path.module}/templates/s3_origin_access_identity.json", {
    origin_access_identity_arn         = aws_cloudfront_origin_access_identity.this.iam_arn,
    origin_access_identity_bucket_name = var.bucket_name
  })
}

As you may noticed, everything is “variablized”. Hence, we’ll need to create a file with our variables stored with their custom values.

In the same folder we have our file “main.tf”, we create our “variables.tf”:

### S3 VARIABLES

variable "bucket_name"      { }
variable "tag_name"         { }
variable "type_acl"         { }
variable "versioning"       { }
variable "destroy"          { }
variable "aliases"          { }
variable "certificate_arn"  { }

In this particular case, you may ask yourselves why these variables don’t have any value stored. And the answer is, we’re using a main file called “terragrunt.hcl” which will store all our values when we call the module.

Another thing that may be confusing is how to specfiy the bucket policy. We can either specifyit with a heredoc file (EOF) or straightaway with a function of Terraform called “templatefile”.

The first example is in Terraform Documentation, a website that I strongly recommend you to visit to understand how powerful modules and its attributes are.

The second example, and the one we used, it’d look something like this:

On the same folder, we must have a folder called “templates” and inside, introduce the bucket policy as a JSON file.

{
    "Version": "2012-10-17",
    "Id": "PolicyForCloudFrontPrivateS3Content",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "${origin_access_identity_arn}"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::${origin_access_identity_bucket_name}/*"
        }
    ]
  }

You may’ve noticed we introduced ${origin_access_identity_arn} and ${origin_access_identity_bucket_name} and this is because we want to set such values manually like this:

policy = templatefile("${path.module}/templates/s3_origin_access_identity.json", {
    origin_access_identity_arn         = aws_cloudfront_origin_access_identity.this.iam_arn,
    origin_access_identity_bucket_name = var.bucket_name
  })

As you see, we introduce the “arn” calling another resource called “aws_cloudfront_origin_access_identity” and the bucket name we introduce it as we have seen before with our variables file.

Now you just need to specify your Terraform provider and to call the module with another file with the values you have specified and deploy it:

provider "aws" {
  region     = var.region
  version    = "~> 2.0"
}

provider "aws" {
  alias  = "aws_us_east_1"
  region = "us-east-1"
}
module "s3" {
    source               = "../../../../../../modules/s3-cloudfront/"
    bucket_name          = var.bucket_name
    tag_name             = var.tag_name
    type_acl             = var.type_acl
    versioning           = var.versioning
    destroy              = var.destroy
    aliases              = var.aliases
    certificate_arn      = var.certificate_arn    
}

Deploy it with a terraform init, terraform plan and terraform apply.

Conclusion

So, we saw how to deploy an S3 Bucket with its policy with Terraform. I strongly recommend you visit the official Terraform Documentation to keep learning about modules and all related to them.

I invite you all in case you need information about the DevOps world, to contact us and keep checking our blog to find other useful posts.

La entrada A quick view of modules in Terraform se publicó primero en Geko Cloud.

From time to time Google introduces new features and changes that sometimes also force the Terraform modules to upgrade themselves. It was our case at Geko, where we were using the GKE module for public-cluster deployment&management at version 5.x. A few days ago, when we planned to update some parameters it came that Google had removed the support for the Kubernetes dashboard. It was completely deprecated and the module was failing because of it, so we were forced to upgrade the module in order to meet the new conditions. There were up to 3 major version upgrades available, so we decided to go for it and use the latest one. However, it was not a standalone solution as it required to handle Terraform state’s incoherences.

The aim of this lab is to learn how to upgrade the official Terraform module intended to deploy&manage a public GKE cluster. We will specially deal with module’s (kubernetes-engine.beta-public-cluster) breaking changes, and we will manage to obtain the consistent status we previously had before the failure which preceded the upgrade.

Estimated time to finish this lab: ~20 minutes

1. Remove the previous resources

It’s strongly encouraged to perform a tfstate file backup before continue!

It’s especially important to remove all the conflicting resources from the Terraform state as soon as they are bounded between them using dependencies. The goal here is to remove any deprecated binding prior to importing them again from the current “picture” there’s already deployed.

The main components on a Kubernetes cluster are the networks (and subnetworks), the node pool and the cluster itself. Let’s focus on them.

terraform state rm module.gke.google_container_cluster.primary
terraform state rm module.gke.google_container_node_pool.pools[0]
terraform state rm module.vpc.google_compute_network.network
terraform state rm module.vpc.google_compute_subnetwork.subnetwork[0]
terraform state rm module.vpc.google_compute_subnetwork.subnetwork[1]

2. Upgrade versions

Once removed the previous states the next step is to set the version for the required modules to the current latest version. For the GKE module the latest now it’s 8.1.0, but it will be allowed to automatically adopt minor upgrades (“~>”).

Upgrade the GKE cluster module

 module "gke" {
   source  = "terraform-google-modules/kubernetes-engine/google//modules/beta-public-cluster"
-  version = "~> 5.0"
+  version = "~> 8.1"

Upgrade the VPC module

 module "vpc" {
-  source  = "github.com/terraform-google-modules/terraform-google-network?ref=v1.1.0"
+  source  = "github.com/terraform-google-modules/terraform-google-network?ref=v2.3.0"

Check the new resources

In order to find out if the new resources have experienced a name change (due to the modules upgrade), a Terraform plan is strongly encouraged.

On this case it has been found that some module’s internal hierarchy and also list’s indexes have changed.

  module.gke.google_container_cluster.primary
  
- module.gke.google_container_node_pool.pools[0]
+ module.gke.google_container_node_pool.pools["default-node-pool"]

  module.vpc.google_compute_network.network
  
- module.vpc.google_compute_subnetwork.subnetwork[0]
+ module.vpc.module.subnets.google_compute_subnetwork.subnetwork["southamerica-east1/my-cluster-public"]

- module.vpc.google_compute_subnetwork.subnetwork[1]
+ module.vpc.module.subnets.google_compute_subnetwork.subnetwork["southamerica-east1/my-cluster-private"]

3. Import fresh resources

Keep in mind that the zone/region depends on your kind of cluster. If it’s zonal you must use the master zone (e.g. southamerica-east1-a). On the other hand, if it’s a regional cluster you must use the region (e.g. southamerica-east1). The following example assumes a regional cluster located at southamerica-east1, in the project “my-project“, and with a cluster name “my-cluster“. The network names were set accordingly to the cluster’s name, just adding the suffixes “private” and “public” to the subnets to properly differentiate them.

Note also the new module hierarchy and indexing.

# Global vars
REGION="southamerica-east1"
PROJECT="my-project"
CLUSTER="my-cluster"

# Cluster
CLUSTER_LOCAL="module.gke.google_container_cluster.primary"
CLUSTER_REMOTE="${PROJECT}/${REGION}/${CLUSTER}"
terraform import $CLUSTER_LOCAL $CLUSTER_REMOTE

# Node pool
POOL_LOCAL="module.gke.google_container_node_pool.pools["default-node-pool"]"
POOL_REMOTE="${CLUSTER_REMOTE}/default-node-pool"
terraform import $POOL_LOCAL $POOL_REMOTE

# Subnetworks
BASE_SUBNET_LOCAL="module.vpc.module.subnets.google_compute_subnetwork.subnetwork"

## Public
PUBLIC_SUBNET_LOCAL="${BASE_SUBNET_LOCAL}["${REGION}/${CLUSTER}-public"]"
PUBLIC_SUBNET_REMOTE="${CLUSTER_REMOTE}-public"
terraform import $PUBLIC_SUBNET_LOCAL $PUBLIC_SUBNET_REMOTE

## Private
PRIVATE_SUBNET_LOCAL="${BASE_SUBNET_LOCAL}["${REGION}/${CLUSTER}-private"]"
PRIVATE_SUBNET_REMOTE="${CLUSTER_REMOTE}-private"
terraform import $PRIVATE_SUBNET_LOCAL $PRIVATE_SUBNET_REMOTE

# Network
NETWORK_LOCAL="module.vpc.module.vpc.google_compute_network.network"
NETWORK_REMOTE="${PROJECT}/${CLUSTER}"
terraform import $NETWORK_LOCAL $NETWORK_REMOTE

 4. Update parameters

It’s very likely you will encounter that after a Terraform plan the google_container_cluster resource still needs to be updated due to a subnetwork parameter change. The new subnet keys have made the indexes to change their order. Just edit your GKE module to replace the subnetwork parameter as below.

- subnetwork = module.vpc.subnets_names[0]
+ subnetwork = module.vpc.subnets_names[1]

Conclusion

As you may have read above, sometimes -when relying on third parties- could happen that a breaking change is introduced and you get yourself into troubles to get the service back again. Beside this, the solution could introduce collateral damages which will require additional sub-solutions. On this particular case regarding Terraform, dealing with inconsistent states is not really common nor recommended, but it comes that is the only method you have available to solve them on your tool-set.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada Upgrade GKE public-cluster’s Terraform module se publicó primero en Geko Cloud.