Provisionar nodos slave de jenkins dinámicamente gracias a aws spot fleet

A few weeks ago, a customer contacted us. His request was simple: What can we do to further optimize our automated CI/CD process in Jenkins?

After assesing the needs and the current infrastructure of the client, we decided to change the current paradigm in terms of task execution. Until now Jenkins executed the tasks in its main node (the machine where Jenkins is running), by changing this paradigm to one based on dynamic provisioning of subordinate nodes we would gain the following benefits:

• Cost savings – By moving all the computational effort away from the main node. It can be hosted on a lower performance machine, reducing the overall cost of our infrastructure. The subordinate nodes would only be running during the execution of a task, also saving money while no task is running.
• Security – As we will not execute any code on the main node, we ensure the system’s integrity even if the code has been compromised, and since it is only executed on an ephemeral instance the attacker will be isolated.
• Performance – The subordinate node exists only to execute the task and can dedicate all of the node’s resources to it.

Requirements

For this change, we will perform the following tasks:

• Spot Fleet creation in AWS
• Installation and configuration of the Jenkins plugin
• Jenkinsfile adaptation

For this publication we will make the following assumptions:

• We have an AWS account already set up
• We host Jenkins on an EC2 server
• We have knowledge of some basic Jenkins and AWS concepts

That said, let’s get down to business.

In this article on “Dynamic nodes in Jenkins via AWS Spot Fleet” we will develop all the necessary steps to obtain a good result.

Creation of launch template and spot fleet

An AWS Spot Fleet is nothing more than an AutoScaling Group of EC2 machines but with spot type instances – Amazon’s surplus hardware that nobody is using and that AWS sells cheaper than usual. The first step will be to create the Launch Template that the Spot Fleet will use:

The important items to be filled in will be:

1. Name and description
2. AMI to use – For example Ubuntu 22
3. Type of instance to use – Choose one according to your performance needs
4. Key Pair – We will need to add it later in Jenkins
5. Network – Use the same Subnet as Jenkins if possible
6. Security Group – For the connection between nodes to be established, it is important that we give access to port 22 from the main node. The easiest way is to create a rule that allows access either from the IP of the main node (if we have a Elastic IP assigned) or from its Security Group
7. Storage – Allocate the storage according to your needs
8. User data – For the Jenkins worker to run on the subordinate node, we will need to install JRE, we also installed Docker as a requirement:

#!/bin/bash
# Install docker
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
apt-get update
apt-get install -y docker-ce
usermod -aG docker ubuntu

# Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.21.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

#Install JRE
sudo apt install -y default-jre

Once created we will have a Launch template similar to this one:

Now it is time to create the Spot fleet as such:

In this case, the items to be filled in are:

1. Launch parameters – Select our Launch Template created above
2. Target Capacity – Set total to 0 for now, as it will be controlled by the Jenkins plugin. Enable Maintain Target Capacity
3. Network – Assign the VPC in which the main node is hosted
4. Instance type requirements – Choose the desired instance types in Manually select instance types, a larger quantity gives a better pool from which to start Spot instances
5. Allocation strategy – Capacity Optimized

And that’s it, we have created our Spot Fleet.

Installing and configuring the Jenkins plugin

Before installing the plugin, we will have to create and assign a new IAM role to the instance hosting the main node so that it will be able to manage our Spot fleet:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "ec2:DescribeSpotFleetInstances",
            "ec2:ModifySpotFleetRequest",
            "ec2:CreateTags",
            "ec2:DescribeRegions",
            "ec2:DescribeInstances",
            "ec2:TerminateInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeSpotFleetRequests"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:UpdateAutoScalingGroup"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:ListInstanceProfiles",
            "iam:ListRoles",
            "iam:PassRole"
         ],
         "Resource":"*"
      }
   ]
}

Once this is done, we will install the ec2-fleet-plugin and configure it from Manage Jenkins > Manage Nodes > Configure Clouds > Add New Cloud > Amazon EC2 Fleet.

1. Assign a name
2. AWS Credentials – We leave this blank, since our machine already has the role described above.
3. Region – Select the region in which we host our servers.
4. EC2 Fleet – It will only appear when we assign the region and the role is assigned to the machine correctly. Once filled in, we select our Spot fleet
5. Upload the Private key generated in the Launch template as SSH Username with private key
6. Private IP – Select if we want the main node to connect to the subordinate node using the private IP instead of the public one
7. Always Reconnect – We recommend not to enable it, as Spot instances are ephemeral by nature
8. Restrict Usage – Restrict the use of this Spot Fleet only to tasks that specify the assigned tag, regardless of the global Jenkins configuration
9. Label – The label that identifies this node cloud, useful for assigning which tasks will be executed in the Fleet
10. Number of executors – The number of simultaneous jobs that each Spot Fleet instance will be able to host
11. Max Idle Minutes Before Scaledown – The number of minutes Jenkins will keep an instance active with no pending tasks. If left at 0 Jenkins will never terminate instances, even if they are unused
12. Minimum/Maximum cluster Size – The minimum and maximum quantity of instances we want the Spot fleet to have. Ideally, the minimum should be set to 0 to reduce costs

Once the configuration is saved, we will have a new section in the Jenkins dashboard, which indicates the current status of our Spot Fleet.

Jenkinsfile adaptation

Now that we have our Spot fleet configured, it is time to make some small changes so Jenkins uses this new type of nod. If we want it to be used systematically on a global level, we can disable the execution of tasks on the main node (the only other node in our case) from the node configuration screen.

If, on the other hand, we want certain tasks to be executed in the subordinate nodes, we will have to specify them at Jenkinsfile level, here is an example of how to do it:

#!groovy
pipeline {
//Especificamos agente a nivel de Pipeline
  agent {
    label 'ec2-fleet'
  }

  stages {
        stage('Test') {
          agent {
            dockerfile {
              // Si usamos un agente en concreto, tendremos que volver a especificar la etiqueta
              label 'ec2-fleet'
              filename 'Dockerfile'
              dir 'docker/images/tests'
              args '-u root'
            }
          }
          steps {
            sh 'Hello World'
          }
        }
  }
}

With this last step, we are ready to execute any task on dynamically provisioned nodes within Spot Fleet instances.

From Geko Consulting Cloud, we hope you liked this post and above all that you find it useful, nothing would make us happier.
We invite you to contact us if you need information about the Cloud and DevOps world and keep checking our blog to find other useful publications.

La entrada Dynamic nodes in Jenkins via AWS Spot Fleet se publicó primero en Geko Cloud.

Recover accidentally deleted AWS elastic IP. It may not be the most common thing for the average AWS user but for those who work in the cloud industry it’s an everyday task to manage several AWS accounts. Either through a catalog of login links or through a list of account IDs (and their matching credentials), or through a role impersonation manager — the fact is that sooner or later it could happen that one account is mistakenly confused with another and an unwanted action is carried out.

That’s the case it’s being exposed today, in which some tasks were being done on a new account and the created infra there was disposable. It happened that by mistake this new account was mistook with the base account — the Geko’s one — where the elastic IP for the company’s bastion is hosted. Yes, indeed, that key point used to access other infrastructures in a secure way, and which these other infrastructures consider as a safe origin. In other words, that particular IP was used as one of the security measures when verifying the origin of the connections. An IP which was placed in many configurations, and which was not trivial to replace. Well, that IP was released by mistake. In AWS the word “release” is used to refer to the action of removing the IP from the account, leaving it available to the rest of the provider’s users.

It’s not necessary to describe the moments of maximum alert that followed the IP’s release but the possibility of having lost that address forever because it had been assigned to another user, began to take over the tension of the moment. However, there’s a reason to have crisis cabinets, and above all there’s a big importance to be able to count on a team composed by professionals in the area. Brainstorming was fogging up the windows when suddenly a lightning of hope made everything to stop.  There was still a chance. AWS had foreseen this catastrophic event and Geko found the way to recover what had always been its own but which it had never valued that much.

Has it happened to you? Would you like to know how it was solved? Keep reading and don’t miss out how this epic story ended!

Recover the AWS elastic IP which was accidentally deleted

Recover accidentally deleted AWS elastic IP. As it was previously stated, once an elastic IP is released it then belongs to the pool of available IPs for the users, so recover it could be impossible in the case it was reassigned. Nonetheless, AWS realized this situation could cause troubles so they developed a procedure to recover from this scenario. Despite this is not something you can do through the web console, recover the lost IP can be achieved by running just one command.

MY_IP=48.151.62.342
aws ec2 allocate-address --domain vpc --address $MY_IP

Yes, you need to know the IP address you want to recover but — if it’s critical enough to make you want to get it again — it should be something you should be able to retrieve.

What else could go wrong?

It happens that in this case there was an additional factor that could add more complexity to the situation, and in fact it did. At Geko — in order to have credentials for the AWS CLI — temporary keys are generated after going through a multi-factor verification process (which can also be revoked at any time), and these credentials are retrieved from the aforementioned bastion. Connecting the dots it can be concluded that the situation had fallen into cyclic dependency since to fix the problem a credentials set needed to be generated while the problem itself was preventing us to do so. Of course there were alternative methods to get out of the loop, but in the end it was enough to use some not-yet-expired credentials we were able to obtain.

Conclusion

Throughout this short — but intense — story, the importance of having a team around you has been shown. Keeping aside pride and rising the alarm was what made the solution to appear in a matter of a few minutes. As it’s commonly said — Two pair of eyes can see more than one.

On the other hand, we learned and reinforced a set of good practices which could be the key next time. They are the following ones.

1. Ensure you have some emergency credentials in case something happens in the bastion.
2. Ensure the account you’re trying to modify is the one you actually want to modify. It’s better to spend a little bit more time checking twice than to be sorry later.
3. Protect valuable resources at all cost, such as creating IAM policies to deny any harmful actions.

We hope this article has helped you to learn something new and continue to expand your knowledge. If you need information about Cloud and the DevOps world, we invite you to contact us and keep checking out our blog in order to find other useful publications. See you soon!

La entrada How to recover an accidentally-deleted AWS elastic IP se publicó primero en Geko Cloud.

By default, the AWS Simple Email Service (SES) service does not offer us to be able to view logs of your actions / events, we can only see some metrics in Cloudwatch. In Geko we have encountered more than once the need to see SES logs, either to diagnose incidents or to better understand the status of the service. Well, to make this possible, we use AWS Kinesis. We are going to explain how we make it. Let’s begin!

Kinesis configuration

Kinesis is a data collector, processor and transmitter. The first step is to create a Delivery Stream. We access the Kinesis service, Delivery Streams and create a Delivery Stream

In “Source” we choose “Direct PUT” and in Destination “Amazon OpenSearch Service”. There are other destination options such as Redshift, S3, Dynatrace .. All available options appear in the drop-down list.

We add a name to the object “Delivery Stream” and add our OpenSearch. In our case, as we already have an OpenSearch created and operational by simply accessing “Browse”, it has appeared to select it.

We add the index name that we want it to create.

And by default it selects VPC, Subnet and Security Group based on our OpenSearch.

Finally we create the object “Delivery Stream”

SES Configuration

Now we access the SES service, we access “Configuration Sets” and “Create Set”

We add the Configuration Set name and create it.

Once the “Configuration Set” is created we go to “Event Destionation” and “Add Destination” to create one.

We select the type of event that we want and go to the next step

We select “Amazon Kinesis Data Firehose”, we put a name and we select the “Delivery Stream” that we have created previously.

Watch out! Now you have to create an “IAM Role” so that SES can write to Firehose, we have created it like this:

Con esta Policy

And this Trust Relationship

Finally we create the “Configuration Set”

Now, to test it, you can send a test email from SES assigning to our test the “Configuration Set” that we have created. This “Configuration Set” can be applied to any Identity that we have in SES. In fact we are going to do a test by sending a test email. We see the test log that we have generated in our OpenSearch:

And in the Kinesis Delivery monitoring:

From Geko we hope that if you have come this far, this entry is just what you were looking for! We also invite you to read other of our labs posts

La entrada AWS Logs to OpenSearch via Kinesis se publicó primero en Geko Cloud.

Today, millions of organisations are leveraging AWS cloud services to build sophisticated applications that help them increase reliability, flexibility and scalability. This post on “What is AWS?” is designed to help you gain an in-depth understanding of this popular cloud platform, its benefits, services and more.

What is AWS and what is it for?

Amazon Web Services (AWS) is a cloud services platform that offers a mix of software-as-a-service (SaaS), platform-as-a-service (PaaS) and infrastructure-as-a-service (IaaS) packages from dozens of data centres and multiple availability zones (AZs) in regions around the world. Each AZ contains multiple data centres. Customers can configure virtual machines and replicate their data across multiple AZs to provide a system that is highly resilient to server or data centre failure.

AWS aims to enable organisations to grow and profit through its wide range of services. With this platform, they can store, access, and manage their business data while protecting it from unauthorized access.

What services/products does AWS offer?

Within all the services offered by AWS, we find various cloud computing products, databases, blockchain, application integration, internet of things, migration and transfer, networking and content delivery, and many more. Some of the most widely used Amazon Web Services technologies are:

1. Amazon EC2
Amazon EC2‘s web interface is used to reduce costly physical servers by creating virtual machines and managing other server features, such as storage, security, ports, etc. With Amazon EC2 you can create servers in minutes with your preferred operating system.

2. Amazon RDS
Amazon helps us in every way it can to make our infrastructure less complicated, which is why it provides us with the RDS service. By using this AWS service, dedicated database instances can be created in a matter of minutes, fully managed by the AWS support team and capable of supporting multiple database engines such as MySQL, PostgreSQL, SQL Server, MariaDB, and many more.

3. Amazon Simple Storage Service (S3)
Amazon S3 is AWS‘s core service for file storage and retrieval via an API (Application Programming Interface). Amazon S3 offers a file storage service as well as high-quality integrations such as PCI-DSS, HIPAA/HITECH and FedRAMP so that data is not compromised. In addition, the S3 service offers you different types of storage to optimise the cost.

4. Amazon CloudFront
Amazon CloudFront helps you improve website speed and access to data in the cloud. It also manages all user content efficiently through a global content delivery service.

5. Amazon VPC
If you are ready to isolate your entire IT infrastructure from exposure, then the only way to do it is with Amazon VPC. This service creates a virtual private network that cannot be accessed by anyone or anything except the people and systems you authorise. These networks are logically isolated from other virtual networks in the AWS clouds.

6. Amazon SNS
SNS is a Simple Notification Service that alerts subscriber services to automatically perform tasks in response to specified triggers. Amazon SNS is a notification system that offers integration with any type of application, be it PHP, Python, Node, etc; and notifications can be sent to all types of users whether on the web or mobile.

7. AWS Lambda
Is your server getting saturated with requests? AWS Lambda is designed to scale to handle any traffic load. It is possibly one of the most innovative services on AWS, as it allows you to run your code in a serverless way, i.e. without the need to manage servers. In addition, it is compatible with the main programming languages such as Node.js, Python, Java, and more.

8. AWS Autoscaling
If you are looking to grow or expand your application to millions of users then AWS Autoscaling is the solution. This tool allows you to manage a fleet of servers when your application demands it, creating multiple instances when needed. The service is free, only charged for the number of instances used.

9. AWS Beanstalk
AWS Beanstalk was created to help developers manage website infrastructure and save time. This tool automates the installation, configuration and provisioning of other AWS services such as EC2, RDS and S3.

As you can see, AWS has a large number of services related to cloud infrastructure, related to data and the use of this data for any business, of any size and sector. If you want to know more about all the products offered by AWS, click on the following link to get all the information: Cloud Products (amazon.com)

Advantages of using AWS

Many companies, small or large, rely on AWS for the wide range of services it offers. Here are some of the most notable advantages that differentiate it from other companies. These are:

SECURITY
Security is one of the most important issues when working in cloud environments. AWS has dozens of data centres around the world that are continuously monitored 24/7 and strictly maintained. In addition, AWS has certifications and accreditations, encryption of data at rest and in transit, hardware security modules and strong physical security.

PAY PER USE
One of the most important advantages is the AWS pay-as-you-go billing model. There is no upfront cost to build a storage system and no need to estimate usage. For organisations that have daily or seasonal peaks, this is a real advantage.

WIDE RANGE OF SERVICES
Over time, AWS has expanded its range of products and services to more than 175 products ranging from basic storage and computing to more specialised services such as robotics. In short, AWS offers such a wide variety of services that there is bound to be a product or service that fits your business needs perfectly.

COST-EFFECTIVE
Whether you are a start-up or a large enterprise organisation, you can benefit from the economic advantages that AWS offers, such as not having to pay upfront to use its services or resources.

GLOBAL LEADER
Amazon Web Services has networks in nearly 190 countries around the world and supports more than one million active customers. Its customers include large enterprises and public sector organisations. As businesses continue to migrate to the cloud, the choices of cloud-based services continue to multiply. However, only a handful of options manage to meet the different business needs that exist, such as Amazon Web Services (AWS).

How does AWS work?

First of all, you need to register with AWS. When you go to create your account, you will be asked for your credit card details. Once your account is created, you can start exploring the AWS management console and view its services. You can also watch 10-minute tutorials on Amazon to learn how each of their products works.

AWS and Geko Cloud

At Geko Cloud we are cloud experts and we work with DevOps methodology, accompanying our customers throughout the process from cloud migration to cloud management. Contact us with no commitment!

La entrada ¿What is AWS? se publicó primero en Geko Cloud.

Introduction

In this post we are going to see how to solve the following error that we face while working with MySQL databases using AWS EFS service for storage:

ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine

Starting situation

Our use case consisted of a Kubernetes EKS cluster, where we needed to create and destroy new environments for development as quickly as possible. Each of these environments had to contain several MySQL databases, in addition to other tools such as Redis, RabbitMQ, etc.

To do this we decided to use Helm Charts that raised these tools as StatefulSets. At first, we did not specify any StorageClass, so the EKS default was used, which raises EBS gp2 (General Purpose SSD) volumes when creating PersistentVolumeClaims.

The problem with using EBS volumes is that each volume is available in a specific Availability Zone, so if the Kubernetes cluster needs to move a pod from, for example MySQL, it will not be able to lift it unless it is available in another worker node that is in the same Availability Zone as that EBS volume.

For this reason, we decided to use a new StorageClass that uses EFS instead of EBS. In this way, by having the EFS mounted on each worker node in the Kubernetes cluster, the pods could move seamlessly between nodes, even if they were in different Availability Zones.

Error with EFS

Since our use case required creating new environments as quickly as possible, when building a new MySQL database, we also executed an import of data from sql files to have an initial data set by default.

It was at this moment that we began to detect the error, since when importing this initial data, we continuously began to see the following error in the logs:

ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine
ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine
ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine
...

This error began to appear after a certain number of sql instructions executed, and from that moment on, it was repeated until the sql file was finished reading.

Resolution

Upon investigation, we discovered that the problem resided with an EFS service boundary. Specifically, it was the limit of 256 unique locked files. We can see this limit in the quotas described in the AWS documentation:

https://docs.aws.amazon.com/efs/latest/ug/limits.html#limits-client-specific

Because our data import was trying to create more than 256 tables, this limit was reached and the error began to appear. This limit cannot be modified, but we were able to avoid it by modifying the MySQL parameters to, as far as possible, not reach those 256 locked files.

The MySQL parameter to modify is innodb_file_per_table. In our MySQL databases, this parameter was activated by default. This causes an .idb data file to be created for each table in the database instead of creating a single data file. We can modify this parameter in the MySQL configuration file as follows:

[mysqld]
innodb_file_per_table=OFF

https://dev.mysql.com/doc/refman/5.7/en/innodb-file-per-table-tablespaces.html

After disabling this parameter, we did not encounter the error again.

We want to thank the blog ops.tips for their work doing the post that we linked to, since it helped us a lot to understand this error, so that we could find this solution.

https://ops.tips/blog/limits-aws-efs-nfs-locks/

Conclusions

When working with MySQL databases using EFS for storage, we can have errors when reaching the limit of 256 locked files whenever we have schemes with a large number of tables or, ultimately, any system that requires simultaneously locking large amounts of files, such as it would be the case of MongoDB, Oracle, etc.

To avoid reaching this limit, we can disable the MySQL parameter innodb_file_per_table so that a data file is not created for each table.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada Error using AWS EFS for MySQL se publicó primero en Geko Cloud.

Migrating to the cloud is becoming increasingly common and convenient for most companies that work on the Internet, given that it offers advantages such as scalability, agility, cost reduction, availability, etc. But when it comes to deciding on a cloud provider, how do we decide which one is the most convenient?

We are going to talk about the most popular providers, which offer a service with certain similarities but work under different schemes, are AWS (Amazon Web Services) and Microsoft Azure, being these the most chosen by companies that want to make the leap to the cloud. But when it comes to choosing, we must take into account the advantages and service offerings of each one applied to our business strategy.

A brief comparison between AWS and Azure

We will only compare two aspects that may be fundamental for this choice, the investment and the resources that each of these two providers have.

• The investment is one of the main unknowns in the equation when we are going to migrate to the cloud, as with any change we decide to make in the company. The increase in the number of cloud providers, which means increased competition, has led to a reduction in prices and the adoption of pay-per-use (PAYG), in order to offer the best option to the client. Something that was previously offered on a monthly payment basis. What has changed is the payment model:

• We can also tell the difference between the resources offered by the two providers:

Other features to compare between AWS and Azure

• Amazon Web Services provides tools to set aside hardware, licenses and added costs, focusing on a serverless model.
• On the other hand, Azure focuses on hybrid clouds, taking advantage of the fact that on-premise models work better because they are licensed by Microsoft.
• AWS offers automated machine learning solution products.
• Azure also has machine learning products but they need more development and may cause limitations for adaptability.
• Scalability and flexibility without minimum consumption is one of the advantages of AWS.
• In the case of Azure, it has a default storage package.

Conclusion

Taking into account the features mentioned above and others that we can find to make a comparison between AWS and Azure, we can establish the following reasons for choosing one provider or the other:

• Why choose AWS?

If your company needs to significantly increase its scalability, has flexibility in payment models and needs consulting to migrate to the cloud yes or yes.

• Why choose Azure?

If your company already uses or has a relationship with Microsoft and also needs a robust and very secure infrastructure.

At Geko we help you migrate to the cloud.

If you are looking for a service provider to migrate or manage your platform in the cloud, do not hesitate to consult with us, at Geko we choose the provider that best suits your needs or technologies, and we adapt the DevOps consultancy to your work methodologies.

Contact us without commitment!

La entrada AWS vs Azure, which one should I choose? se publicó primero en Geko Cloud.

Modules in Terraform are a collection of resources that can be linked or just created individually. When we call a module, we are calling a file with a bunch of resources with some parameters inside, parameters we need either to specify in the resource itself or in another file which stores our variables.

Why do we use Terraform?

• It is a very useful tool to develop, change and have a versioning of our infrastructure in an efficient and safe way.
• It is a tool that allows you to define infrastructure as code and enables you to change and track the infrastructure with ease.
• It is independent of the platform or ‘cloud agnostic’ and allows working with multiple cloud providers.

What do we need for this lab?

• Terraform installed. In this case we run v.0.12.7.
• In our case, we are using AWS as our Cloud Provider, so we need a user with programmatic access. We need as well to give access to this user with the proper IAM Policy.
• Your preferred code editor.

Let’s get started!

Usually, first we should either set up a Terraform module with our required parameters or just get some official module from the network. In our case, I did myself a module with an S3 Bucket and its own policy and therefore, I specify what options I’d like the module to have.

It’d look something like this:

resource "aws_s3_bucket" "this" {
  bucket            = var.bucket_name
  acl               = var.type_acl
  force_destroy     = var.destroy

  tags = {
    Name            = var.tag_name
  }
  versioning {
    enabled         = var.versioning
  } 
}

resource "aws_s3_bucket_policy" "this" {
  bucket = var.bucket_name
  policy = templatefile("${path.module}/templates/s3_origin_access_identity.json", {
    origin_access_identity_arn         = aws_cloudfront_origin_access_identity.this.iam_arn,
    origin_access_identity_bucket_name = var.bucket_name
  })
}

As you may noticed, everything is “variablized”. Hence, we’ll need to create a file with our variables stored with their custom values.

In the same folder we have our file “main.tf”, we create our “variables.tf”:

### S3 VARIABLES

variable "bucket_name"      { }
variable "tag_name"         { }
variable "type_acl"         { }
variable "versioning"       { }
variable "destroy"          { }
variable "aliases"          { }
variable "certificate_arn"  { }

In this particular case, you may ask yourselves why these variables don’t have any value stored. And the answer is, we’re using a main file called “terragrunt.hcl” which will store all our values when we call the module.

Another thing that may be confusing is how to specfiy the bucket policy. We can either specifyit with a heredoc file (EOF) or straightaway with a function of Terraform called “templatefile”.

The first example is in Terraform Documentation, a website that I strongly recommend you to visit to understand how powerful modules and its attributes are.

The second example, and the one we used, it’d look something like this:

On the same folder, we must have a folder called “templates” and inside, introduce the bucket policy as a JSON file.

{
    "Version": "2012-10-17",
    "Id": "PolicyForCloudFrontPrivateS3Content",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "${origin_access_identity_arn}"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::${origin_access_identity_bucket_name}/*"
        }
    ]
  }

You may’ve noticed we introduced ${origin_access_identity_arn} and ${origin_access_identity_bucket_name} and this is because we want to set such values manually like this:

policy = templatefile("${path.module}/templates/s3_origin_access_identity.json", {
    origin_access_identity_arn         = aws_cloudfront_origin_access_identity.this.iam_arn,
    origin_access_identity_bucket_name = var.bucket_name
  })

As you see, we introduce the “arn” calling another resource called “aws_cloudfront_origin_access_identity” and the bucket name we introduce it as we have seen before with our variables file.

Now you just need to specify your Terraform provider and to call the module with another file with the values you have specified and deploy it:

provider "aws" {
  region     = var.region
  version    = "~> 2.0"
}

provider "aws" {
  alias  = "aws_us_east_1"
  region = "us-east-1"
}
module "s3" {
    source               = "../../../../../../modules/s3-cloudfront/"
    bucket_name          = var.bucket_name
    tag_name             = var.tag_name
    type_acl             = var.type_acl
    versioning           = var.versioning
    destroy              = var.destroy
    aliases              = var.aliases
    certificate_arn      = var.certificate_arn    
}

Deploy it with a terraform init, terraform plan and terraform apply.

Conclusion

So, we saw how to deploy an S3 Bucket with its policy with Terraform. I strongly recommend you visit the official Terraform Documentation to keep learning about modules and all related to them.

I invite you all in case you need information about the DevOps world, to contact us and keep checking our blog to find other useful posts.

La entrada A quick view of modules in Terraform se publicó primero en Geko Cloud.

We have different options when using these NFS volumes in our containers:

• Firstly, we can create a volume that we can use either in docker compose or using a docker run. This is the reason why we will use the following command: 

    $ docker volume create --driver local 
        --opt type=nfs 
        --opt o=nfsvers=4,addr=nfs.example-domain.com,rw 
        --opt device=:/path/to/dir 
        volume-name

  Once we have created the volume, we can mount it inside our container:

    $ docker run -it --rm 
      --mount type=volume,dst=/container/path,volume-driver=local,volume-opt=type=nfs,"volume-opt=o=nfsvers=4,addr=nfs.example-domain.com",volume-opt=device=:/host/path 
      volume-name

• The same configuration can be applied within our docker-comper. Inside our docker-compose.yaml we will have to create the volume and mount it inside our container.

  # Container conf
    ...
    container-conf:
      container_name: foo/bar
      image: imagename:tag
      ports:
        - 80:80
      volumes:
        - new_volume:/path/inside/container
    ...
  # Create volume
    ...
    volumes:
      new_volume:
        driver: local
        driver_opts:
          type: nfs
          o: nfsvers=4,addr=nfs.example-domain.com,rw
          device: ":/host/path/to/dir"
    ...

  We have found ourselves in the same situation needing to fit the container with an EFS (Elastic File System) from Amazon Web Services. This AWS service provides scalable, elastic and concurrent storage and we will be able to add it using the same NFS docker driver.

  # Container conf
   ...
    container-conf:
      container_name: foo/bar
      image: imagename:tag
      ports:
        - 80:80
      volumes:
        - new_volume:/path/inside/container
   ... 
  # Create volume
   ...
    volumes:
      efs_volume:
        driver: local
        driver_opts:
          type: nfs
          o: addr=[EFS_DNS],nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
          device: [EFS_DNS]:/
   ...

  Please note that if you have just created this EFS, it may be that when you try to create this volume you will get an error warning that you cannot find the EFS endpoint. This is due to the fact that these DNS will take a few minutes to propagate. If you cannot add it via the endpoint, you can do it directly from the EFS’s IP.

We hope that this information has been useful to you and don’t forget that at Geko cloud we are at your disposal for the management of cloud infrastructures and microservices.

If you have other questions related to docker or Kubernetes, perhaps you can find answers in other posts on our blog.

See you next time!

Contact us and let’s talk about how we could help you make your platform more agile and secure!

La entrada NFS volumes in Docker containers se publicó primero en Geko Cloud.

RDS is the managed database service offered by AWS that solves the main management, operation and maintenance problems involved in having a relational database platform. With RDS everything is very simple, you guarantee the service and the operative allowing to grow and to decrease according to necessity.

In addition Amazon has its own engine based on Mysql, Aurora that even allows to have a serverless database platform! Which we will talk about in another post later.

The scenario we face

A client with its full AWS-based platform with all its relational databases using RDS manifests in one of its platforms performance problems, crashes, wandering and unpredictable behavior etc.

It turns out that this problem does not occur from the beginning but at a certain point in time it started to degrade until it became a real problem. We appreciate that RDS was deployed using T3 bursting instance where credits seemed to decrease periodically.

We decided to change the instance to type M5 to have a better performance and resource availability . The performance improves very significantly until we noticed the following:

The metrics continue to warn that there is write/read latency and IOPS are glued. How is this possible in a general purpose SSD?

The metric that for us is important to follow in any RDS is the one referring to the Disk QueueDepth, this metric indicates the size of the queue that the system has waiting to be processed in disk IO operation, that is, the debt in disk operations that the system has. Read Write disk latencies are nothing more than mirages of this fact.

The solution

The explanation is simple, AWS RDS provides a total of 3 IOPS/Gb depending on the size of the EBS disk that is created. The client RDS was created with 20Gb total size = 60 IOPS , which AWS by default always provides a minimum of 100 IOPS.

This explained why the CPU and RAM were not affected but the performance as such was.

As we detected the bottleneck, we simply proceeded to expand via Terraform the size of the associated EBS volume to 100Gb with its corresponding 300IOPS (three times as before).

The result here you have it :

Nothing more to add, no more latencies, no more blockages and no more problems with this platform, we leave you the link to the official AWS documentation on RDS and EBS

Remember that for productive environments, the recommendation is always to provide at least 100Gb per disk (even if we don’t need it) to obtain those 300 IOPS minimum.

Remember that at Geko we are still at your disposal to help you with any performance problems you may have with your platform.

La entrada RDS performance issues by IOPS on GP SSD se publicó primero en Geko Cloud.

Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

That said, let’s enumerate in which situations Velero can help us:

• Backup stateful applications
• Backup applications installed in a non-declarative way
• Backup PVC information
• Cluster migrations
• Replicate cluster configurations (for example, from production to testing or development clusters)

After the boring introduction we will start with the funny part.

Install Velero on your kubernetes cluster

For this example we will use AWS EKS as our cloud provider.

First of all, you must install the Velero client on your workstation. To to do that, simply go here and download the latest tarball according to your workstation operating system and extract the tarball:

tar -xvf RELEASE-TARBALL-NAME.tar.gz

and move the Velero binary to somewhere in your $PATH. On MacOS simply execute this:

brew install velero

Once we have the Velero client installed on our system we can proceed to install Velero on our kubernetes cluster. Just execute this:

velero install 
    --provider aws 
    --bucket where-to-store-velero-backups 
    --use-restic 
    --secret-file ~/.aws/credentials 
    --use-volume-snapshots=false 
    --plugins=velero/velero-plugin-for-aws:v1.0.0 
    --backup-location-config region=eu-west-1

By default Velero is installed in the velero namespace but you can change it adding the –namespace flag. Velero uses restic to store the data of attached volumes. When the installation finishes you should see something like this in your k8s cluster:

$ kubectl get all -n velero
NAME                          READY   STATUS    RESTARTS   AGE
pod/restic-4xr6v              1/1     Running   0          157d
pod/restic-bg58w              1/1     Running   0          157d
pod/restic-hf2fm              1/1     Running   0          157d
pod/restic-w4rvh              1/1     Running   0          157d
pod/velero-57cd659988-bd6nd   1/1     Running   0          73d

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   4         4         4       4            4                     157d

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/velero   1/1     1            1           157d

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/velero-57cd659988   1         1         1       157d

In our case, we have 4 worker nodes and restic is installed on every node we have (configures a daemonset).

By default, Velero will not include the volumes in the backup. To backup them you have to add an annotation. You can add it like this:

$ kubectl -n annotate pod/your_pod backup.velero.io/backup-volumes=volume_1,volume_2

(add volumes comma separated) or simply add it in your deployment definition, for example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        backup.velero.io/backup-volumes: myapp-logs
    spec:
      volumes:
        - name: myapp-logs
          persistentVolumeClaim:
           claimName: myapp-logs
      containers:
      - image: myapp:latest
        name: myapp
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/var/log/myapp"
            name: myapp-logs
            readOnly: false

 Note: hostPath volumes are not supported, but the new local volume type is supported.

And finally, some utilization examples to create, restore and show backups:

One time backup:

# Backup ALL resources in the cluster (the whole cluster)
$ velero backup create my-backup-20200515

# Backup a namespace
$ velero backup create my-backup-20200515 --include-namespaces namespace_to_backup

# Backup ALL namespaces except ones specified
$ velero backup create my-backup-20200515 --exclude-namespaces namespace_1_to_exclude,namespace_2_to_exclude

Scheduled backup:

$ velero create schedule myapp-backup-daily --schedule="0 7 * * *" --include-namespaces namespace_to_backup

Velero will backup all resources included in your selection (pods, deployments, services, …)

The default backup retention is 30 days. If you want to change it add the –ttl flag. This flag allows you to specify the backup retention period with the value specified in hours, minutes and seconds in the form –ttl 24h0m0s. If not specified, a default TTL value of 30 days will be applied.

Restore a backup:

$ velero restore create --from-backup backup_name

Some other useful commands:

# To show all stored backups list (name, status, creation and expiration date)
$ velero get backups

# To show one specific backup details
$ velero describe backup backup_name

Velero is quite easy to install and run and it can save your life in some circumstances.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you with your projects.

See you in the next post!

La entrada Kubernetes backups with Velero se publicó primero en Geko Cloud.