Based on the current trends and the opinions of experts in the cloud industry, we leave you with some of the trends that could take shape in the cloud market in 2023:

Increased adoption of hybrid cloud:

Hybrid cloud is a combination of public and private cloud, allowing enterprises to use public cloud infrastructure for non-critical workloads and use private cloud infrastructure for non-critical workloads. critical that require a high level of control and security. Hybrid cloud allows companies to get the best of both worlds, which means they can take advantage of the scalability and flexibility of the public cloud and the security and control of the private cloud. Additionally, hybrid cloud enables easy migration of workloads between public and private cloud, which is ideal for businesses that need an agile and flexible IT infrastructure.

Increased focus on cloud security:

With the increase in the amount of data stored in the cloud, security has become an increasingly important concern for businesses. In 2023, companies are expected to invest in cloud security tools and services to ensure the protection of their data. Cloud security solutions can include authentication and authorization, data encryption, security monitoring and analytics, as well as threat prevention tools. Additionally, enterprises can also adopt cloud security practices such as network segmentation, identity and access management, and regular security risk assessment.

Increased adoption of AI and machine learning in the cloud:

The cloud is an ideal platform for processing and analyzing large amounts of data, and the use of AI and machine learning is expected to expand in the cloud to improve business efficiency and productivity.

Greater emphasis on sustainability:

Companies are looking for ways to reduce their environmental impact, and the cloud can be a solution in terms of energy efficiency. Companies are expected to increasingly focus on sustainable cloud solutions in 2023.

Increased adoption of the cloud without a server (Serverless):

The cloud without a server or serverless allows companies to reduce costs and increase efficiency by paying only for the resources they use. Instead of having to pay for an entire infrastructure, companies only pay for the service requests they use. This allows for greater scalability and agility in the deployment of applications and services. In addition, the serverless cloud also eliminates the need to manage and maintain infrastructure, allowing companies to focus on application development and innovation. Serverless cloud adoption is expected to continue to rise in 2023 as more businesses realize the benefits of this technology and the ease of use it offers.

In conclusion, the cloud market continues to evolve rapidly and companies are adopting new strategies to take full advantage of cloud capabilities. Hybrid cloud adoption allows enterprises to combine the best of both worlds, the scalability of the public cloud and the control and security of the private cloud. Additionally, enterprises are increasingly focusing on cloud security, adopting security tools and services to protect their critical data. Artificial intelligence and machine learning are also expanding into the cloud to improve business efficiency and productivity. Additionally, companies are embracing sustainable cloud solutions and focusing on reducing their environmental impact. Finally, the serverless cloud continues to be an important trend, allowing companies to reduce costs and increase efficiency by paying only for the resources they use. Summing up all of the above, these trends are expected to continue into the future and companies will continue to adopt new strategies to reap the benefits of the cloud.

If you want to be in the latest cloud trends, at Geko Cloud we can help you establish new strategies in your hybrid cloud to evolve your business.

You just have to leave us your contact and we will write to you!

La entrada MAIN CLOUD TRENDS FOR 2023 se publicó primero en Geko Cloud.

Engineering Culture is the nice name that has been given to the concept that is trying to take root in new companies to share the knowledge you have, the tools you find useful, or the help that can be given to other projects. It goes with the idea of the DevOps methodology of eliminating silos of knowledge and work and that sharing this knowledge generates benefits for everyone involved.

What are the benefits?

Sharing knowledge with the rest of the internal teams, for example through a slack channel or shared proofs of concept, or with the rest of the world through open projects or blog posts (like this one!) not only helps to get external contributions to improve ideas and processes, but it is also a very good way to gain publicity and quality public outreach, which generates better leads and is positive image-wise.

How can I promote Engineering Culture?

There are many ways to drive this culture in your company to take advantage of its many benefits:

• Create quality content with lessons learned in blogs, videos or social media.
• Dedicate internal time to teaching and training of tools at the technical level by the teams themselves.
• Dedicating resources to creating open source tools for the public
• Dedicating resources to “giving back” to open source projects that you use, either financially or in terms of programming time or bug fixing, or documentation.

Keeping things in-house is no longer in, part of your marketing and lead creation strategy can, and perhaps should, include giving back to the community that supports your product every day!

We invite you to continue reading our articles where you will find a lot of updated information about the sector.

La entrada Engineering Culture: keeping knowledge is not in anymore se publicó primero en Geko Cloud.

Introduction to SSM

SSM is the acronym for Systems Manager, a set of tools that make our lives easier, not only for accessing machines through a virtual console, but also for executing remote commands, extracting records from the equipment, and of course , apply updates.

With the SSM software suite, you can update not only EC2 instances, but also on-prem computers, as well as virtual machines, on-the-edge (IoT) computers such as raspberrys, etc.

Systems Manager is a service that runs an agent on the computer itself (ec2 / on-prem, etc), in the same way as virtual agents in VMware environments, used to extract metrics from the instances, but in the case of SSM the console management is remote, or what is the same, it is divided into different end-points. That is why the agent must communicate with the SSM service over the network. The computers that want to be managed must have not only the agent installed but also network connectivity to reach the AWS SSM endpoints.

In this article we have used a couple of EC2 instances, one Linux and one Windows:

For Linux we have used a RedHat AMI, and in the case of the Windows computer, it is version 2019.

Role for SSM

In order for the computer to communicate with SSM, a role is required. For this reason we are going to generate a new “service role”, accessing the IAM section and selecting the “Roles” option and clicking on the “Create role” button:

We select the “AWS Service” as “Trusted entity”, and below in the “Use case” section, the EC2 service. We select the following and add the policies that are necessary:

To do this, enter the name “ssm” in the search box, and choose from the options that appear the “AmazonSSMManagedInstanceCore”

We assign the name we want and make sure that in the “trusted entities” section, the value of “principal” selected is the service “ec2.amazonaws.com”, the service with which this role is going to be invoked and to which we are giving permission:

The next step will be to add this role to the instances. The easiest way is to mark the instance to be modified from the panel, as shown in the image.

In the dropdown, assign it the role that we just created.

EC2 instances / managed nodes

In our case, we deployed a RedHat machine for the Linux variety for a reason: The official AMIs distributed by RedHat do not have the agent built-in, just like Windows AMIs distributed by Microsoft, or Amazon Linux AMIs do.

In case our distribution is agentless, it can be installed manually. Check the following link to see which distributions are supported:

https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html

In the case of RedHat we will proceed to install it with a command like the following:

sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

There are also binaries available for installing the agents for Windows or MacOS environments.

Fleet Manager

Now that we have the computers with the SSM agent installed and with the associated “service role”, we could access the “fleet manager” section, where the computers managed with SSM are displayed as a record, both EC2 computers and on-prem equipment, etc. This tool is within the SSM options:

The devices managed through SSM are called nodes, which is why, in the view of this service, the registered devices are identified in the “Node Id” column:

The computers that appear are those that are registered in the current region. Keep in mind that AWS SSM is a regional service, which means that if we have an account in several regions, it will be necessary to repeat the process described in this article for each region independently.

Like all software, the SSM agent receives constant improvements. It is therefore recommended to enable automatic updating in the “settings” section:

Baselines

Baselines are statements that define a certain level of compliance with a security policy. In the “patch manager” service there are different “baselines”, but it is highly recommended to create your own one with the configurations that interest us. It is a really simple process.
To access the “baselines” section, go to “Node Management” and select the “Patch Manager” option:

And on the screen that appears, access the “View predefined baselines” button

In the “baselines” view, there are a lot of them, each one for a different operating system. It is at this moment when we are going to create a personalized one. To do this, click on the “Create patch baseline” button.

In the new section, select a name and an OS family that will be associated with the “baseline”:

Define the version of the OS, as well as other aspects that will define the level of compliance:

There are options such as “auto approval”, which allow you to approve the installation of a certain update, after a few days of its publication, or after a certain date, thus avoiding that day 0 patches, not very tested, could break stability of the teams. This would even allow PRE or TEST teams to install patches days before production teams as “canary testing”.

Just as important is the package exceptions section. This prevents the installation of certain packages that can break the service (systems with hand-compiled kernels, Windows packages that don’t get along with any other third-party program, etc).

To add exceptions the following format must be used:

At this moment we already have our own “baseline”. When the “baselines” are your own, to find them later, you have to specify that we are the owner. To do this, in the baseline search engine, specify “Owner: Self” as shown below:

Otherwise the default view only shows standard baselines provided by AWS.

Patch groups

Once we have defined the “baselines”, we are going to define the “patch groups”. A “patch group” establishes the relationship between the “baselines” and the equipment on which they are going to be applied, depending on the value of a tag that we will specify later in the equipment.

To create a “patch group”, it is very simple, we just have to select the “baseline” that we have just created, we display the “Actions” menu and select the “Modify patch groups” option.

A section will open to generate the patch groups, entering a name. In our case “Windows_Production”

Once the name has been entered, taking care to respect the nomenclature, click on the “Add” button and close the wizard.

Tags

Tags are metadata that help us identify who owns a resource, a cost allocation unit, etc. In this case, with the tags we can also associate the nodes to the corresponding patch groups. For this, the same “Patch Group” tag value is always used. Eye! It is case sensitive

We go to the instances section, select one of them, and edit the tags section to add the “Patch Group”:

If when entering the tag, we find an error of the type “‘Patch Group’ is not a valid tag key.”

es necesario deshabilitar la check box de metadata tagging en la configuración de la instancia:

Dejar la casilla siguiente deseleccionada y volver a probar a añadir el tag en el equipo.

Maintainance Windows

Let’s recap. At this moment we already have the instance with the SSM agent installed and the role to interact with the service. We have created our own “baseline”, and then we have assigned it to some “patch groups”. Finally we have “tagged” the instances with a tag that points directly to the “pach group” that interests us.

The maintenance windows are an important section because they not only contain the scheduling of the updates, but also the list of executions.

We are going to proceed by creating a maintenance window through the following section:

We must enter the name for the window, and set the cron. You can establish cron expressions with the typical nomenclature or by selecting the pre-established ones of every hour or every day.

The values of “Duration” and “Stop initiating task” are mandatory, and it is recommended to leave them with values such as the following:

Finally, choose the time zone to match the schedules of the window with the one that interests us.

At the end we will have a list of the scheduled windows, and interesting data such as its next execution:

Patching configuration

At this time we are going to carry out the process that brings together all the previous steps, and that enables the “patch manager” to carry out system updates. To do this, access the “Patch Manager” section and click on “Configure Patching”

Choose the “patch groups” that you want to update:

Choose the maintenance window that was created in the previous step:

And leave the operation that best suits what we want to do. In this case we are going to choose “Scan and Install”

At this moment, in the “dashboard” of “Patch Manager”, we should have a view of equipment pending update. The values in this section will change depending on whether there are pending packages, update failures, etc.
In order to track the actions performed, we can consult them in the “Run Command”

In this section there are two tabs. One with the commands currently running, which will come out with a clock next to it:

And another tab with the history of actions:

If in the process of installing updates, there is any package marked as restart required, the computer will restart immediately. Take this into account to establish the windows according to the needs of the business:

After the installation of the patches, and the reboots if any, in the “dashboard” of “Patch Manager” the instances will now appear as compliant according to the “baselines” established:

Cloudwatch

In the command history console, we can see the executed tasks, as in this case. If we click on the id of the task, we can access its details:

Inside we can see the computers on which the action has been executed, and we can, by selecting one of those computers, see the details of the operations. To do this, select and click on the “View output” button.

The operations are testing all the platforms, for this reason they will first try to execute the patching on a Windows, then on a Linux and finally on a MacOS. Therefore, to see the status of the updates, we choose the platform of the team we are consulting.

In the “Output” window we will see the record, but it is often truncated because it is excessively long:

If we want to consult this data, we can send it to a Cloudwatch “log group”. To do this, first we go to CloudWatch and create a “log group”:

We put a name and specify the retention period.
The second step is to modify the role of the machine so that it has permissions to write to Cloudwatch, so you will have to add an “in-line policy” like this:

We put a name to this “policy” and save it:

Leaving the role of the team as:

We return to the system manager, section “Maintenance Window” and within the tasks (I mark the task and select edit):

And in the “Output Options” section, write the name of the log group where we are going to send it:

We save the changes, and from now on when selecting the output of the tasks, we will see that a link to Cloudwatch appears:

And if we consult that link, it will take us to the output information, but without truncating:

From Geko Cloud Consulting, we hope you liked this post and above all that you find it useful.
If you need information about the Cloud and DevOps world, we invite you to contact us and keep checking our blog to find other useful posts.

La entrada Patching equipment with Patch Manager se publicó primero en Geko Cloud.

La entrada ChatGPT: What is it and what are its implications? se publicó primero en Geko Cloud.

As we exploit our infrastructure hosted in the Amazon cloud, our bill also begins to “explode”. This is why at a certain point (or not), it is convenient to start looking for resources so that your bill is not so bulky.

The first step, before anything else, is to be clear about what and how many resources you need for your application and/or infrastructure to have adequate performance and make a forecast about how long you think you will need these resources running at least.

For this estimate, you must have a good knowledge of your architecture and have been working on it for some time, otherwise you will not have enough capacity to predict how your infra will behave in atypical situations.

Let’s get in position. Let’s simulate several ways to save on your EC2 bill.

Reserved Instances

It consists of making a forecast of EC2 instances that you will need at least 1 or 3 years from now (it depends on the commitment time you choose).

A good practice would be to play with the equivalence that instance types have in EC2. For example, reserve 2 t3.small for each t3.medium. In this way, all your hardware will be distributed according to the needs of your infrastructure, and if at any time the number or type of instances changes, that reserved hardware will continue to be distributed and used in the most optimal way.

What is the prize for making this great foresight and commitment? Well, AWS gives you discounts of at least 25-30% and up to 50-60% on the EC2 bill!

The longer your commitment time and the more you pay up front, the bigger the discount.

Spot requests

These instance types allow you to take advantage of unused EC2 capacity in the AWS cloud. Spot Instances are available at a discount of up to 90% compared to on-demand prices.

Where’s the catch? Although it is hard to believe, they are instances that if AWS needs for another user or purpose, they can be removed instantly with just a 2-minute notice (if you find out, turn off the machines and they won’t be removed )

This is why it is recommended to use these instances only for container environments, CI/CD, testing… or combine them in an ASG with your on-demand instances.

Scheduled Instances

An effective practice if you know how to detect what hours or days of the week your infra processes more load than normal.

It is about programming a number of machines during a specific time. If, for example, there is a time of day where there is more traffic than usual and the rest of the day is much quieter, you could schedule one or two more instances to be added to your ASG.And why is it a good option to save? Because you adjust the number of machines to the maximum based on the needs of the moment, and you never pay for what you don’t use.

Saving plans

If you have a more or less large number of instances and you have been with that number for quite some time and you believe that for now it will not vary excessively, personally a Saving Plan is your best option.

What the saving plan basically does is make an analysis of the resources and power that your infra needs (it takes away that work) and proposes savings in exchange for a commitment.

The truth is the most comfortable measure and if you want to spend some time checking for yourself that the prices they offer are realistic, do so and you will see that even if you reserve instances by your own means, the discount they offer is the same or higher.

There are 3 types of Saving Plans:

• Compute: Provides more flexibility and reduces costs by up to 66%. Based on EC2, AWS Fargate, and AWS Lambda usage, regardless of instance family, size, zone, region, OS, and tenancy
• EC2 Instance: Provides the lowest prices and reduces costs by up to 72%, in exchange for the commitment to use individual instance families in a region.
• SageMaker: Reduce costs by up to 64% in SageMaker. They are based on the use of SageMaker regardless of instance family, size, component, and AWS Region.

To apply a Saving Plan, we select this option:

Once inside, we can choose the time and price ratios that we are willing to pay and see what discount is applied, or click here and see the recommendations that AWS proposes based on the last X days.

In the recommendations, depending on the commitment time and the upfront, we will see some prices or others.

My personal proposal about

Once you know all these methods, why not combine them?
You can easily combine Spot Instances with On-Demand, Reserved, and Savings Plans instances to further optimize workload cost with performance.

You can also shut down machines while you are not using them or use the Scheduled Instances method.

Other services where reservations are a good ally
In this post, we wanted to emphasize the reservation of instances and the EC2 Saving Plans, but AWS offers many services where, thanks to the forecast of resources that you will need at least in a specific time, you can save yourself a good pinch.

For example, in the DynamoDB service, you can reserve capacity. It works the same as reserving instances, in exchange for a good discount, you can reserve disk capacity for a specific time with different payment formats in advance.

Another example is Opensearch, the SAS of ELK (Elasticsearch, Logstash and Kibana) that incorporates AWS. In this service you can make reservations for instances, but with a very important peculiarity to take into account.

Unlike other reservations, in this service you have to reserve exactly the type and the exact number of instances that your Opensearch uses. Basically, there is no use reserving 2 t3.small instances for one t3.medium to use its resources, for example. If your Opensearch uses 2 Data Nodes t3.medium and 3 Master Nodes m5.large, you must reserve exactly these types and numbers of instances.
I personally recommend it anyway, the savings are very remarkable.

Close

As the title of this post indicates, the best option you have to save money in all AWS services where machines are involved is forecasting. If your infrastructure has the expected performance and you know its behavior well, you will know how to forecast the number of instances or resources you need in 1 or 3 years.
It is worth making an effort to analyze your resources and taking a few hours or days looking at your ASG and other services, you can easily reduce 50% on your next bill.

If you want help to carry out these practices, do not hesitate to contact us!
The Geko team has experience in different fields, working with DevOps methodology, accompanying our clients throughout the process from migration to cloud management, and adapting 100% to their infrastructures and technologies.

La entrada THE FORECAST. Your best ally to save on your AWS bill se publicó primero en Geko Cloud.

By default when creating persistent volumes in AWS the GP2 type is created, however, for some time now we have had GP3 available, which provides us with different advantages, including cost savings. In the following table we can see a comparison of both versions:

Here is a little howto on how we can change the default storageclass to Gp3 step by step.

Create IAM Role

Get OIDC provider url:

Create the following file to define the role by changing 111122223333 to our account ID, region-code to the aws region, and EXAMPLED539D4633E53DE1B71EXAMPLE with the value returned from the previous step:

Create the role with the file created previously:

Attach the policy to the role:

Create the identity provider from the aws console by following these steps:
1. Within IAM in the Access management section go to Identity provider – add provider – OpenID Connect
2. In the Provider URL add the url obtained in point 1.a
3. Click on Get thumbprint
4. In audience put “sts.amazonaws.com” and click on save

Install aws ebs-csi-controller driver

Create the following values.yaml

Add the helm repo:

Install helm chart:

Once all the previous steps have been followed, we will be able to observe how the default storageclass is of the gp3 type, and from now on all the volumes will be created using this type of ebs:

La entrada AWS: Enable EBS GP3 for eks by default se publicó primero en Geko Cloud.

To understand what replica lag is, we first have to put a bit of context. Replication lag occurs in a special type of database instance that we call read replicas. These are created from a source DB instance that acts as the primary database. Updates (inserts of new data) made to the primary database are copied asynchronously to the read replica. Having these types of replicas allows you to reduce the load on the primary DB instance by routing read-type queries to those instances. Replication between these instances occurs through a secure communication channel and asynchronously.

So what is replica lag and how does it affect our instances? Replication lag is the delay that occurs in asynchronous replication between the primary and read databases. To identify the status of the replication, you will need to look at the binlog dump, io_thread, and sql_thread threads. Commands such as show master status, show slave status executed on the respective instances and continuous monitoring help us identify if there is any delay that could affect performance causing data inconsistency.

We cannot eliminate replica lag but we must reduce it as much as possible. Determine if there are configuration differences between the primary and replica instance, too much write workload on the primary instance, too long transactions, incorrect parameter settings, review version changelogs… these are what we usually do at Geko Cloud to decide if any type of update or other type of action should be carried out.

Lag replication example

This is what recently happened to a client who, after updating the version of the main database and its respective mariadb replicas to v.10.4.24, began to notice a deterioration in the replica lag as we can see in the graph.

Our team was investigating the problem and after reviewing the Changelogs from version 10.5.0 to 10.5.17, among other things, we were able to notice that this latest version solved the replication lag problems caused by the previous version. Also, we found that the best thing to do in this case was to upgrade the replicas to version 10.15.17 and leave the primary database at version 10.4.24. To do this, we created two more reading replicas, which we updated to that version in such a way that we could verify that the replica lag was effectively reduced again, as can be seen in the graph.

These graphs were taken from the two read replicas with versions 10.4.24 and 10.5.17 of mariadb in the same time slot to verify that the replica lag had actually decreased.

We hope this can give you an idea of how the Geko Cloud team can help you if you need to deploy your services in a Cloud environment. Just contact us and we will be happy to help you.

La entrada What is replica lag? se publicó primero en Geko Cloud.

The Service Manager has become a key piece for IT management in organizations.
It is the person responsible for the implementation and monitoring of all processes linked to the management of corporate information systems.

The function of the Service Manager at Geko is to provide versatility in the vision of both the client and the technical team in the different phases.
Adapting the client’s needs and ensuring the correct execution of the agreed actions.

On the other hand, the Service Manager, together with the TAM, is in charge of aligning the roadmap in the medium and long term, contributing their business/technology vision thanks to having accredited technical knowledge with the main cloud and Kubernetes certifications.

Some of its most relevant functions are:

• Ensure compliance with SLAs
• Responsible for IT implementation
• IT partner
• IT profitability
• Team formation
• Skills for dealing with customers
• Skills for dealing with management
• Skills to communicate and train

In addition, the Service Manager must have essential qualities and skills such as:

• Teamwork
• Ability to make decisions and solve problems
• Plan, organize and prioritize work
• Communication capacity
• Ability to obtain and process information
• technical knowledge

At Geko Cloud, we are a Cloud Consulting formed by a team that is integrated into the day-to-day of our clients, to give value to a quality service and a totally personalized experience.

We take off together?

We share some Case Studies where our values ​​and efforts are reflected in the results of the projects we have carried out.

We hope that this article has helped you to learn something new and continue expanding your knowledge.
We invite you to contact us if you need information about the Cloud and DevOps world and keep checking our blog to find other useful publications.

La entrada Service Manager role se publicó primero en Geko Cloud.

The convention, organized by the Cloud Native Computing Foundation, is aimed at the main open source and cloud native communities.
The celebration at Feria València of KubeCon and CloudNativeCon marks the return to face-to-face attendance at this convention.

Part of the Geko Cloud team has been present at the congress to promote networking between providers and clients.
They have also attended the most relevant conferences, with the aim of reinforcing and consolidating knowledge.

The congress has brought together more than 5,000 highly specialized professionals in the field of open source and cloud computing, mainly from Europe but also from the United States and another 10,000 who have followed the event online. In addition, more than 200 leading firms in this technology have also participated as exhibitors and sponsors.

This new edition has surprised us thanks to the great ecosystem and the great potential of the project (CNCF) related to the cloud native.

This year a wide portfolio of products oriented to Companies has been presented. It should be noted that the novelty curve is flattening to make way for the Enterprise part. For the last 4 years, the technical community has been paving the way and giving way to companies and we are not just talking about startups, but corporates or large-scale companies in sectors such as banking, pharmaceuticals, food, etc.

We share the most outstanding news and trends of the KubeCon and CloudNativeCon congress

• OpenTelemetry: Open source project incubated by the CNCF (second in number of commits), provides a single open source standard and set of technologies to capture and export traces, metrics and, in the near future, logs from your infrastructure and applications cloud native.
• GitOps: A best-practice process for managing infrastructure and app configurations using Git, an open source version control system, as the single source of truth for applications and declarative infrastructure.
• Gatekeeper: Allows you to manage policies in Kubernetes. Policies enforce what fields and values ​​are allowed to be used in a given resource. Has the ability to modify resources based on conditions.
• Cluster API: Set of APIs focused on provisioning, updating and operating multiple Kubernetes clusters. It allows managing these resources through CRDs.
• FluxCD: GitOps tool equivalent to ArgoCD, it does a lot of application in security (avoid possible privilege escalations, etc), although it does not have a GUI (all management is done through cli). There is a Terraform provider to manage your resources.
• KubeLinter: Tool that analyzes YAMLs and Helm charts against a series of best practices with a focus on security. It is highly configurable (enable/disable checks) and allows you to generate a recommendation report.
• Sigstore: Modular collection of projects that can be used independently or together, with the aim of simplifying and making software signing more transparent.

About KubeCon + CloudNativeCon 2022 Europe

Cloud-native technologies enable organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they enable engineers to make high-impact changes frequently and predictably with minimal effort.

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and maintaining an ecosystem of open source and vendor-independent projects. We democratize the latest generation patterns so that these innovations are accessible to everyone.

More information about KubeCon + CloudNativeCon 2022 HERE

La entrada Geko Cloud at KubeCon and CloudNativeCon 2022 se publicó primero en Geko Cloud.

Provisionar nodos slave de jenkins dinámicamente gracias a aws spot fleet

A few weeks ago, a customer contacted us. His request was simple: What can we do to further optimize our automated CI/CD process in Jenkins?

After assesing the needs and the current infrastructure of the client, we decided to change the current paradigm in terms of task execution. Until now Jenkins executed the tasks in its main node (the machine where Jenkins is running), by changing this paradigm to one based on dynamic provisioning of subordinate nodes we would gain the following benefits:

• Cost savings – By moving all the computational effort away from the main node. It can be hosted on a lower performance machine, reducing the overall cost of our infrastructure. The subordinate nodes would only be running during the execution of a task, also saving money while no task is running.
• Security – As we will not execute any code on the main node, we ensure the system’s integrity even if the code has been compromised, and since it is only executed on an ephemeral instance the attacker will be isolated.
• Performance – The subordinate node exists only to execute the task and can dedicate all of the node’s resources to it.

Requirements

For this change, we will perform the following tasks:

• Spot Fleet creation in AWS
• Installation and configuration of the Jenkins plugin
• Jenkinsfile adaptation

For this publication we will make the following assumptions:

• We have an AWS account already set up
• We host Jenkins on an EC2 server
• We have knowledge of some basic Jenkins and AWS concepts

That said, let’s get down to business.

In this article on “Dynamic nodes in Jenkins via AWS Spot Fleet” we will develop all the necessary steps to obtain a good result.

Creation of launch template and spot fleet

An AWS Spot Fleet is nothing more than an AutoScaling Group of EC2 machines but with spot type instances – Amazon’s surplus hardware that nobody is using and that AWS sells cheaper than usual. The first step will be to create the Launch Template that the Spot Fleet will use:

The important items to be filled in will be:

1. Name and description
2. AMI to use – For example Ubuntu 22
3. Type of instance to use – Choose one according to your performance needs
4. Key Pair – We will need to add it later in Jenkins
5. Network – Use the same Subnet as Jenkins if possible
6. Security Group – For the connection between nodes to be established, it is important that we give access to port 22 from the main node. The easiest way is to create a rule that allows access either from the IP of the main node (if we have a Elastic IP assigned) or from its Security Group
7. Storage – Allocate the storage according to your needs
8. User data – For the Jenkins worker to run on the subordinate node, we will need to install JRE, we also installed Docker as a requirement:

#!/bin/bash
# Install docker
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
apt-get update
apt-get install -y docker-ce
usermod -aG docker ubuntu

# Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.21.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

#Install JRE
sudo apt install -y default-jre

Once created we will have a Launch template similar to this one:

Now it is time to create the Spot fleet as such:

In this case, the items to be filled in are:

1. Launch parameters – Select our Launch Template created above
2. Target Capacity – Set total to 0 for now, as it will be controlled by the Jenkins plugin. Enable Maintain Target Capacity
3. Network – Assign the VPC in which the main node is hosted
4. Instance type requirements – Choose the desired instance types in Manually select instance types, a larger quantity gives a better pool from which to start Spot instances
5. Allocation strategy – Capacity Optimized

And that’s it, we have created our Spot Fleet.

Installing and configuring the Jenkins plugin

Before installing the plugin, we will have to create and assign a new IAM role to the instance hosting the main node so that it will be able to manage our Spot fleet:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "ec2:DescribeSpotFleetInstances",
            "ec2:ModifySpotFleetRequest",
            "ec2:CreateTags",
            "ec2:DescribeRegions",
            "ec2:DescribeInstances",
            "ec2:TerminateInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeSpotFleetRequests"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:UpdateAutoScalingGroup"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:ListInstanceProfiles",
            "iam:ListRoles",
            "iam:PassRole"
         ],
         "Resource":"*"
      }
   ]
}

Once this is done, we will install the ec2-fleet-plugin and configure it from Manage Jenkins > Manage Nodes > Configure Clouds > Add New Cloud > Amazon EC2 Fleet.

1. Assign a name
2. AWS Credentials – We leave this blank, since our machine already has the role described above.
3. Region – Select the region in which we host our servers.
4. EC2 Fleet – It will only appear when we assign the region and the role is assigned to the machine correctly. Once filled in, we select our Spot fleet
5. Upload the Private key generated in the Launch template as SSH Username with private key
6. Private IP – Select if we want the main node to connect to the subordinate node using the private IP instead of the public one
7. Always Reconnect – We recommend not to enable it, as Spot instances are ephemeral by nature
8. Restrict Usage – Restrict the use of this Spot Fleet only to tasks that specify the assigned tag, regardless of the global Jenkins configuration
9. Label – The label that identifies this node cloud, useful for assigning which tasks will be executed in the Fleet
10. Number of executors – The number of simultaneous jobs that each Spot Fleet instance will be able to host
11. Max Idle Minutes Before Scaledown – The number of minutes Jenkins will keep an instance active with no pending tasks. If left at 0 Jenkins will never terminate instances, even if they are unused
12. Minimum/Maximum cluster Size – The minimum and maximum quantity of instances we want the Spot fleet to have. Ideally, the minimum should be set to 0 to reduce costs

Once the configuration is saved, we will have a new section in the Jenkins dashboard, which indicates the current status of our Spot Fleet.

Jenkinsfile adaptation

Now that we have our Spot fleet configured, it is time to make some small changes so Jenkins uses this new type of nod. If we want it to be used systematically on a global level, we can disable the execution of tasks on the main node (the only other node in our case) from the node configuration screen.

If, on the other hand, we want certain tasks to be executed in the subordinate nodes, we will have to specify them at Jenkinsfile level, here is an example of how to do it:

#!groovy
pipeline {
//Especificamos agente a nivel de Pipeline
  agent {
    label 'ec2-fleet'
  }

  stages {
        stage('Test') {
          agent {
            dockerfile {
              // Si usamos un agente en concreto, tendremos que volver a especificar la etiqueta
              label 'ec2-fleet'
              filename 'Dockerfile'
              dir 'docker/images/tests'
              args '-u root'
            }
          }
          steps {
            sh 'Hello World'
          }
        }
  }
}

With this last step, we are ready to execute any task on dynamically provisioned nodes within Spot Fleet instances.

From Geko Consulting Cloud, we hope you liked this post and above all that you find it useful, nothing would make us happier.
We invite you to contact us if you need information about the Cloud and DevOps world and keep checking our blog to find other useful publications.

La entrada Dynamic nodes in Jenkins via AWS Spot Fleet se publicó primero en Geko Cloud.