Based on the current trends and the opinions of experts in the cloud industry, we leave you with some of the trends that could take shape in the cloud market in 2023:

Increased adoption of hybrid cloud:

Hybrid cloud is a combination of public and private cloud, allowing enterprises to use public cloud infrastructure for non-critical workloads and use private cloud infrastructure for non-critical workloads. critical that require a high level of control and security. Hybrid cloud allows companies to get the best of both worlds, which means they can take advantage of the scalability and flexibility of the public cloud and the security and control of the private cloud. Additionally, hybrid cloud enables easy migration of workloads between public and private cloud, which is ideal for businesses that need an agile and flexible IT infrastructure.

Increased focus on cloud security:

With the increase in the amount of data stored in the cloud, security has become an increasingly important concern for businesses. In 2023, companies are expected to invest in cloud security tools and services to ensure the protection of their data. Cloud security solutions can include authentication and authorization, data encryption, security monitoring and analytics, as well as threat prevention tools. Additionally, enterprises can also adopt cloud security practices such as network segmentation, identity and access management, and regular security risk assessment.

Increased adoption of AI and machine learning in the cloud:

The cloud is an ideal platform for processing and analyzing large amounts of data, and the use of AI and machine learning is expected to expand in the cloud to improve business efficiency and productivity.

Greater emphasis on sustainability:

Companies are looking for ways to reduce their environmental impact, and the cloud can be a solution in terms of energy efficiency. Companies are expected to increasingly focus on sustainable cloud solutions in 2023.

Increased adoption of the cloud without a server (Serverless):

The cloud without a server or serverless allows companies to reduce costs and increase efficiency by paying only for the resources they use. Instead of having to pay for an entire infrastructure, companies only pay for the service requests they use. This allows for greater scalability and agility in the deployment of applications and services. In addition, the serverless cloud also eliminates the need to manage and maintain infrastructure, allowing companies to focus on application development and innovation. Serverless cloud adoption is expected to continue to rise in 2023 as more businesses realize the benefits of this technology and the ease of use it offers.

In conclusion, the cloud market continues to evolve rapidly and companies are adopting new strategies to take full advantage of cloud capabilities. Hybrid cloud adoption allows enterprises to combine the best of both worlds, the scalability of the public cloud and the control and security of the private cloud. Additionally, enterprises are increasingly focusing on cloud security, adopting security tools and services to protect their critical data. Artificial intelligence and machine learning are also expanding into the cloud to improve business efficiency and productivity. Additionally, companies are embracing sustainable cloud solutions and focusing on reducing their environmental impact. Finally, the serverless cloud continues to be an important trend, allowing companies to reduce costs and increase efficiency by paying only for the resources they use. Summing up all of the above, these trends are expected to continue into the future and companies will continue to adopt new strategies to reap the benefits of the cloud.

If you want to be in the latest cloud trends, at Geko Cloud we can help you establish new strategies in your hybrid cloud to evolve your business.

You just have to leave us your contact and we will write to you!

La entrada MAIN CLOUD TRENDS FOR 2023 se publicó primero en Geko Cloud.

Engineering Culture is the nice name that has been given to the concept that is trying to take root in new companies to share the knowledge you have, the tools you find useful, or the help that can be given to other projects. It goes with the idea of the DevOps methodology of eliminating silos of knowledge and work and that sharing this knowledge generates benefits for everyone involved.

What are the benefits?

Sharing knowledge with the rest of the internal teams, for example through a slack channel or shared proofs of concept, or with the rest of the world through open projects or blog posts (like this one!) not only helps to get external contributions to improve ideas and processes, but it is also a very good way to gain publicity and quality public outreach, which generates better leads and is positive image-wise.

How can I promote Engineering Culture?

There are many ways to drive this culture in your company to take advantage of its many benefits:

• Create quality content with lessons learned in blogs, videos or social media.
• Dedicate internal time to teaching and training of tools at the technical level by the teams themselves.
• Dedicating resources to creating open source tools for the public
• Dedicating resources to “giving back” to open source projects that you use, either financially or in terms of programming time or bug fixing, or documentation.

Keeping things in-house is no longer in, part of your marketing and lead creation strategy can, and perhaps should, include giving back to the community that supports your product every day!

We invite you to continue reading our articles where you will find a lot of updated information about the sector.

La entrada Engineering Culture: keeping knowledge is not in anymore se publicó primero en Geko Cloud.

Introduction to SSM

SSM is the acronym for Systems Manager, a set of tools that make our lives easier, not only for accessing machines through a virtual console, but also for executing remote commands, extracting records from the equipment, and of course , apply updates.

With the SSM software suite, you can update not only EC2 instances, but also on-prem computers, as well as virtual machines, on-the-edge (IoT) computers such as raspberrys, etc.

Systems Manager is a service that runs an agent on the computer itself (ec2 / on-prem, etc), in the same way as virtual agents in VMware environments, used to extract metrics from the instances, but in the case of SSM the console management is remote, or what is the same, it is divided into different end-points. That is why the agent must communicate with the SSM service over the network. The computers that want to be managed must have not only the agent installed but also network connectivity to reach the AWS SSM endpoints.

In this article we have used a couple of EC2 instances, one Linux and one Windows:

For Linux we have used a RedHat AMI, and in the case of the Windows computer, it is version 2019.

Role for SSM

In order for the computer to communicate with SSM, a role is required. For this reason we are going to generate a new “service role”, accessing the IAM section and selecting the “Roles” option and clicking on the “Create role” button:

We select the “AWS Service” as “Trusted entity”, and below in the “Use case” section, the EC2 service. We select the following and add the policies that are necessary:

To do this, enter the name “ssm” in the search box, and choose from the options that appear the “AmazonSSMManagedInstanceCore”

We assign the name we want and make sure that in the “trusted entities” section, the value of “principal” selected is the service “ec2.amazonaws.com”, the service with which this role is going to be invoked and to which we are giving permission:

The next step will be to add this role to the instances. The easiest way is to mark the instance to be modified from the panel, as shown in the image.

In the dropdown, assign it the role that we just created.

EC2 instances / managed nodes

In our case, we deployed a RedHat machine for the Linux variety for a reason: The official AMIs distributed by RedHat do not have the agent built-in, just like Windows AMIs distributed by Microsoft, or Amazon Linux AMIs do.

In case our distribution is agentless, it can be installed manually. Check the following link to see which distributions are supported:

https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html

In the case of RedHat we will proceed to install it with a command like the following:

sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

There are also binaries available for installing the agents for Windows or MacOS environments.

Fleet Manager

Now that we have the computers with the SSM agent installed and with the associated “service role”, we could access the “fleet manager” section, where the computers managed with SSM are displayed as a record, both EC2 computers and on-prem equipment, etc. This tool is within the SSM options:

The devices managed through SSM are called nodes, which is why, in the view of this service, the registered devices are identified in the “Node Id” column:

The computers that appear are those that are registered in the current region. Keep in mind that AWS SSM is a regional service, which means that if we have an account in several regions, it will be necessary to repeat the process described in this article for each region independently.

Like all software, the SSM agent receives constant improvements. It is therefore recommended to enable automatic updating in the “settings” section:

Baselines

Baselines are statements that define a certain level of compliance with a security policy. In the “patch manager” service there are different “baselines”, but it is highly recommended to create your own one with the configurations that interest us. It is a really simple process.
To access the “baselines” section, go to “Node Management” and select the “Patch Manager” option:

And on the screen that appears, access the “View predefined baselines” button

In the “baselines” view, there are a lot of them, each one for a different operating system. It is at this moment when we are going to create a personalized one. To do this, click on the “Create patch baseline” button.

In the new section, select a name and an OS family that will be associated with the “baseline”:

Define the version of the OS, as well as other aspects that will define the level of compliance:

There are options such as “auto approval”, which allow you to approve the installation of a certain update, after a few days of its publication, or after a certain date, thus avoiding that day 0 patches, not very tested, could break stability of the teams. This would even allow PRE or TEST teams to install patches days before production teams as “canary testing”.

Just as important is the package exceptions section. This prevents the installation of certain packages that can break the service (systems with hand-compiled kernels, Windows packages that don’t get along with any other third-party program, etc).

To add exceptions the following format must be used:

At this moment we already have our own “baseline”. When the “baselines” are your own, to find them later, you have to specify that we are the owner. To do this, in the baseline search engine, specify “Owner: Self” as shown below:

Otherwise the default view only shows standard baselines provided by AWS.

Patch groups

Once we have defined the “baselines”, we are going to define the “patch groups”. A “patch group” establishes the relationship between the “baselines” and the equipment on which they are going to be applied, depending on the value of a tag that we will specify later in the equipment.

To create a “patch group”, it is very simple, we just have to select the “baseline” that we have just created, we display the “Actions” menu and select the “Modify patch groups” option.

A section will open to generate the patch groups, entering a name. In our case “Windows_Production”

Once the name has been entered, taking care to respect the nomenclature, click on the “Add” button and close the wizard.

Tags

Tags are metadata that help us identify who owns a resource, a cost allocation unit, etc. In this case, with the tags we can also associate the nodes to the corresponding patch groups. For this, the same “Patch Group” tag value is always used. Eye! It is case sensitive

We go to the instances section, select one of them, and edit the tags section to add the “Patch Group”:

If when entering the tag, we find an error of the type “‘Patch Group’ is not a valid tag key.”

es necesario deshabilitar la check box de metadata tagging en la configuración de la instancia:

Dejar la casilla siguiente deseleccionada y volver a probar a añadir el tag en el equipo.

Maintainance Windows

Let’s recap. At this moment we already have the instance with the SSM agent installed and the role to interact with the service. We have created our own “baseline”, and then we have assigned it to some “patch groups”. Finally we have “tagged” the instances with a tag that points directly to the “pach group” that interests us.

The maintenance windows are an important section because they not only contain the scheduling of the updates, but also the list of executions.

We are going to proceed by creating a maintenance window through the following section:

We must enter the name for the window, and set the cron. You can establish cron expressions with the typical nomenclature or by selecting the pre-established ones of every hour or every day.

The values of “Duration” and “Stop initiating task” are mandatory, and it is recommended to leave them with values such as the following:

Finally, choose the time zone to match the schedules of the window with the one that interests us.

At the end we will have a list of the scheduled windows, and interesting data such as its next execution:

Patching configuration

At this time we are going to carry out the process that brings together all the previous steps, and that enables the “patch manager” to carry out system updates. To do this, access the “Patch Manager” section and click on “Configure Patching”

Choose the “patch groups” that you want to update:

Choose the maintenance window that was created in the previous step:

And leave the operation that best suits what we want to do. In this case we are going to choose “Scan and Install”

At this moment, in the “dashboard” of “Patch Manager”, we should have a view of equipment pending update. The values in this section will change depending on whether there are pending packages, update failures, etc.
In order to track the actions performed, we can consult them in the “Run Command”

In this section there are two tabs. One with the commands currently running, which will come out with a clock next to it:

And another tab with the history of actions:

If in the process of installing updates, there is any package marked as restart required, the computer will restart immediately. Take this into account to establish the windows according to the needs of the business:

After the installation of the patches, and the reboots if any, in the “dashboard” of “Patch Manager” the instances will now appear as compliant according to the “baselines” established:

Cloudwatch

In the command history console, we can see the executed tasks, as in this case. If we click on the id of the task, we can access its details:

Inside we can see the computers on which the action has been executed, and we can, by selecting one of those computers, see the details of the operations. To do this, select and click on the “View output” button.

The operations are testing all the platforms, for this reason they will first try to execute the patching on a Windows, then on a Linux and finally on a MacOS. Therefore, to see the status of the updates, we choose the platform of the team we are consulting.

In the “Output” window we will see the record, but it is often truncated because it is excessively long:

If we want to consult this data, we can send it to a Cloudwatch “log group”. To do this, first we go to CloudWatch and create a “log group”:

We put a name and specify the retention period.
The second step is to modify the role of the machine so that it has permissions to write to Cloudwatch, so you will have to add an “in-line policy” like this:

We put a name to this “policy” and save it:

Leaving the role of the team as:

We return to the system manager, section “Maintenance Window” and within the tasks (I mark the task and select edit):

And in the “Output Options” section, write the name of the log group where we are going to send it:

We save the changes, and from now on when selecting the output of the tasks, we will see that a link to Cloudwatch appears:

And if we consult that link, it will take us to the output information, but without truncating:

From Geko Cloud Consulting, we hope you liked this post and above all that you find it useful.
If you need information about the Cloud and DevOps world, we invite you to contact us and keep checking our blog to find other useful posts.

La entrada Patching equipment with Patch Manager se publicó primero en Geko Cloud.

As we exploit our infrastructure hosted in the Amazon cloud, our bill also begins to “explode”. This is why at a certain point (or not), it is convenient to start looking for resources so that your bill is not so bulky.

The first step, before anything else, is to be clear about what and how many resources you need for your application and/or infrastructure to have adequate performance and make a forecast about how long you think you will need these resources running at least.

For this estimate, you must have a good knowledge of your architecture and have been working on it for some time, otherwise you will not have enough capacity to predict how your infra will behave in atypical situations.

Let’s get in position. Let’s simulate several ways to save on your EC2 bill.

Reserved Instances

It consists of making a forecast of EC2 instances that you will need at least 1 or 3 years from now (it depends on the commitment time you choose).

A good practice would be to play with the equivalence that instance types have in EC2. For example, reserve 2 t3.small for each t3.medium. In this way, all your hardware will be distributed according to the needs of your infrastructure, and if at any time the number or type of instances changes, that reserved hardware will continue to be distributed and used in the most optimal way.

What is the prize for making this great foresight and commitment? Well, AWS gives you discounts of at least 25-30% and up to 50-60% on the EC2 bill!

The longer your commitment time and the more you pay up front, the bigger the discount.

Spot requests

These instance types allow you to take advantage of unused EC2 capacity in the AWS cloud. Spot Instances are available at a discount of up to 90% compared to on-demand prices.

Where’s the catch? Although it is hard to believe, they are instances that if AWS needs for another user or purpose, they can be removed instantly with just a 2-minute notice (if you find out, turn off the machines and they won’t be removed )

This is why it is recommended to use these instances only for container environments, CI/CD, testing… or combine them in an ASG with your on-demand instances.

Scheduled Instances

An effective practice if you know how to detect what hours or days of the week your infra processes more load than normal.

It is about programming a number of machines during a specific time. If, for example, there is a time of day where there is more traffic than usual and the rest of the day is much quieter, you could schedule one or two more instances to be added to your ASG.And why is it a good option to save? Because you adjust the number of machines to the maximum based on the needs of the moment, and you never pay for what you don’t use.

Saving plans

If you have a more or less large number of instances and you have been with that number for quite some time and you believe that for now it will not vary excessively, personally a Saving Plan is your best option.

What the saving plan basically does is make an analysis of the resources and power that your infra needs (it takes away that work) and proposes savings in exchange for a commitment.

The truth is the most comfortable measure and if you want to spend some time checking for yourself that the prices they offer are realistic, do so and you will see that even if you reserve instances by your own means, the discount they offer is the same or higher.

There are 3 types of Saving Plans:

• Compute: Provides more flexibility and reduces costs by up to 66%. Based on EC2, AWS Fargate, and AWS Lambda usage, regardless of instance family, size, zone, region, OS, and tenancy
• EC2 Instance: Provides the lowest prices and reduces costs by up to 72%, in exchange for the commitment to use individual instance families in a region.
• SageMaker: Reduce costs by up to 64% in SageMaker. They are based on the use of SageMaker regardless of instance family, size, component, and AWS Region.

To apply a Saving Plan, we select this option:

Once inside, we can choose the time and price ratios that we are willing to pay and see what discount is applied, or click here and see the recommendations that AWS proposes based on the last X days.

In the recommendations, depending on the commitment time and the upfront, we will see some prices or others.

My personal proposal about

Once you know all these methods, why not combine them?
You can easily combine Spot Instances with On-Demand, Reserved, and Savings Plans instances to further optimize workload cost with performance.

You can also shut down machines while you are not using them or use the Scheduled Instances method.

Other services where reservations are a good ally
In this post, we wanted to emphasize the reservation of instances and the EC2 Saving Plans, but AWS offers many services where, thanks to the forecast of resources that you will need at least in a specific time, you can save yourself a good pinch.

For example, in the DynamoDB service, you can reserve capacity. It works the same as reserving instances, in exchange for a good discount, you can reserve disk capacity for a specific time with different payment formats in advance.

Another example is Opensearch, the SAS of ELK (Elasticsearch, Logstash and Kibana) that incorporates AWS. In this service you can make reservations for instances, but with a very important peculiarity to take into account.

Unlike other reservations, in this service you have to reserve exactly the type and the exact number of instances that your Opensearch uses. Basically, there is no use reserving 2 t3.small instances for one t3.medium to use its resources, for example. If your Opensearch uses 2 Data Nodes t3.medium and 3 Master Nodes m5.large, you must reserve exactly these types and numbers of instances.
I personally recommend it anyway, the savings are very remarkable.

Close

As the title of this post indicates, the best option you have to save money in all AWS services where machines are involved is forecasting. If your infrastructure has the expected performance and you know its behavior well, you will know how to forecast the number of instances or resources you need in 1 or 3 years.
It is worth making an effort to analyze your resources and taking a few hours or days looking at your ASG and other services, you can easily reduce 50% on your next bill.

If you want help to carry out these practices, do not hesitate to contact us!
The Geko team has experience in different fields, working with DevOps methodology, accompanying our clients throughout the process from migration to cloud management, and adapting 100% to their infrastructures and technologies.

La entrada THE FORECAST. Your best ally to save on your AWS bill se publicó primero en Geko Cloud.

To understand what replica lag is, we first have to put a bit of context. Replication lag occurs in a special type of database instance that we call read replicas. These are created from a source DB instance that acts as the primary database. Updates (inserts of new data) made to the primary database are copied asynchronously to the read replica. Having these types of replicas allows you to reduce the load on the primary DB instance by routing read-type queries to those instances. Replication between these instances occurs through a secure communication channel and asynchronously.

So what is replica lag and how does it affect our instances? Replication lag is the delay that occurs in asynchronous replication between the primary and read databases. To identify the status of the replication, you will need to look at the binlog dump, io_thread, and sql_thread threads. Commands such as show master status, show slave status executed on the respective instances and continuous monitoring help us identify if there is any delay that could affect performance causing data inconsistency.

We cannot eliminate replica lag but we must reduce it as much as possible. Determine if there are configuration differences between the primary and replica instance, too much write workload on the primary instance, too long transactions, incorrect parameter settings, review version changelogs… these are what we usually do at Geko Cloud to decide if any type of update or other type of action should be carried out.

Lag replication example

This is what recently happened to a client who, after updating the version of the main database and its respective mariadb replicas to v.10.4.24, began to notice a deterioration in the replica lag as we can see in the graph.

Our team was investigating the problem and after reviewing the Changelogs from version 10.5.0 to 10.5.17, among other things, we were able to notice that this latest version solved the replication lag problems caused by the previous version. Also, we found that the best thing to do in this case was to upgrade the replicas to version 10.15.17 and leave the primary database at version 10.4.24. To do this, we created two more reading replicas, which we updated to that version in such a way that we could verify that the replica lag was effectively reduced again, as can be seen in the graph.

These graphs were taken from the two read replicas with versions 10.4.24 and 10.5.17 of mariadb in the same time slot to verify that the replica lag had actually decreased.

We hope this can give you an idea of how the Geko Cloud team can help you if you need to deploy your services in a Cloud environment. Just contact us and we will be happy to help you.

La entrada What is replica lag? se publicó primero en Geko Cloud.