Introduction to SSM

SSM is the acronym for Systems Manager, a set of tools that make our lives easier, not only for accessing machines through a virtual console, but also for executing remote commands, extracting records from the equipment, and of course , apply updates.

With the SSM software suite, you can update not only EC2 instances, but also on-prem computers, as well as virtual machines, on-the-edge (IoT) computers such as raspberrys, etc.

Systems Manager is a service that runs an agent on the computer itself (ec2 / on-prem, etc), in the same way as virtual agents in VMware environments, used to extract metrics from the instances, but in the case of SSM the console management is remote, or what is the same, it is divided into different end-points. That is why the agent must communicate with the SSM service over the network. The computers that want to be managed must have not only the agent installed but also network connectivity to reach the AWS SSM endpoints.

In this article we have used a couple of EC2 instances, one Linux and one Windows:

For Linux we have used a RedHat AMI, and in the case of the Windows computer, it is version 2019.

Role for SSM

In order for the computer to communicate with SSM, a role is required. For this reason we are going to generate a new “service role”, accessing the IAM section and selecting the “Roles” option and clicking on the “Create role” button:

We select the “AWS Service” as “Trusted entity”, and below in the “Use case” section, the EC2 service. We select the following and add the policies that are necessary:

To do this, enter the name “ssm” in the search box, and choose from the options that appear the “AmazonSSMManagedInstanceCore”

We assign the name we want and make sure that in the “trusted entities” section, the value of “principal” selected is the service “ec2.amazonaws.com”, the service with which this role is going to be invoked and to which we are giving permission:

The next step will be to add this role to the instances. The easiest way is to mark the instance to be modified from the panel, as shown in the image.

In the dropdown, assign it the role that we just created.

EC2 instances / managed nodes

In our case, we deployed a RedHat machine for the Linux variety for a reason: The official AMIs distributed by RedHat do not have the agent built-in, just like Windows AMIs distributed by Microsoft, or Amazon Linux AMIs do.

In case our distribution is agentless, it can be installed manually. Check the following link to see which distributions are supported:

https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html

In the case of RedHat we will proceed to install it with a command like the following:

sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

There are also binaries available for installing the agents for Windows or MacOS environments.

Fleet Manager

Now that we have the computers with the SSM agent installed and with the associated “service role”, we could access the “fleet manager” section, where the computers managed with SSM are displayed as a record, both EC2 computers and on-prem equipment, etc. This tool is within the SSM options:

The devices managed through SSM are called nodes, which is why, in the view of this service, the registered devices are identified in the “Node Id” column:

The computers that appear are those that are registered in the current region. Keep in mind that AWS SSM is a regional service, which means that if we have an account in several regions, it will be necessary to repeat the process described in this article for each region independently.

Like all software, the SSM agent receives constant improvements. It is therefore recommended to enable automatic updating in the “settings” section:

Baselines

Baselines are statements that define a certain level of compliance with a security policy. In the “patch manager” service there are different “baselines”, but it is highly recommended to create your own one with the configurations that interest us. It is a really simple process.
To access the “baselines” section, go to “Node Management” and select the “Patch Manager” option:

And on the screen that appears, access the “View predefined baselines” button

In the “baselines” view, there are a lot of them, each one for a different operating system. It is at this moment when we are going to create a personalized one. To do this, click on the “Create patch baseline” button.

In the new section, select a name and an OS family that will be associated with the “baseline”:

Define the version of the OS, as well as other aspects that will define the level of compliance:

There are options such as “auto approval”, which allow you to approve the installation of a certain update, after a few days of its publication, or after a certain date, thus avoiding that day 0 patches, not very tested, could break stability of the teams. This would even allow PRE or TEST teams to install patches days before production teams as “canary testing”.

Just as important is the package exceptions section. This prevents the installation of certain packages that can break the service (systems with hand-compiled kernels, Windows packages that don’t get along with any other third-party program, etc).

To add exceptions the following format must be used:

At this moment we already have our own “baseline”. When the “baselines” are your own, to find them later, you have to specify that we are the owner. To do this, in the baseline search engine, specify “Owner: Self” as shown below:

Otherwise the default view only shows standard baselines provided by AWS.

Patch groups

Once we have defined the “baselines”, we are going to define the “patch groups”. A “patch group” establishes the relationship between the “baselines” and the equipment on which they are going to be applied, depending on the value of a tag that we will specify later in the equipment.

To create a “patch group”, it is very simple, we just have to select the “baseline” that we have just created, we display the “Actions” menu and select the “Modify patch groups” option.

A section will open to generate the patch groups, entering a name. In our case “Windows_Production”

Once the name has been entered, taking care to respect the nomenclature, click on the “Add” button and close the wizard.

Tags

Tags are metadata that help us identify who owns a resource, a cost allocation unit, etc. In this case, with the tags we can also associate the nodes to the corresponding patch groups. For this, the same “Patch Group” tag value is always used. Eye! It is case sensitive

We go to the instances section, select one of them, and edit the tags section to add the “Patch Group”:

If when entering the tag, we find an error of the type “‘Patch Group’ is not a valid tag key.”

es necesario deshabilitar la check box de metadata tagging en la configuración de la instancia:

Dejar la casilla siguiente deseleccionada y volver a probar a añadir el tag en el equipo.

Maintainance Windows

Let’s recap. At this moment we already have the instance with the SSM agent installed and the role to interact with the service. We have created our own “baseline”, and then we have assigned it to some “patch groups”. Finally we have “tagged” the instances with a tag that points directly to the “pach group” that interests us.

The maintenance windows are an important section because they not only contain the scheduling of the updates, but also the list of executions.

We are going to proceed by creating a maintenance window through the following section:

We must enter the name for the window, and set the cron. You can establish cron expressions with the typical nomenclature or by selecting the pre-established ones of every hour or every day.

The values of “Duration” and “Stop initiating task” are mandatory, and it is recommended to leave them with values such as the following:

Finally, choose the time zone to match the schedules of the window with the one that interests us.

At the end we will have a list of the scheduled windows, and interesting data such as its next execution:

Patching configuration

At this time we are going to carry out the process that brings together all the previous steps, and that enables the “patch manager” to carry out system updates. To do this, access the “Patch Manager” section and click on “Configure Patching”

Choose the “patch groups” that you want to update:

Choose the maintenance window that was created in the previous step:

And leave the operation that best suits what we want to do. In this case we are going to choose “Scan and Install”

At this moment, in the “dashboard” of “Patch Manager”, we should have a view of equipment pending update. The values in this section will change depending on whether there are pending packages, update failures, etc.
In order to track the actions performed, we can consult them in the “Run Command”

In this section there are two tabs. One with the commands currently running, which will come out with a clock next to it:

And another tab with the history of actions:

If in the process of installing updates, there is any package marked as restart required, the computer will restart immediately. Take this into account to establish the windows according to the needs of the business:

After the installation of the patches, and the reboots if any, in the “dashboard” of “Patch Manager” the instances will now appear as compliant according to the “baselines” established:

Cloudwatch

In the command history console, we can see the executed tasks, as in this case. If we click on the id of the task, we can access its details:

Inside we can see the computers on which the action has been executed, and we can, by selecting one of those computers, see the details of the operations. To do this, select and click on the “View output” button.

The operations are testing all the platforms, for this reason they will first try to execute the patching on a Windows, then on a Linux and finally on a MacOS. Therefore, to see the status of the updates, we choose the platform of the team we are consulting.

In the “Output” window we will see the record, but it is often truncated because it is excessively long:

If we want to consult this data, we can send it to a Cloudwatch “log group”. To do this, first we go to CloudWatch and create a “log group”:

We put a name and specify the retention period.
The second step is to modify the role of the machine so that it has permissions to write to Cloudwatch, so you will have to add an “in-line policy” like this:

We put a name to this “policy” and save it:

Leaving the role of the team as:

We return to the system manager, section “Maintenance Window” and within the tasks (I mark the task and select edit):

And in the “Output Options” section, write the name of the log group where we are going to send it:

We save the changes, and from now on when selecting the output of the tasks, we will see that a link to Cloudwatch appears:

And if we consult that link, it will take us to the output information, but without truncating:

From Geko Cloud Consulting, we hope you liked this post and above all that you find it useful.
If you need information about the Cloud and DevOps world, we invite you to contact us and keep checking our blog to find other useful posts.

La entrada Patching equipment with Patch Manager se publicó primero en Geko Cloud.

La entrada ChatGPT: What is it and what are its implications? se publicó primero en Geko Cloud.

By default when creating persistent volumes in AWS the GP2 type is created, however, for some time now we have had GP3 available, which provides us with different advantages, including cost savings. In the following table we can see a comparison of both versions:

Here is a little howto on how we can change the default storageclass to Gp3 step by step.

Create IAM Role

Get OIDC provider url:

Create the following file to define the role by changing 111122223333 to our account ID, region-code to the aws region, and EXAMPLED539D4633E53DE1B71EXAMPLE with the value returned from the previous step:

Create the role with the file created previously:

Attach the policy to the role:

Create the identity provider from the aws console by following these steps:
1. Within IAM in the Access management section go to Identity provider – add provider – OpenID Connect
2. In the Provider URL add the url obtained in point 1.a
3. Click on Get thumbprint
4. In audience put “sts.amazonaws.com” and click on save

Install aws ebs-csi-controller driver

Create the following values.yaml

Add the helm repo:

Install helm chart:

Once all the previous steps have been followed, we will be able to observe how the default storageclass is of the gp3 type, and from now on all the volumes will be created using this type of ebs:

La entrada AWS: Enable EBS GP3 for eks by default se publicó primero en Geko Cloud.

Provisionar nodos slave de jenkins dinámicamente gracias a aws spot fleet

A few weeks ago, a customer contacted us. His request was simple: What can we do to further optimize our automated CI/CD process in Jenkins?

After assesing the needs and the current infrastructure of the client, we decided to change the current paradigm in terms of task execution. Until now Jenkins executed the tasks in its main node (the machine where Jenkins is running), by changing this paradigm to one based on dynamic provisioning of subordinate nodes we would gain the following benefits:

• Cost savings – By moving all the computational effort away from the main node. It can be hosted on a lower performance machine, reducing the overall cost of our infrastructure. The subordinate nodes would only be running during the execution of a task, also saving money while no task is running.
• Security – As we will not execute any code on the main node, we ensure the system’s integrity even if the code has been compromised, and since it is only executed on an ephemeral instance the attacker will be isolated.
• Performance – The subordinate node exists only to execute the task and can dedicate all of the node’s resources to it.

Requirements

For this change, we will perform the following tasks:

• Spot Fleet creation in AWS
• Installation and configuration of the Jenkins plugin
• Jenkinsfile adaptation

For this publication we will make the following assumptions:

• We have an AWS account already set up
• We host Jenkins on an EC2 server
• We have knowledge of some basic Jenkins and AWS concepts

That said, let’s get down to business.

In this article on “Dynamic nodes in Jenkins via AWS Spot Fleet” we will develop all the necessary steps to obtain a good result.

Creation of launch template and spot fleet

An AWS Spot Fleet is nothing more than an AutoScaling Group of EC2 machines but with spot type instances – Amazon’s surplus hardware that nobody is using and that AWS sells cheaper than usual. The first step will be to create the Launch Template that the Spot Fleet will use:

The important items to be filled in will be:

1. Name and description
2. AMI to use – For example Ubuntu 22
3. Type of instance to use – Choose one according to your performance needs
4. Key Pair – We will need to add it later in Jenkins
5. Network – Use the same Subnet as Jenkins if possible
6. Security Group – For the connection between nodes to be established, it is important that we give access to port 22 from the main node. The easiest way is to create a rule that allows access either from the IP of the main node (if we have a Elastic IP assigned) or from its Security Group
7. Storage – Allocate the storage according to your needs
8. User data – For the Jenkins worker to run on the subordinate node, we will need to install JRE, we also installed Docker as a requirement:

#!/bin/bash
# Install docker
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
apt-get update
apt-get install -y docker-ce
usermod -aG docker ubuntu

# Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.21.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

#Install JRE
sudo apt install -y default-jre

Once created we will have a Launch template similar to this one:

Now it is time to create the Spot fleet as such:

In this case, the items to be filled in are:

1. Launch parameters – Select our Launch Template created above
2. Target Capacity – Set total to 0 for now, as it will be controlled by the Jenkins plugin. Enable Maintain Target Capacity
3. Network – Assign the VPC in which the main node is hosted
4. Instance type requirements – Choose the desired instance types in Manually select instance types, a larger quantity gives a better pool from which to start Spot instances
5. Allocation strategy – Capacity Optimized

And that’s it, we have created our Spot Fleet.

Installing and configuring the Jenkins plugin

Before installing the plugin, we will have to create and assign a new IAM role to the instance hosting the main node so that it will be able to manage our Spot fleet:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "ec2:DescribeSpotFleetInstances",
            "ec2:ModifySpotFleetRequest",
            "ec2:CreateTags",
            "ec2:DescribeRegions",
            "ec2:DescribeInstances",
            "ec2:TerminateInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeSpotFleetRequests"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:UpdateAutoScalingGroup"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:ListInstanceProfiles",
            "iam:ListRoles",
            "iam:PassRole"
         ],
         "Resource":"*"
      }
   ]
}

Once this is done, we will install the ec2-fleet-plugin and configure it from Manage Jenkins > Manage Nodes > Configure Clouds > Add New Cloud > Amazon EC2 Fleet.

1. Assign a name
2. AWS Credentials – We leave this blank, since our machine already has the role described above.
3. Region – Select the region in which we host our servers.
4. EC2 Fleet – It will only appear when we assign the region and the role is assigned to the machine correctly. Once filled in, we select our Spot fleet
5. Upload the Private key generated in the Launch template as SSH Username with private key
6. Private IP – Select if we want the main node to connect to the subordinate node using the private IP instead of the public one
7. Always Reconnect – We recommend not to enable it, as Spot instances are ephemeral by nature
8. Restrict Usage – Restrict the use of this Spot Fleet only to tasks that specify the assigned tag, regardless of the global Jenkins configuration
9. Label – The label that identifies this node cloud, useful for assigning which tasks will be executed in the Fleet
10. Number of executors – The number of simultaneous jobs that each Spot Fleet instance will be able to host
11. Max Idle Minutes Before Scaledown – The number of minutes Jenkins will keep an instance active with no pending tasks. If left at 0 Jenkins will never terminate instances, even if they are unused
12. Minimum/Maximum cluster Size – The minimum and maximum quantity of instances we want the Spot fleet to have. Ideally, the minimum should be set to 0 to reduce costs

Once the configuration is saved, we will have a new section in the Jenkins dashboard, which indicates the current status of our Spot Fleet.

Jenkinsfile adaptation

Now that we have our Spot fleet configured, it is time to make some small changes so Jenkins uses this new type of nod. If we want it to be used systematically on a global level, we can disable the execution of tasks on the main node (the only other node in our case) from the node configuration screen.

If, on the other hand, we want certain tasks to be executed in the subordinate nodes, we will have to specify them at Jenkinsfile level, here is an example of how to do it:

#!groovy
pipeline {
//Especificamos agente a nivel de Pipeline
  agent {
    label 'ec2-fleet'
  }

  stages {
        stage('Test') {
          agent {
            dockerfile {
              // Si usamos un agente en concreto, tendremos que volver a especificar la etiqueta
              label 'ec2-fleet'
              filename 'Dockerfile'
              dir 'docker/images/tests'
              args '-u root'
            }
          }
          steps {
            sh 'Hello World'
          }
        }
  }
}

With this last step, we are ready to execute any task on dynamically provisioned nodes within Spot Fleet instances.

From Geko Consulting Cloud, we hope you liked this post and above all that you find it useful, nothing would make us happier.
We invite you to contact us if you need information about the Cloud and DevOps world and keep checking our blog to find other useful publications.

La entrada Dynamic nodes in Jenkins via AWS Spot Fleet se publicó primero en Geko Cloud.

One of the most important mantras to remember when creating a cyber risk model for your company is that you’re never going to have a completely secure application ecosystem. As many penetration tests you run and as many pipeline code quality tests you implement, something’s always left along the way. Sometimes it’s because the return of investment in fixing it is not worth the time, sometimes the tests don’t catch the problem because it’s dependant on the interaction of multiple errors on the system interacting with each other. Complex systems fail, that’s something you cannot avoid. You can control what software you use, what code runs, but a slip-up is only a matter of time.

Why do I apply these security guardrails then?

This is why your development and security plan needs to prioritize containment and isolation: if you assume that something is going to happen, prepare your environment so that the incident has as little impact as possible. A user got a chell to your website? Can’t do much as a well-scoped webserver user. Production database server access? Well, but only for some databases and tables, no administrator rights. Minimize impact area and prepare for damage control, those should be the cornerstones of your fight against cyberattacks.

One of the ways you can implement this security guardrail control is to use specialized security tools that protect against common attacks from threat actors, that in a simple and unattended way will detect, alert and stop external attacks, either from bots or from sophisticated threat actors. Another layer of protection that, if maybe it doesn’t stop every attack, it can save you more than one incident, or cut an exploit chain. Everything without the need of having your security team check every false positive and dedicate their time to more quality work.

What part of my environment do I begin with?

In this case the biggest attack surface we usually need to aproach is websites. Every company today has a web application that they need to use, be it internal or a user-facing corporate site. Whichever it is, it can represent a pivoting point in an incident. To avoid this in web environments exposed to threat actors we need appropiate tools that detect common attacks and stop them. In our case, after an analysis of the available options for this scenario, we’ve found OWASP’s ModSecurity Core Rule Set.

ModSecurity Core Rule Set is a pack of rules maintained by the Open Web Application Security Project Foundation that keeps a detection ruleset aligned with what this world-renowned organization considers the most dangerous attack vectors against modern web applications. These rules are loaded with the ModSecurity module on your web server (nginx, apache2, iis server) and intercepts and analyzes requests and responses that reach your site, then takes alert or block decisions based on its priorities and threat scores.

Should I migrate my current webserver to one with ModSecurity?

Thanks to the modular implementation of ModSecurity this tool can be applied in existing webservers in a simple with a low amount of overhead in the site’s architecture. It’s quickly implemented in the environment that you already have set up and allows you to immediately start processing alerts and blocking attacks. As it doesn’t modify the core functionality of your webserver you can reuse most of your current configuration, making its implementation in your existing environment a low-friction improvement.

How do I enable this in my pre-existing image?

If you don’t have a monolithic webserver set-up, don’t fret, this isn’t a show stopper. Even in a microservice architecture you can still apply the pre-existing solutions created by the open-source community that actively contributes to the Core Rule Set project, you can use drop-in solutions for almost any environment you come from.

Starting up your site with docker-compose? the modsecurity-crs docker image that’s kept updated in the owasp docker repository implements the Core Rule Set set-up on top of the nginx container that you already know and love. Kubernetes fan? You probably already noticed at Geko we are too, and we recently discovered that the nginx-ingress-controller helm chart already implements ModSecurity and Core Rule Set support built-in, you just need to enable a feature flag and without downtime and a couple of tags added to your ingress objects you’ll sleep better knowing you just put one more stone in a cybercriminal’s path to your information.

How do these blocks work, exactly?

ModSecurity handles every request through 4 phases of security controls. In the first layer it applies the CoreRuleSet rules that look for compromise indicators on HTTP request headers. Details like using a PUT where only GETs should be accepted, unwanted protocols or SQL injections in the URL are examples of checks implemented on this first step. After this, if score isn’t high enough to drop the request, it will continue to check for the request body for alarms like malicious data in POST requests. If these checks raise no alarms, the request will be forwarded to the application for processing.

But that’s only the first half of communication, and some requests could get through if they’re obfuscated enough, like base64-encoded requests. To further avoid these potential detection leaks, response data from the application is also checked. Detection of headers with sensitive information, or an abnormally large body field that could be an SQL dump or a binary execution, would never get back to the attacker, as ModSecurity will discard this response before it got back to the threat actor.

How sensitive are these detection rules? Can I tune them?

ModSecurity implements two main elements to control the blocking sensitivity and handle false positives: Score and Paranoia Level. The first one is a stored score assigned to each rule on the CoreRuleSet that defines how critical the incident is: the alarm level from getting a PUT on a GET-only endpoint is not the same as detecting a log4java command injection on a form submit. Score is accumulated accros the communication, and if it goes over the limit, ModSecurity alerts about the incident or cuts the connection.

To control the needed score to alert about an incident or block a request ModSecurity uses Paranoia Level, an environment variable in the ModSecurity setup that, in a level from 1 to 4,, is configured as more or less sensitive to alerts, level 1 being the lowest that requires a score of 5 (the most critical score) to act on a request, and 4 being the highest paranoia level which only requires a score of 1, the lowest in any given rule. The higher the paranoia level, the more secure the environment is, but the more false positives we will get.

But we may not always want to block or allow the same rule level. Sometimes we want to debug without interrupting service. To help with this there’s a special environment variable: the Executing paranoia level. This variable establishes the score and paranoia level required to alert about a rule, but allowing the request to continue. For example, setting paranoia level to 5 and executing paranoia level to 1 will alert of every single rule trigger on the environment, but only interrupt communications for the most critical events that are detected so we can debug and tune rules without service interruption.

That sounds like a lot to take in…

Here at geko we’ve already been fighting with ModSecurity tuning and configuration for a bit. If you want to implement this security suite in your environment but you don’t have the time to handle false positives let’s have a chat and we’ll take care of making your environment more secure so that your team can dedicate their time to their tasks without having to worry about a cybercriminal slipping into the cracks of your infrastructure. ModSecurity will catch and stop problems that you don’t even know exist.

La entrada Modsecurity and the OWASP Core Rule Set: is your web app secure? se publicó primero en Geko Cloud.

General rules:

• Configure a main account dedicated to billing, and then configure subaccounts/projects linking invoice to that account
• In each account/project, configure a budget and an alert if consum exceeds a certain amount (to be defined depending on your usual consume) (AWS, GCP, Azure. This way you’ll be warned during the month… and not at the end receiving bill.
•  If your account has several projects/environments, use labels/tags to be able to identify each resource.
• Export your billing to a database (Athena / Bigquery ) This has a cost (not really expensive if you don’t pass your day querying it) but allows you to have much more details in some cases. For instance in this AWS invoice we can see 595$ of datatransfer out without any details.

 If you go on Athena, we can see that this is mainly due to a loadbalancer.

If we want to save money on that resource, we know where to start.

Finaly review periodically your bills (every 3/6/12 months)

Bill review:

One of best automatic tool you could use is CloudCheckr. After configuring an account in your project it gives you instances that are oversized. Bad thing is that:

• It’s done checking CPU but not RAM. One instance with few CPU usage but full of RAM can’t be resized.
• It doesn’t take into “auto-scalling” instances that are up and down automatically

So another way is to check bill manually. Go to billing details, and review costs, starting by highest one to lowest. Here are some question you can ask yourself for each service

S3:

• Is versioning active and needed? Versioning on a backup’s bucket can be very costly for instance
• All files are regularly accessed? If not you can implement lifecycle to save money in buckets with lots of data
• Files can be deleted after xx days? If so you can also implement a lifecycle

Instances:

• Check CPU / RAM use in your instances and resize your instances if possible
• Is autoscalling correctly configured? You may not need that much instances always up.
• Could you stop some instances at night / on week ends?

Volums:

•  Check unmounted volums and delete them if they are useless

Databases:

• Check CPU / RAM use and resize them if possible

Load balancer:

• Check if loadbalancers could be group in a single one.
• All loadbalancers are usefull? (one loabalancer with a single instance, or returning 200 could be useless)

Those are example for some services. Obviously each case is diferent, so if you’re still lost with all of this, feel free to contact us. We could help you to configure everything.

Instance reservation

Finally, once infrastructure is correctly provisioned (Instances with correct sizing, up/down xx hours per day) we can reserve instances. In other words you commit to keep instances for a certain time, and eventually paying upfront. In exchange you’ll have some discounts. The longest we commit and the most we pay upfront… The most will be our discount.
But we’ll see that in an incoming article.

We hope that this article has been useful to you and we are very attentive that soon we will continue publishing more related content.

From Geko we can adives you to maximize cost savings in your cloud system.

If you would like to get to know us better, check out our services. Please do not hesitate to contact us if you require further information

La entrada Cost Control in Cloud se publicó primero en Geko Cloud.

Property management

A very common practice in cloudformation involves add parameters in templates in order to customize them. In CDK, parameters already exist. However, it is not a good practice to use parameters as you can read in official documentation. AWS recommends centralizing property management in the CDK context.  You should add the properties your app needs in context.json file or equivalent. In CDK there are so many ways to modify the context. In the below image you can see an example:

These properties will be used within the application as shown in the following image:

The stage property define which properties CDK have to use, pre, pro or dev:

You can pass properties on the command line using the -c flag, so it is possible to deploy in any environment just changing the value of stage property. It is a good option, however, the greatest advantage of using CDK remains in use developer tools. So, why not use your favorite developer tools to manage properties? If you like manage your properties with YAML files then use them, if you store your properties in consul, use consul. Feel free to use any tool. Use a familiar tool will reduce learning curve, increase the quality of you developments, and it’s very likely you already implemented a properties management system in your company. Reuse you workflows it’s always a win-win.

If you don’t know how to manage secrets or you are considering new methods, we suggest you use AWS Systems Manager Parameter Store service. Parameter Store allows you to store configuration properties and secrets in a secure way. Retrieving values ​​stored in the parameter store using CDK is pretty easy as you can see in the CDK documentation. Here is an example:

Stacks

In our experience working with CDK we have detected confusion about when and how to use stacks. A very typical question is to define the number of stacks that my CDK application should have. The answer to this question depends on several factors. We are going to see some guidelines that will help you structure your code, but bear in mind these guidelines are not set in stone.

Regarding the number of stacks, you should take into account what resources you will create and if it makes sense to group them within a stack or not. For example, if you need to create an infrastructure for a new software application, it is a good idea to create all resources that the application needs in just one stack. However, if you have to create resources shared for multiple applications or resources critical for your business, then consider creating those resources in their own stacks.

In short, you should group resources that are related to each other in some way, either because they are necessary for an application to work or because they are elements that work together, such as a VPC with its respective subnets and ACL networks. If a specific set of resources are critical for your company then create them into separate stack in order to avoid accidental updates. Then active stack termination protection and monitor it in some way, for instance, using cloudwatch events.

Constructs

Constructs are the base building blocks in CDK. Every resource is generated by a construct. There are different types of constructs, Level 1 (L1), Level 2 (L2) and Level 3 (L3). L1 constructs match 1 to 1 with Cloudformation resources. For each Cloudformation resource,  there is a construct that implements it. L1 constructs are always named with the prefix Cfn. L1 constructs will have the same attributes as their equivalent in Cloudformation.

The L2 are the “curated” constructs. They have default values and they are very easy to use even if you don’t have much knowledge of cloud formation. Finally, the L3 corresponds to patterns that are designed to solve common problems in AWS, such as an API Gateway with a Lambda and a BD Dynamo.

All constructs are maintained by the CDK team, however, you can create your own construct and customize it, adapting it to your needs. We don’t recommend this practice but sometimes it is useful in certain cases, especially when you use an L2 type construct that has default values ​​that don’t fit your needs. In any case, we recommend that you do not abuse this practice. Try to create as few custom constructs as possible.

Imagine the compliance of your company request that buckets cannot be publicly accessible and that all stored objects must be encrypted at rest. One way to satisfy those requirements would be to create your own custom construct and ask the DevOps team to use this construct instead of CDK L2 construct. Below we can see a custom implementation that meets compliance requirements. In the example you can see that the construct inherits from the Construct class instead of the Bucket class. We advise you to do it this way because in this way you can design more extensible constructs.

If you are looking for an alternative way to enforce the compliance in your CDK applications, then you should take a look at using Aspects. Aspects are the way to apply operations to all constructs for a given scope. With an Aspect you can verify that a certain resource is private, or it is already encrypted. In the following image, we can see a stack that creates a private S3 bucket using the custom construct from the previous example:

The kms_key attribute is empty, so objects stored within the bucket will not be encrypted. For this reason, we have created an Aspect called BucketChecker that checks that the buckets of the GekoPrivateBucket class have the encryption_key attribute defined. In case they do not have it defined, an exception will be thrown. In this way, making checks on the created resources is easier.

Resume

We have dealt with various topics, however, we have given priority to two key concepts. The first key concept is how to structure the code of a CDK application, that is, the number of stacks that an application should have, when and how to create a custom construct, how to create tests in CDK or how to deal with identifiers. A good structure helps save technical debt and improves the quality of the delivered code.

The second key concept is the compliance. Throughout both posts, we have provided ideas on how to meet the hypothetical requirements of a compliance. One of the most important lessons we have learned managing our clients’ infrastructure is that meet compliance requirements are really important. Defining a compliance could be painful at first, however the pros outweigh the cons. Compliance allows reducing IT risks, improves infrastructure governance and creates a framework that helps the different teams of a company (SRE, Development, SecOps, …) to interact and reach agreements. For that reason the code you generate in CDK should always keep in mind how to satisfy your company’s compliance.

We hope that you have enjoyed reading  both posts and you have learned something useful, nothing would make us happier. If you need information about Cloud and the DevOps world, we invite you to contact us and keep checking out our blog in order to find other useful publications. See you soon!

La entrada Practical tips to improve your developments in CDK [Part 2] se publicó primero en Geko Cloud.

Practical tips to improve your developments in CDK
The new AWS infrastructure as code framework, CDK, it’s awakening interest in the DevOps world due to its multiple advantages. At Geko we are already using CDK and we can say that it is a very interesting alternative to both Cloudformation and Terraform. For those who want to introduce in the DevOps world or those companies that want to implement a DevOps model, CDK is the perfect tool since, it allows the use of programming languages already used by developers, reducing the learning curve and facilitating its adoption.

However, after using CDK in production we can say that adopting CDK is not as easy as it might appear. For this reason, we have created this post with the intention of explaining some basic concepts of CDK and at the same time sharing with you advices based on our experience, for all those who are considering starting to use CDK or those who are already using it and they want to improve its use.

Naming convention for identifiers

In CDK there are many types of identifiers and each one has a purpose. Our first recommendation is that you inform yourself about how identifiers work and the importance they have in order to create resources in cloudformation. Identifiers in CDKs must be unique within the context in which they are created. This has many notable implications.

For example, the same ID can be used for two constructs that are on two different stacks. However, if you try to use the same ID for two constructs on the same Stack, the application will return an error synthesizing or deploying in AWS. Our advice is to centralize, somehow, the generation of all identifiers, following the same nomenclature. This will help you to avoid repeat identifiers and it will make easier to read the cloudformation templates created with the CDK. You can use a helper function that, given a set of parameters, returns the identifier to be used in the constructs, as shown below.

Try to simplify the naming convention that you define for the identifiers as much as possible, since the logical and physical identifiers generated by CDK are usually very long and include hash codes that make them difficult to read. Always keep in mind that, once you have defined resource identifiers, you should not change them unless it is completely and absolutely necessary. Once you have deployed a resource with an identifier, if the identifier changes, the resource will be replaced with a new one. Imagine you’ve created an RDS instance in production and assigned an ID to it. If you change the RDS instance identifier in your CDK code, the next time you deploy to production the RDS instance will be removed and replaced with a new one. An RDS instance is created in the following CDK code snippet:

By deploying this code, an RDS instance will automatically deployed in a cloudormation stack:

This RDS instance has been created with the identifier Pro-Geko-Cloud-RDS, the values ​​defined for the environment and project attributes are the following:

The cloudformation template generated by CDK is as follows:

The logical name of the RDS instance is ProGekoCloudRDS and a concatenated hash identifier as suffix. We will now modify the static get_identifier method to add the dpt attribute, as shown below:

We also set a value for the static attribute department:

If we execute a cdk diff command we can see that there are pending changes. Changes include replacing the RDS instance with a new instance called ProGekoCloudsalesRDS:

As we have seen, identifiers are vital for a CDK application. Before you start generating source code, define a naming convention and follow it. One way to avoid unwanted identifier changes is to run the CDK diff command in your deploy pipeline, prior to deploying your infrastructure.

Finally, try not to define names for any resource you provision in your application, let the CDK choose the names of the resources. Keep in mind that I’m talking about resource names, not identifiers. Every time you make changes in  a cloudformation resource, it is possible that resource will be replaced due to the changes you want to make are not compatible with the update of the resource. then, it is possible the update fails due to you defined a name for a resource that has to be replaced. AWS CloudFormation doesn’t replace a resource that has a custom name unless that custom name is changed to a different name. If you allow CDK to handle resource naming, in case you make any changes that involve replacing a resource, CDK automatically replaces the resource and assigns it a new name, avoiding errors.

Tagging

Adding tags to AWS resources is a must. The advantages of using tags are many and there aren’t drawbacks. Adding tags in CDK is very easy, as shown in the following image:

Regarding which tags to use, it depends on several factors, however, the tag Name is essential especially because, as we have seen in the previous section, identifiers that CDK assigns to resources are unintelligible. Adding a tag Name to all resources makes it much easier to easily identify resources. We could also use the Owner tag in order to identify which department owns  the infrastructure. The Project tag is a classic example, it is usually used to identify which resources belong to a project. In any case, even if you don’t agree with the recommended tags, you should define a tagging policy aligned with the compliance of your company because it will greatly facilitate the management of your infrastructure, either to locate resources or to identify costs.

Testing

Testing your application is always a best practice and good advice to follow. It helps us find bugs more easily, deliver higher quality software, and it can even help us develop faster if we use development methodologies like TDD. In CDK there are two types of tests that we are going to see.

On the one hand, there are fine-grained tests, that are very similar to unit tests. In a fine-grained test you can perform various checks. For instance, you can check that your application is creating a certain type of resource, for example an RDS instance. You can also verify the properties of your resources. For example, it’s possible to verify that the RDS instance that we have seen in the identifiers section are using MySQL version 8.0.16 as database engine:

To implement test in CDK you will need to import the assertions module. If you use Python you can use Pytest to implement your tests.

On the other hand are the snapshot tests. These tests are used when you want to refactor the code of your CDK application. Use snapshot tests to check you don’t introduce changes when refactoring your applications. In order to run snapshot tests in Python, you have to install the Syrupy python library. Once installed you can start creating your snapshot tests:

This test has two parameters, the first one is the CDK environment and the second one is the snapshot parameter that represents the cloudformation template used as a reference to execute the snapshot test. Snapshot parameter is managed by the Syrupy library, you just have to define it. The test creates the stack to check in the first place. It obtains the cloudformation template of the stack and compares it with the snapshot parameter. If the template doesn’t match with the snapshot parameter, the test fails. When executing a snapshot type test for the first time, you must define the –snapshot-update flag, like this:

When you run this command, Syrupy library will create a folder called __snapshots__ in your tests folder.

The cloudformation template generated by the CdkGekoStack stack will be saved in __snapshot__ folder. The next time you want to run the snapshot type test, you have to run the python -m pytest command without the –snapshot-update flag. If the CdkGekoStack stack returns a template that doesn’t match with the template that Syrupy has already saved in the __snapshot__ folder, the test will fail.

Resume

In this post we have talked about naming convention, tagging and testing. Concepts as basic as important. We promise that in the next post, we are going to talk about more complex concepts such as property management and constructs. So far, we say goodbye. See you!

If you require any further information, feel free to contact us.

La entrada Practical tips to improve your developments in CDK [Part 1] se publicó primero en Geko Cloud.

Kubernetes is a platform that has revolutionized the world of application development, from lifecycle management to scalability architecture. All of thhe components that you used to manage with a fleet of VM instances about as big as your EC2 monthly bill, now lives inside of a group of containers that scale up and down on capacity (and costs!) as necessary. The dream scenario of pay-as-you-go. But as it’s always the case, a new design pattern implies a new cyberattack front to defend.

Let’s explore what can happen in a microservices-oriented model, and how we can find out about it and prevent it from affecting us.

Why does microservice isolation not stop attacks?

This is only partially correct. As everything we find on this industry today, a good part of that depends on how you manage it. For example, after getting rid of the traditional monolith model, an attacker with shell access from your web server (or web container, in this case) doesn’t have automatic access to your database backend, because it’s in a completely different container. Why doesn’t that completely stop attacks that involve accessing your database? Well, the problem is that your web application still needs access to your database backend, and thus still neets to have access credentials mounted on the system to be able to use them.

Does this mean the microservice architecture is useless and I should go back to the monolith?

Not exactly. It is true that a potential attack vector still exists in the microservice architecture, but this atack model is very different compared to the ones used against a monolithic system. It hasn’t solved all of your problms, and it’s not worse than the last design you had, it’s just that your security paradigm has changed. It requires more access layers to execute an attack, which helps you defend using the onion layer model, a more powerful defense than the alternative. Think it as a compromise that requires you to learn a new threat landscape, in exchange of letting your application scale much faster and better as required for a better adjusted cost.

I like the idea. Can I use the defense tools I already know?

At the application level, for example a web application or site, the tools are the same you already know and love. A WordPress is a WordPress, wether you install it in a LAMP stack on a Raspberry Pi or in a horizontally scalable container cluster. At an infrastructure level, however, it’s a different story. New stack, new defense. If you use a managed Kubernetes cluster from a cloud provider like AWS, you probably can’t even access your cluster nodes through SSH. The bruteforce threat disappears, but in exchange, what is an attacker finds your Kubernetes API endpoint and manages to start a malicious container? Would your current monitoring stack alert you?

What parts of the paradigm do I need to protect?

Kubernetes is a container orchestration tool that at first glance can seem like an impossible to assume level of complexity, but if we separate it in its simple purposes to make it into manageable pieces (like with microservices!) we end up discovering that there are two main pillars to protect: resources (what you deploy in your cluster), and infrastructure (how you configure your cluster). Let’s tackle these topics one by one.

The first one, more developer-oriented, covers what do we need to check when we speak about application design and how we deploy it. For example, Kubernetes configures a flat non-segmented network for all pods, where all containers can reach any other container in the cluster. If you don’t allow your VM fleet to live in a flat unmanaged network, you should not let your microservices do either. Check what secrets and credentials can your resources see from production and development. Check what the service account attached to your web pod can do, because it might surprise you. Good permission management will allow you to use the Kubernetes horizontal scalability capability without sacrificing security or isolation.

On the other hand we have the cluster configuration. If an attacker finds your Kubernetes cluster API endpoint, think about what they can do with it. If anonymous authentication is enabled, your attacker can do a very strong first information gathering step that will be very useful for future attacks. Any information discovered about your cluster can be used against you.

I should check all of this, but it’s a big task to handle by hand…

The Open Source community has your back. You don’t need to check all your resources looking for risks that you may not understand, or that don’t even know they exist. Let specialized tools do the hard work for you and focus on makin your infrastructure more secure. The tool list is long for any part of the development cycle you need to check.

Are the images coming out of your CICD pipeline secure? Snyk and Trivy will give you the CVE alerts you need in a neat and categorized report. Doubts about what the third-party stacks on your cluster are doing? Falco is on the lookout for any suspicious behavior inside your cluster to give you actionable alerts. Is your cluster properly configured? kube-bench will show you what security improvements need to be applied to your API server, plus some quality recommendations you can use to step up your security posture. The list of tools to protect your new infrastructure model is endless and they deserve a good look, wether they are community-driven or require a license.

Learning these tools looks like a learning curve

Geko can take that weight off of your shoulders. Instead of investing valuable work time in learning a series of constantly evolving tools and rules, you can let the Geko team the task of finding this information and giving you actionable reports to act upon. Like the idea? Let’s have a chat and find out how we can give you peace of mind while you keep working on what matters to you.

La entrada Kubernetes security – microservices, macrothreats se publicó primero en Geko Cloud.

Recover accidentally deleted AWS elastic IP. It may not be the most common thing for the average AWS user but for those who work in the cloud industry it’s an everyday task to manage several AWS accounts. Either through a catalog of login links or through a list of account IDs (and their matching credentials), or through a role impersonation manager — the fact is that sooner or later it could happen that one account is mistakenly confused with another and an unwanted action is carried out.

That’s the case it’s being exposed today, in which some tasks were being done on a new account and the created infra there was disposable. It happened that by mistake this new account was mistook with the base account — the Geko’s one — where the elastic IP for the company’s bastion is hosted. Yes, indeed, that key point used to access other infrastructures in a secure way, and which these other infrastructures consider as a safe origin. In other words, that particular IP was used as one of the security measures when verifying the origin of the connections. An IP which was placed in many configurations, and which was not trivial to replace. Well, that IP was released by mistake. In AWS the word “release” is used to refer to the action of removing the IP from the account, leaving it available to the rest of the provider’s users.

It’s not necessary to describe the moments of maximum alert that followed the IP’s release but the possibility of having lost that address forever because it had been assigned to another user, began to take over the tension of the moment. However, there’s a reason to have crisis cabinets, and above all there’s a big importance to be able to count on a team composed by professionals in the area. Brainstorming was fogging up the windows when suddenly a lightning of hope made everything to stop.  There was still a chance. AWS had foreseen this catastrophic event and Geko found the way to recover what had always been its own but which it had never valued that much.

Has it happened to you? Would you like to know how it was solved? Keep reading and don’t miss out how this epic story ended!

Recover the AWS elastic IP which was accidentally deleted

Recover accidentally deleted AWS elastic IP. As it was previously stated, once an elastic IP is released it then belongs to the pool of available IPs for the users, so recover it could be impossible in the case it was reassigned. Nonetheless, AWS realized this situation could cause troubles so they developed a procedure to recover from this scenario. Despite this is not something you can do through the web console, recover the lost IP can be achieved by running just one command.

MY_IP=48.151.62.342
aws ec2 allocate-address --domain vpc --address $MY_IP

Yes, you need to know the IP address you want to recover but — if it’s critical enough to make you want to get it again — it should be something you should be able to retrieve.

What else could go wrong?

It happens that in this case there was an additional factor that could add more complexity to the situation, and in fact it did. At Geko — in order to have credentials for the AWS CLI — temporary keys are generated after going through a multi-factor verification process (which can also be revoked at any time), and these credentials are retrieved from the aforementioned bastion. Connecting the dots it can be concluded that the situation had fallen into cyclic dependency since to fix the problem a credentials set needed to be generated while the problem itself was preventing us to do so. Of course there were alternative methods to get out of the loop, but in the end it was enough to use some not-yet-expired credentials we were able to obtain.

Conclusion

Throughout this short — but intense — story, the importance of having a team around you has been shown. Keeping aside pride and rising the alarm was what made the solution to appear in a matter of a few minutes. As it’s commonly said — Two pair of eyes can see more than one.

On the other hand, we learned and reinforced a set of good practices which could be the key next time. They are the following ones.

1. Ensure you have some emergency credentials in case something happens in the bastion.
2. Ensure the account you’re trying to modify is the one you actually want to modify. It’s better to spend a little bit more time checking twice than to be sorry later.
3. Protect valuable resources at all cost, such as creating IAM policies to deny any harmful actions.

We hope this article has helped you to learn something new and continue to expand your knowledge. If you need information about Cloud and the DevOps world, we invite you to contact us and keep checking out our blog in order to find other useful publications. See you soon!

La entrada How to recover an accidentally-deleted AWS elastic IP se publicó primero en Geko Cloud.