By default, the AWS Simple Email Service (SES) service does not offer us to be able to view logs of your actions / events, we can only see some metrics in Cloudwatch. In Geko we have encountered more than once the need to see SES logs, either to diagnose incidents or to better understand the status of the service. Well, to make this possible, we use AWS Kinesis. We are going to explain how we make it. Let’s begin!

Kinesis configuration

Kinesis is a data collector, processor and transmitter. The first step is to create a Delivery Stream. We access the Kinesis service, Delivery Streams and create a Delivery Stream

In “Source” we choose “Direct PUT” and in Destination “Amazon OpenSearch Service”. There are other destination options such as Redshift, S3, Dynatrace .. All available options appear in the drop-down list.

We add a name to the object “Delivery Stream” and add our OpenSearch. In our case, as we already have an OpenSearch created and operational by simply accessing “Browse”, it has appeared to select it.

We add the index name that we want it to create.

And by default it selects VPC, Subnet and Security Group based on our OpenSearch.

Finally we create the object “Delivery Stream”

SES Configuration

Now we access the SES service, we access “Configuration Sets” and “Create Set”

We add the Configuration Set name and create it.

Once the “Configuration Set” is created we go to “Event Destionation” and “Add Destination” to create one.

We select the type of event that we want and go to the next step

We select “Amazon Kinesis Data Firehose”, we put a name and we select the “Delivery Stream” that we have created previously.

Watch out! Now you have to create an “IAM Role” so that SES can write to Firehose, we have created it like this:

Con esta Policy

And this Trust Relationship

Finally we create the “Configuration Set”

Now, to test it, you can send a test email from SES assigning to our test the “Configuration Set” that we have created. This “Configuration Set” can be applied to any Identity that we have in SES. In fact we are going to do a test by sending a test email. We see the test log that we have generated in our OpenSearch:

And in the Kinesis Delivery monitoring:

From Geko we hope that if you have come this far, this entry is just what you were looking for! We also invite you to read other of our labs posts

La entrada AWS Logs to OpenSearch via Kinesis se publicó primero en Geko Cloud.

• 13.10.3
• 13.9.6
• 13.8.8

It may happen that we have some Gitlab even older than these versions, then we will have to update it to the most recent version through several intermediate versions.

Next, we will show the steps to update Gitlab on Ubuntu 18. We will start with the gitlab-ee 12.7.5-ee example and take it to version 14.4.2-ee.

We consider that Gitlab is previously installed from the official repository with its embedded postgresql and redis services.

Previous steps

If we are working in the Cloud and we have our Gitlab installed on an EC2 instance, it is recommended to create an image of the instance and clone it into a new one to perform these migration steps.

We will also need access to the Linux console of that instance and have root permissions.

Updates + snapshots

We execute the following <apt install> commands in sequence, expecting in each case that they do not give us any error and they end correctly.

sudo apt install gitlab-ee=12.10.14-ee.0
sudo apt install gitlab-ee=13.0.14-ee.0
sudo apt install gitlab-ee=13.1.11-ee.0
sudo apt install gitlab-ee=13.8.8-ee.0
sudo apt install gitlab-ee=13.12.10-ee.0
sudo apt install gitlab-ee=13.12.12-ee.0

Once these steps are finished, we will have to execute the following command in the console, since gitlab needs to migrate its existing projects to the newe hashed_storage system that the following versions use.

sudo gitlab-rake gitlab:storage:migrate_to_hashed

At the end of the previous command, we must restart the instance and reconnect to it.

At this point we have Gitlab 13.12.12

Let’s continue with the updates

sudo apt install gitlab-ee=14.0.11-ee.0

sudo apt install gitlab-ee=14.1.6-ee.0
sudo apt install gitlab-ee=14.2.0-ee.0
sudo apt install gitlab-ee=14.2.6-ee.0
sudo apt install gitlab-ee=14.3.0-ee.0
sudo apt install gitlab-ee=14.3.4-ee.0
sudo apt install gitlab-ee=14.4.0-ee.0
sudo apt install gitlab-ee=14.4.2-ee.0

sudo tail -f /varlog/gitlab/gitlab-rails/production.log

→ Post written by Christian Tagliapietra and Álvaro Abad.

La entrada How to update Gitlab se publicó primero en Geko Cloud.

In this article we will talk about one of the hot tools in the topic of continuous integration and deployment processes “CICD” in Kubernetes, ArgoCD. In recent months, many leading companies in the Internet sector have publicly declared the use of ArgoCD to deploy applications in their clusters. You can see a list here.

To begin with, let’s review what ArgoCD is for and how far ArgoCD’s functionalities go. Then, we will see a typical use case of application deployment using ArgoCD and the advantages of its implementation. Finally, we will comment on the conclusions we have drawn in terms of pros and cons, and we will analyze what other tools complement ArgoCD to further optimize the process of integration and continuous deployment of applications.

What is ArgoCD?

ArgoCD is a tool that allows us adopt GitOps methodologies for continuous deployment of applications in Kubernetes clusters.

The main feature is that ArgoCD synchronizes the state of the deployed applications with their respective manifests declared in git. This allows developers to deploy new versions of the application by simply modifying the git content, either with commits to development branches or by modifying main branch.
Once the code has been modified in git, ArgoCD detects, via webhook or periodic checks every few minutes, that there have been changes in the application manifests. It then compares the manifests declared in git with those applied in the clusters and updates the latter until they are synchronised.

Its user-friendly user interface allows us to visualize very well the content, structure and state of the clusters as well as manipulate resources.

Can ArgoCD automate the entire CI/CD process of an application?

No, ArgoCD takes care of deploying the application once the artifact already exists in a container registry, such as Dockerhub or ECR. This implies that previously the application code has already been tested and containerised in an image. At the end of this article we will talk about what options currently exist to accomplish this previous task in an automated gitops way.

As we have already explained, ArgoCD synchronizes the state of deployed applications with their respective manifests declared in git. But it does not refer to the git repository of the application code itself, but to a separate repository, as best practices suggest, that contains the application’s kubernetes infrastructure code, which can be in the form of helm charts, kustomize application, ksonnet…

To better explain the main benefits offered by ArgoCD let’s see an use case example.

Using ArgoCD

In this example we will see how ArgoCD can deploy either applications developed by third parties, which have their own helm chart maintained by another organization, or one of our own where we have defined the chart ourselves.

For the example we will deploy a monitoring stack consisting of Prometheus, Grafana and Thanos using their helm charts.

ArgoCD deploys the applications through a custom object called Application. This object has as attributes a source and a destination. The source can read several formats, in this exaple our application object will read and deploy helm charts from a chart repository, and charts from a git repository. The destination is the cluster to which the content of the source will be deployed. In the application configuration we can enable that ArgoCD automatically keeps the state of the deployed kubernetes objects synchronized with the configuration indicated in the source (charts/git). This option is very interesting because it ensures that ArgoCD is going to be aware every few minutes that everything is still in sync, by contrast, deploying applications directly with helm commands only ensure synchronization at the time of deployment.

Now that we have explained what the Application object is, for our monitoring-stack, we are going to create four. Why four applications if there will only be 3 services in the stack? (Prometheus, Grafana and Thanos).

ArgoCD also offers the possibility of creating groups of applications that follow the “app of apps pattern” concept. This is an ArgoCD application that deploys other applications and so on recursively. In the case of our monitoring stack, we are going to create a fourth application that will deploy the other 3 applications, the parent application will be called “monitoring-stack”.

To create an application we can define an ArgoCD Application manifest, as indicated in this page of the documentation. We can also do it by command line. But ArgoCD has a great UI that allows you to create applications manually as well, you can see how here.

The “monitoring-stack” application will point its source to a git repository with a Helm chart. This chart will contain the manifests of the other three applications in the “templates” directory in yaml format. These files are Application object definitions that point to the relevant official Helm chart of each service. Using the “values files”, we will be able to deploy different versions in different environments.

Once the templates of the “monitoring-stack” chart have been defined, we will create the parent ArgoCD Application, and in source we will point to the previously mentioned repository. ArgoCD will detect that it is a helm chart and we can indicate the path of the specific values file, for example “prod_values.yaml”.

At the end of the manual configuration of the application, in the user interface we will see how all the created objects are represented, organized in a hierarchical way.

Since the applications are synchronized with our repository, and the charts are parameterized with templates and values. To deploy a new version of any of our applications we will only have to modify the values file through git commits.
ArgoCD will detect the changes in the repository and apply them in the kubernetes cluster through a rolling update deployment.

As a note, using ArgoCD Image Updater can save us from doing this last step manually, or even having to develop a complex pipeline to update the values.yaml file in git when we want to deploy the new image.
This tool periodically queries the latest tags in our image repository looking for new artifacts to deploy. This way, once it has found a new one, it takes care of automating the deployment process by editing the git configuration with the name of the new tag.
It is worth mentioning that there is not yet a stable version of ArgoCD Image Updater but it is expected soon.

In this example we have created an application that points to a repository that creates applications which point to official helm charts, but this hierarchical loop can be extended much further, following the “app of apps pattern”.

Another interesting feature of ArgoCD is that it allows us to deploy applications in different clusters. There are several ways to do this, but the most direct way is through the Application Set resource.
In its manifest we can specify a list of clusters where to deploy simultaneously with different paths of the repository. Since in our repository we can specify different versions for each cluster.

The relative ease of installation that ArgoCD has is another positive point to take into account, here you can consult the steps.

Automation of the entire CICD process with Argo tools

If we want to go a step further and automate the entire CICD process in Kubernetes, we can complement ArgoCD with the rest of the tools presented by the Argo project.
By combining Argo Events, Argo Workflows, ArgoCD and Argo Rollouts, further automation is possible following best practices in the current continuous integration standards.
Victor Farcic explains it very coherently in this video.

As a solution to the added complexity of installing and managing all these Argo project tools, some applications that encompass this entire stack have already been released allowing us to configure the pipelines for integration and deployment from a simplified higher level layer. Below we mention a couple of them, although in this post we are not going to analyze the particular functionalities.

Devtron is an open source tool that installs underneath this Argo stack and other tools and promises that it will let us automate the entire CICD process completely from the user interface. Devtron simplifies the configuration quite a lot as we interact with the internal tools from a high level layer, without manually installing any of them. Although after testing it, we do not believe that the tool is mature enough to be implemented in a production environment for the time being.

Similar to Devtron’s approach, Codefresh also uses all of the Argo stack to automate all integration and deployment. But apart from the fact that the tool is still in early-access, a big difference is that access will be in SaaS format. As we can see in the pricing section, the full automation option will be paid and the price is not mentioned on the website.

Conclusions

ArgoCD is a very useful tool to automate the deployment process using GitOps best practices. Thanks to its implementation, developers can test new versions of applications more quickly and deploy to production safely once testing is complete. In addition, thanks to the auto-sync feature and its beautiful interface, ArgoCD allows us to keep track of the status of applications and their resources deployed in Kubernetes at all times. Combined with the other tools of the Argo project we can automate the entire CICD process (and many other utilities outside the scope of this post) following good practices for current standards.

On the downside, using ArgoCD will introduce an extra layer of complexity to our configuration, as it has many different options, introduces custom objects and concepts we are not familiar with yet. It can be “overkill” if we have a very small cluster with only a handful of applications.

La entrada Firsts steps with ArgoCD se publicó primero en Geko Cloud.

Here at Geko we’ve had to go through our share of operations, IT and security incidents. It’s something that is just bound to happen for a number of reasons (which we’ll get into later), and we’ve come to adopt a lot of habits and practices that makes these far easier to quickly detect, pinpoint, handle and resolve incidents of all kinds. It becomes a lot more manageable once you assume these will happen, and prepare for it. Can’t ever be too cautious when you’re talking about critical infrastructure for your entire team to work on, or your clients to access, it’s a “can’t afford to fail” scenario and you need to be ready for it.

There’s some countermeasures and checks you absolutely need to build into your infrastructure and ecosystem that makes this easier to handle:

• A robust monitoring platform
• A sensible alerting plan
• Service failover
• data snapshotting and backups
• A disaster recovery plan

Let’s go through each one of these and define why these are important.

How do I know when it’s down?

The very first step towards knowing your infrastructure has failed is knowing when it is not failing. You need to build a system that constantly checks every important part of your ecosystem so it notices deviations of that “working correctly” state. Status drift is that will absolutely be the first step of your fail state. Everything just works, then kinda works. Then one day, it doesn’t, and it’s deviated so much from the original state that you need to rebuild everything almost from scratch. You do not want to get here. So you monitor for system abnormalities. If something has to move, monitor it. If something doesn’t have to move, monitor it in case it moves.

Do I just sit someone looking at metrics?

Not much use of a monitoring stack if it doesn’t yell at you when something is going haywire. So as you set up the monitoring infrastructure, at the same pace you add alerting. How you do this is up to you, depends on the urgency of the task. It’s a server that has 70% of its disk full? maybe send a Slack message about it to your IT team. Is the company program server not responding to pings? You probably want it to call someone immediately to turn it on as soon as possible. Depends on the system’s urgency and how big the problem indicator is. Your environment, your priorities.

But that doesn’t keep the service running, does it?

Do you absolutely need a way to keep service up even on the event of failure? Keep a failover system. Maybe you can skip calling about the ping-failing server, or you can move that “change drive” down the priority list because you got another one on the RAID for now. Two is one, one is none. Keep a failover in basically anything important, even if it’s a manual failover system. Have something lined up to quickly change to and keep service.

What do I do when I get a fail alarm?

Let’s get into a disaster scenario for a second. Let’s say you ignored that S.M.A.R.T. alert for one day too long. Your main machine is gone, you can’t just press the “on” button for it to go back up and forget about it, and you somehow had your failover on that machine too, for example a scenario where your proxmox machine bit the dust. Congratulations, you’re facing an incident. So, for this case, which is bound to happen, you’ve prepared backups (hopefully) and you can swap that drive and restore it in. You may lose a day of work, but it’s nothing compared to losing everything and spending hundreds of hours rebuilding your company from scratch. It is especially important to remember the basic rule of backups generally known as the 3-2-1 rule:

• 3 copies of your data
• On 2 different types of storage media
• At least one of them in an offsite location

So that, even in case of an especially bad incident, like a fire, you can just restore from the offsite backup (even though that may take substantially more time depending on your solution). Also, check that your backups work. Please. A backup is not a backup until you test it.

Also, as an extra,  do not pull a Michael Scott on your team.

But I can’t plan for everything that’ll happen, can’t I?

There’s a lot of things that can happen, and unfortunately you can’t predict all of them. Anything can happen and the more complex your infrastructure setup is, the more points of failure there are on it, so you need to prepare as much as possible. Maybe you can’t get all of them nailed down, but the more you plan for, the better, so if something happens, your on-call staff can just walk through a runbook on your documentation and fix the issue without much of a problem. This plan usually includes examples of scenarios that you consider possible based on your infrastructure setup, the elements affected by it, and how to fix this issue.

Sounds like I need one of those.

If an important element of your infrastructure fails, would you get a call, an email, or a Slack notification? Are you sure your backups work? How resistant is your product to a drive failure? If these scenarios sound like a problem in your case, maybe you’d find it useful to Contact us and we’ll talk about setting you up in a better state. You just have to make a choice: are you doing it now, or are you waiting after your next IT incident? Remember Picard’s words on management:

La entrada Disaster recovery: expect the unexpected se publicó primero en Geko Cloud.

Introduction

In this post we are going to see how to solve the following error that we face while working with MySQL databases using AWS EFS service for storage:

ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine

Starting situation

Our use case consisted of a Kubernetes EKS cluster, where we needed to create and destroy new environments for development as quickly as possible. Each of these environments had to contain several MySQL databases, in addition to other tools such as Redis, RabbitMQ, etc.

To do this we decided to use Helm Charts that raised these tools as StatefulSets. At first, we did not specify any StorageClass, so the EKS default was used, which raises EBS gp2 (General Purpose SSD) volumes when creating PersistentVolumeClaims.

The problem with using EBS volumes is that each volume is available in a specific Availability Zone, so if the Kubernetes cluster needs to move a pod from, for example MySQL, it will not be able to lift it unless it is available in another worker node that is in the same Availability Zone as that EBS volume.

For this reason, we decided to use a new StorageClass that uses EFS instead of EBS. In this way, by having the EFS mounted on each worker node in the Kubernetes cluster, the pods could move seamlessly between nodes, even if they were in different Availability Zones.

Error with EFS

Since our use case required creating new environments as quickly as possible, when building a new MySQL database, we also executed an import of data from sql files to have an initial data set by default.

It was at this moment that we began to detect the error, since when importing this initial data, we continuously began to see the following error in the logs:

ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine
ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine
ERROR 1030 (HY000) at line 1744: Got error 168 from storage engine
...

This error began to appear after a certain number of sql instructions executed, and from that moment on, it was repeated until the sql file was finished reading.

Resolution

Upon investigation, we discovered that the problem resided with an EFS service boundary. Specifically, it was the limit of 256 unique locked files. We can see this limit in the quotas described in the AWS documentation:

https://docs.aws.amazon.com/efs/latest/ug/limits.html#limits-client-specific

Because our data import was trying to create more than 256 tables, this limit was reached and the error began to appear. This limit cannot be modified, but we were able to avoid it by modifying the MySQL parameters to, as far as possible, not reach those 256 locked files.

The MySQL parameter to modify is innodb_file_per_table. In our MySQL databases, this parameter was activated by default. This causes an .idb data file to be created for each table in the database instead of creating a single data file. We can modify this parameter in the MySQL configuration file as follows:

[mysqld]
innodb_file_per_table=OFF

https://dev.mysql.com/doc/refman/5.7/en/innodb-file-per-table-tablespaces.html

After disabling this parameter, we did not encounter the error again.

We want to thank the blog ops.tips for their work doing the post that we linked to, since it helped us a lot to understand this error, so that we could find this solution.

https://ops.tips/blog/limits-aws-efs-nfs-locks/

Conclusions

When working with MySQL databases using EFS for storage, we can have errors when reaching the limit of 256 locked files whenever we have schemes with a large number of tables or, ultimately, any system that requires simultaneously locking large amounts of files, such as it would be the case of MongoDB, Oracle, etc.

To avoid reaching this limit, we can disable the MySQL parameter innodb_file_per_table so that a data file is not created for each table.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada Error using AWS EFS for MySQL se publicó primero en Geko Cloud.