What is a container?

Let’s start with a brief definition of what exactly a container is. Containers are a form of operating system virtualisation. A single container can be used to run anything from a small microservice or software process to a larger application.

A container is like a compact box that contains all the dependencies of an application in one place. This not only allows an application to run quickly on a system, but also makes it portable, easy to transfer from one environment to another.

Unlike virtual machines, containers only provide virtualization for the operating system layer and necessary dependencies, making them lighter and more portable, with a significantly lower system load. They can be easily transferred from one environment to another and operate consistently at all times by containing everything necessary for the application to function.

What is Docker?

Docker is an open source containerisation platform launched by Docker Inc. in 2013, i.e. Docker is a technology used to create and run software containers. In essence Docker is a system that allows to build, transfer, deploy and run containers with the applications inside in a very simple and reliable way, guaranteeing a scalable deployment in an efficient way regardless of the host operating system.

Docker streamlines the creation of containers with tools such as dockerfiles, and streamlines the code-like definition of containerised applications through docker-compose. These tools are used to create a development lifecycle, from defining what goes inside a container, to establishing its environment and access to resources. It also helps developers move workloads from their local environment, to testing to production, eliminating inconsistencies and dependencies between environments. As the container being deployed will be the same as the one in which it was developed, the environment will have the same predictable outcome.

What is Kubernetes?

Kubernetes (also known as k8s) is a container platform orchestrator, like Docker. To be more precise, Kubernetes is a set of open source tools for building a scalable, fault-tolerant platform designed to automate and centrally manage containerised applications. Learn more in detail about this tool in our post What is Kubernetes?

So how are Docker and Kubernetes similar or different?

As we can understand from the above, Docker and Kubernetes are related technologies, but they cannot be directly compared with each other. In fact, the two tools complement each other and help build cloud-native or microservices architectures. In other words, Kubernetes builds on top of Docker, using it as an engine to run containers, but it takes care of exactly how these containers are run: configuration, replication, access to system resources, or network communication. It takes care of generating these configurations and communicating them to Docker so that it can apply them.

What is the difference between Docker and Kubernetes? Kubernetes is meant to run on a cluster, whereas Docker runs on a single node by default, with no replication management or container health maintenance, just running a container as declared. So the fundamental difference between Docker and Kubernetes is that Docker is a containerisation platform, meaning that it handles container creation, whereas Kubernetes is a container orchestration platform.

Therefore, the comparison between Kubernetes and Docker is not as simple as creating a list of pros and cons or feature by feature. However, there is a Kubernetes-like orchestration technology developed by Docker Inc. called Docker Swarm, which provides us with Kubernetes-like elements such as replication management or load balancing between containers.

Conclusion

In summary, Docker provides an open standard for packaging and distributing containerised applications and is sufficient to address simple use cases and Kubernetes leverages a broad ecosystem of tools along with continuous integration/continuous deployment (CI/CD) and other DevOps practices to orchestrate large sets of containers from development to production environments.

At Geko Cloud we are at your disposal if you are looking for a partner to implement Kubernetes or Docker, migrate to the cloud, move to a microservices model, or manage your platform in the cloud. Contact us to find out more about our services.

La entrada Kubernetes vs Docker, find out how they are different se publicó primero en Geko Cloud.

In this article we will talk about one of the hot tools in the topic of continuous integration and deployment processes “CICD” in Kubernetes, ArgoCD. In recent months, many leading companies in the Internet sector have publicly declared the use of ArgoCD to deploy applications in their clusters. You can see a list here.

To begin with, let’s review what ArgoCD is for and how far ArgoCD’s functionalities go. Then, we will see a typical use case of application deployment using ArgoCD and the advantages of its implementation. Finally, we will comment on the conclusions we have drawn in terms of pros and cons, and we will analyze what other tools complement ArgoCD to further optimize the process of integration and continuous deployment of applications.

What is ArgoCD?

ArgoCD is a tool that allows us adopt GitOps methodologies for continuous deployment of applications in Kubernetes clusters.

The main feature is that ArgoCD synchronizes the state of the deployed applications with their respective manifests declared in git. This allows developers to deploy new versions of the application by simply modifying the git content, either with commits to development branches or by modifying main branch.
Once the code has been modified in git, ArgoCD detects, via webhook or periodic checks every few minutes, that there have been changes in the application manifests. It then compares the manifests declared in git with those applied in the clusters and updates the latter until they are synchronised.

Its user-friendly user interface allows us to visualize very well the content, structure and state of the clusters as well as manipulate resources.

Can ArgoCD automate the entire CI/CD process of an application?

No, ArgoCD takes care of deploying the application once the artifact already exists in a container registry, such as Dockerhub or ECR. This implies that previously the application code has already been tested and containerised in an image. At the end of this article we will talk about what options currently exist to accomplish this previous task in an automated gitops way.

As we have already explained, ArgoCD synchronizes the state of deployed applications with their respective manifests declared in git. But it does not refer to the git repository of the application code itself, but to a separate repository, as best practices suggest, that contains the application’s kubernetes infrastructure code, which can be in the form of helm charts, kustomize application, ksonnet…

To better explain the main benefits offered by ArgoCD let’s see an use case example.

Using ArgoCD

In this example we will see how ArgoCD can deploy either applications developed by third parties, which have their own helm chart maintained by another organization, or one of our own where we have defined the chart ourselves.

For the example we will deploy a monitoring stack consisting of Prometheus, Grafana and Thanos using their helm charts.

ArgoCD deploys the applications through a custom object called Application. This object has as attributes a source and a destination. The source can read several formats, in this exaple our application object will read and deploy helm charts from a chart repository, and charts from a git repository. The destination is the cluster to which the content of the source will be deployed. In the application configuration we can enable that ArgoCD automatically keeps the state of the deployed kubernetes objects synchronized with the configuration indicated in the source (charts/git). This option is very interesting because it ensures that ArgoCD is going to be aware every few minutes that everything is still in sync, by contrast, deploying applications directly with helm commands only ensure synchronization at the time of deployment.

Now that we have explained what the Application object is, for our monitoring-stack, we are going to create four. Why four applications if there will only be 3 services in the stack? (Prometheus, Grafana and Thanos).

ArgoCD also offers the possibility of creating groups of applications that follow the “app of apps pattern” concept. This is an ArgoCD application that deploys other applications and so on recursively. In the case of our monitoring stack, we are going to create a fourth application that will deploy the other 3 applications, the parent application will be called “monitoring-stack”.

To create an application we can define an ArgoCD Application manifest, as indicated in this page of the documentation. We can also do it by command line. But ArgoCD has a great UI that allows you to create applications manually as well, you can see how here.

The “monitoring-stack” application will point its source to a git repository with a Helm chart. This chart will contain the manifests of the other three applications in the “templates” directory in yaml format. These files are Application object definitions that point to the relevant official Helm chart of each service. Using the “values files”, we will be able to deploy different versions in different environments.

Once the templates of the “monitoring-stack” chart have been defined, we will create the parent ArgoCD Application, and in source we will point to the previously mentioned repository. ArgoCD will detect that it is a helm chart and we can indicate the path of the specific values file, for example “prod_values.yaml”.

At the end of the manual configuration of the application, in the user interface we will see how all the created objects are represented, organized in a hierarchical way.

Since the applications are synchronized with our repository, and the charts are parameterized with templates and values. To deploy a new version of any of our applications we will only have to modify the values file through git commits.
ArgoCD will detect the changes in the repository and apply them in the kubernetes cluster through a rolling update deployment.

As a note, using ArgoCD Image Updater can save us from doing this last step manually, or even having to develop a complex pipeline to update the values.yaml file in git when we want to deploy the new image.
This tool periodically queries the latest tags in our image repository looking for new artifacts to deploy. This way, once it has found a new one, it takes care of automating the deployment process by editing the git configuration with the name of the new tag.
It is worth mentioning that there is not yet a stable version of ArgoCD Image Updater but it is expected soon.

In this example we have created an application that points to a repository that creates applications which point to official helm charts, but this hierarchical loop can be extended much further, following the “app of apps pattern”.

Another interesting feature of ArgoCD is that it allows us to deploy applications in different clusters. There are several ways to do this, but the most direct way is through the Application Set resource.
In its manifest we can specify a list of clusters where to deploy simultaneously with different paths of the repository. Since in our repository we can specify different versions for each cluster.

The relative ease of installation that ArgoCD has is another positive point to take into account, here you can consult the steps.

Automation of the entire CICD process with Argo tools

If we want to go a step further and automate the entire CICD process in Kubernetes, we can complement ArgoCD with the rest of the tools presented by the Argo project.
By combining Argo Events, Argo Workflows, ArgoCD and Argo Rollouts, further automation is possible following best practices in the current continuous integration standards.
Victor Farcic explains it very coherently in this video.

As a solution to the added complexity of installing and managing all these Argo project tools, some applications that encompass this entire stack have already been released allowing us to configure the pipelines for integration and deployment from a simplified higher level layer. Below we mention a couple of them, although in this post we are not going to analyze the particular functionalities.

Devtron is an open source tool that installs underneath this Argo stack and other tools and promises that it will let us automate the entire CICD process completely from the user interface. Devtron simplifies the configuration quite a lot as we interact with the internal tools from a high level layer, without manually installing any of them. Although after testing it, we do not believe that the tool is mature enough to be implemented in a production environment for the time being.

Similar to Devtron’s approach, Codefresh also uses all of the Argo stack to automate all integration and deployment. But apart from the fact that the tool is still in early-access, a big difference is that access will be in SaaS format. As we can see in the pricing section, the full automation option will be paid and the price is not mentioned on the website.

Conclusions

ArgoCD is a very useful tool to automate the deployment process using GitOps best practices. Thanks to its implementation, developers can test new versions of applications more quickly and deploy to production safely once testing is complete. In addition, thanks to the auto-sync feature and its beautiful interface, ArgoCD allows us to keep track of the status of applications and their resources deployed in Kubernetes at all times. Combined with the other tools of the Argo project we can automate the entire CICD process (and many other utilities outside the scope of this post) following good practices for current standards.

On the downside, using ArgoCD will introduce an extra layer of complexity to our configuration, as it has many different options, introduces custom objects and concepts we are not familiar with yet. It can be “overkill” if we have a very small cluster with only a handful of applications.

La entrada Firsts steps with ArgoCD se publicó primero en Geko Cloud.

Before we dive into Kubernetes, first of all, you need to know what container orchestration is. Container technologies have radically changed the landscape of systems and the way software is packaged and deployed. Some of the benefits that containers bring are portability, agility, speed, immutability, and fault isolation.

A container orchestrator is a software that manages and coordinates containers in an automated way. There are several container orchestration tools such as Docker Swarm, Google Container Engine, Amazon ECS, and others. However, in this post, we will focus on Kubernetes, where we will explain the concept, most notable components of the tool and discuss the benefits that Kubernetes can offer, among other points.

Definition – What is Kubernetes?

Originally developed by Google, Kubernetes, or k8s for short, is a container orchestration tool. Kubernetes facilitates the deployment and operation of applications in a microservices architecture. It does this by creating an abstraction layer on top of a cluster of hosts, so development teams can deploy their applications and let Kubernetes manage the deployment of containers.

Platforms that offer Kubernetes

There are several cloud platforms that offer Kubernetes as a service. These platforms make the job easier by providing an interface that makes it easy to deploy applications on Kubernetes without having to worry about configuring and managing the various components of the cluster. Some of them are Amazon EKS, Digitalocean, OpenShift, or Rancher, among others.

Some Kubernetes cluster elements

Kubernetes has a number of features that allow you to provision and deploy your own containerized software. If you’re new to Kubernetes, the terminology can be new, and require some extra learning. Here are some of the basics:

• Pods: The minimum unit that can be deployed with Kubernetes. Each pod can contain one to several containers running on the same shared storage/network resources as the pod.
• Nodes: A physical or virtual machine that hosts the pods that perform the workload of an application. 
• Control plane: The control plane runs the Kubernetes components that provide the core functionalities: exposing the Kubernetes API, scheduling workload deployments, managing the cluster, and direct communications throughout the system. 
• Cluster: A group of nodes working together, which run containerized applications. Clusters consist of master and worker nodes. There can be one or more master nodes and zero or more worker nodes. For example, you can have several nodes in the same cluster running the same containers.  
• Deployment: Kubernetes deployments define the scale at which you want to run your application by allowing you to set the details of how you want pods to be replicated across Kubernetes nodes. Deployments describe the number of identical pod replicas you want to run and the preferred update strategy for updating the deployment, among other things. Kubernetes will track the health of the pods, and remove or add pods as needed to bring the application deployment to the desired state. The deployment provides an abstraction layer that allows pods to be replaced easily and transparently. Pods in a deployment can be located on different nodes.
• Service: A set of pods running the same application or microservice can be grouped together to form a deployment, as we have already seen. Since these pods can be replaced and, due to the dynamic nature of Kubernetes, the different IP addresses of the pods will change, there is a need for a network resource to establish an entry point to these pods. A service abstracts away the underlying network complexity, making it possible to provide a single, unchanging entry point while making the layout of the pods that make up the deployment transparent.
• Ingress controller: It is software that provides reverse proxy functionality, configurable traffic routing, and TLS termination for Kubernetes services. A gateway driver is installed on the cluster, and then gateway rules are configured for a specific service or services.
• Kubectl: a command-line interface that sends a request to the Kubernetes API, either with specific commands, with the contents of a YAML manifest, or with data fetch requests.
• Kubelet: is a service running inside each node, which communicates with the control plane.

There are other Kubernetes components such as daemonsets, namespaces, replicaSets, secrets, and even custom resources. The ones mentioned here are the ones you are most likely to interact with when deploying applications on Kubernetes.

Learn more about when and how to use Kubernetes  

Do you want to know more about k8s? In our Youtube channel, we explain you from a business perspective, what needs it covers, its advantages, and when to use it.

Geko and Kubernetes – Why choose Geko Cloud?

If you are using or considering using containers to streamline building, scaling, and deploying your microservices-based application, Kubernetes gives you the ability to manage the infrastructure using declarative YAML files.

Container lifecycle management with Kubernetes using your deployments and operators along with a DevOps approach allows software development and IT operations to adapt to support the CI/CD channel. At Geko Cloud we are at your disposal if you are looking for a partner to implement Kubernetes, migrate to the cloud, move to a microservices model or manage your platform in the cloud. Contact us to find out more about our services.

La entrada What is Kubernetes? se publicó primero en Geko Cloud.

Today we are going to cover the latter. A solution to deploy a registry inside Kubernetes itself: Harbor.

But what is Harbor?

Harbor is open-source and is part of the Cloud Native Computing Foundation. It implements some basic functionalities like a role-based authentication (with support for LDAP) and also ships with some interesting more advanced features like the ability to upload and store helm charts, perform vulnerability scans on the uploaded images, and much more.

Enough talk, I want a hands-on!

Let’s see how can we implement Harbor in our cluster. I assume you are meeting the following requirements:

• You have a working k8s cluster with ingress-controller. Minikube is acceptable.
• You have helm installed (read how to install helm)
• You have the docker client installed
• 10 spare minutes

IMPORTANT: You should have cert-manager installed in your cluster since Harbor requires to have a valid HTTPS certificate to login with docker from your terminal.

Installing Harbor

We are going to use the official helm chart for this part. To do this, you will need to add the harbor repo first:

helm repo add harbor https://helm.goharbor.io

To customize your deployment you can create a new values.yaml file, replace it with your desired contents:

expose:
  type: ingress
  tls: 
    commonName: "YOUR_CN"
  ingress:
    hosts:
      core: "harbor.YOUR_DOMAIN"
persistence:
  enabled: true
externalURL: "https://harbor.YOUR_DOMAIN"
harborAdminPassword: admin

Now create the namespace and deploy the chart. The values used for the deployment will vary depending on your needs:

kubectl create harbor-system
helm install --wait harbor --namespace harbor-system harbor/harbor -f values.yaml

After install finishes (can take up to 5 minutes) you can go to the domain using your web browser and log in to harbor with the admin username and the password you previously defined. Should end up on this screen:

Pushing your first image

First, you will need to login to the registry from your terminal by using the docker login command and entering your credentials:

docker login https://harbor.YOUR_DOMAIN

*Note that in order to access harbor through this domain you must create the appropiate DNS entry in your DNS manager AND is mandatory to have a valid HTTPS certificate

Click the “NEW PROJECT” button and fill the inputs:

After clicking OK you should be able to see your new project on the main screen.

Now, let’s try to upload a new image to this project. Pull the official nginx image, tag it with your harbor/project and push it:

docker pull nginx
docker tag nginx harbor.YOUR_DOMAIN/myproject/nginx:latest
docker push harbor.YOUR_DOMAIN/myproject/nginx:latest

If everything went fine, you should see the nginx image inside the project you created in harbor:

What’s next?

You have deployed a new private Harbor registry and you already uploaded your first image.

In the next post of this series, we will cover how to create the required credentials for your pods to be able to pull the images from your private Harbor and how to tell Kubernetes when and how to use those credentials.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!

La entrada Harbor: Private docker registry in Kubernetes se publicó primero en Geko Cloud.

Jenkins Kubernetes Pod Template

Last friday we faced with a non very common issue with a customer that use jenkins and kubernetes for their CICD pipelines…

Some of their Jenkins pipelines freezed because the Kubernetes nodes which runs those executions still pending/offline for some reason and every console output shows us the following message:

[Pipeline] node
Still waiting to schedule task
All nodes of label ‘docker-build-xxxxxx’ are offline

Here I attach you a plugin link with very usefull information and configurations about the Kubernetes plugin (we strongly recommend you)

The Problem

We decided to check the Kubernetes cluster in order to inspect the pods execution inside the cicd namespaces to understand what is happening… then we notice that we are not able to talk with the Kubernates API thourgh the kubectl client, all the time we try to do some request to the API we get the following message:

Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]

Bingo! That’s the issue, our Kubernetes certificates we use to connect to the api (and used for etcd as well) has expired. This means a little problem because in order to renew a Kubernetes certificate you must to use a client to connect to the K8S API, but as I told you, the client doesn’t work because the certificates already expired… sooooo

What we gonna do?

Geko to the rescue!

There is a workaround to solve it, basically we must to fake our Kubernetes by destroying (moving) the current certificates manually and then force re-create it using init config into the whole cluster.

We strongly suggest you to previously move all the old certificates in a temporary folder, then force init config and finally reboot the K8S to startup with the new certificates we will use in our client side.

$ cd /etc/kubernetes/pki/ 
$ mkdir -p /tmp/oldcerts/etcd
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} /tmp/oldcerts 
$ mv etcd/* /tmp/oldcerts/etcd
$ kubeadm init phase certs all --apiserver-advertise-address  
$ cd /etc/kubernetes/ 
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/oldcerts 
$ kubeadm init phase kubeconfig all 
$ reboot

After reboot , our cluster will startup using the new certificates and will create a new .kubeconfig file which we will copy locally to use through our kubectl client.

$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

Once we copy the new kubeconfig file , we are fully available to work with our K8S cluster again.

In our case we delete all the orphan pods previously generated inside the “cicd” namespace (we love awk   )

for docker in `kubectl get pods -n cicd | awk {'print $1'}`;do kubectl delete pod $docker -ncicd;done

We hope this post could help you with your Kubernetes expired certificates in order to restore the API connectivity with your clients.

If we can help you somehow don’t hesitate to contact us here.

La entrada Renew your expired Kubernetes certificates se publicó primero en Geko Cloud.

You probably already know something about what Kubernetes is, or you’ve heard of it, but if you don’t quite understand the advantages of this powerful tool, in Geko we want to guide you.

As you may already know, Kuberentes is a container orchestrator that allows you to deploy your production applications to the cloud in an automated way. It was developed by Google and subsequently donated to the community, making it an Open Source project.

With Kubernetes, multiple parts of the same application can run in isolated containers, so if one part of an application becomes inoperative, only that part can be restored instead of the entire server. This saves time and increases productivity when developing applications and putting them into production.

At Geko Cloud we are experts in Kubernetes and the use of microservices architectures, something that addresses both the need to make better use of computing resources and the need to maintain increasingly complex web applications.

What are the advantages of using Kubernetes in your business?

We will tell you some of them:

• Unify the deployment model.
• Increases accountability and autonomy of the development team.
• Achieves high-speed horizontal growth, i.e. scalability, in addition to vertical node autoscaler and HPA.
• DRP or business continuity.
• Kubernetes is a cutting-edge technology, but it has been around long enough to warrant its use, and there is a large community behind it to support it.
• There are a number of cloud-managed K8s platforms that allow you to eliminate complexity.
• It is cloud agnostic, offering scalability between environments and infrastructures.
• It provides tracking and security.

But you also need to know when not to use Kubernetes

As we indicated at the beginning, we want to guide you on when to use or not to use Kubernetes and what to consider when using it.

• You should keep in mind that Kubernetes needs an initial implementation and operational cost.
• It is advisable to implement it as a business model not as a platform.
• Operating a Kuberentes cluster without Cloud as the management layer adds some complexity, if a whole K8S cluster goes down, everything will go down.
• It is a platform that was designed for small stateless apps and their interconnection, not designed to deploy DB infrastructure or other high-risk infrastructures. But with the improvements in high availability and new features it allows more and more structures to be brought in internally. The new stateful support mitigates this.

Learn more about when and how to use Kubernetes in our webinar

Watch it here:

We hope you find it useful to learn more about Kubernetes and we encourage you to subscribe to stay up to date with new content.
At Geko Cloud we are at your disposal if you are looking for a partner to implement Kubernetes, migrate to the cloud, switch to a microservices model or manage your platform in the cloud.

Contact us to learn more about our services, thank you for reading and leave us your comments!

La entrada What is Kubernetes and when to use it se publicó primero en Geko Cloud.

Introduction

Many times to fullfill our daily tasks we find ourselves in the situation of having to build our own tools to solve specific problems or situations. In this note I am going to show you how easy it is to create a plugin for kubectl.

A plugin is simply an executable program that allows you to extend the functionality of kubectl, allowing us to implement our own subcommands.
It is important to mention that we can code it in the language we like the most: Bash, Python, Go, etc…

What do we need?

There are certain conditions that we must meet for our plugin to work:

1. The file name must begin with: kubectl- and must not contain an extension.
2. It must be an executable file.
3. It should be located in our $PATH and this will be all the installation it would require. Pretty simple, no?

We can list available plugins on our system using:

kubectl plugin list

This command will search on our PATH for all executable files that meet the required conditions.

Let’s do it!

Create a test environment

Let’s make a simple plugin, more entertaining than the trivial «Hello World!», that sends the logs of all running pods to a single output to be able to view this output jointly.

To make it more entertaining we will add the name of the pod in colors for each log line in the output. And we can also select with the numeric keys from 1 to N (where N is the nth pod: 1 on-the-fly and see only the output of that pod.

And we will do all this in Bash!

For this demo we are going to use a docker image nginx-log-generator that generates Nginx fake-logs. We will first create a kubernetes deployment that will contain the pods. We create a new file called fake-logs.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-fake-logger
spec:
  template:
    metadata:
      name: logger-1
      labels:
        app: logger
    spec:
      containers:
        - name: logger
          image: kscarlett/nginx-log-generator
  replicas: 3
  selector:
    matchLabels:
      app: logger

Apply the deployment:

kubectl create -f fake-logs.yaml

Let’s check that our deploy is working:

$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
my-fake-logger-66dcffbccd-6kx8b   1/1     Running   4          6d2h
my-fake-logger-66dcffbccd-7q4tm   1/1     Running   1          3d1h
my-fake-logger-66dcffbccd-cff4s   1/1     Running   1          3d1h

$ kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
my-fake-logger   3/3     3            3           6d2h

Create the plugin:

First: We will create our plugin file named kubectl-demo with the following script.

#!/bin/bash
trap ctrl_c INT
function ctrl_c() {
  echo "**************************************** Bye Bye ****************************************"
  for pid in ${PIDS[@]}
  do
      kill -TERM $pid 
  done
  rm $NAMED_PIPE
  rm $sync
  exit 0
}

function colorize() {

  pod=$1
  counter=$2
  # colors from 31 to 39
  pre_colour="\033[3${counter}m"
  post_colour="\033[0m"
  if [ "$colorize_output" = true ]
  then
    colour_pod="${pre_colour}[${pod}]${post_colour}"
  else
    colour_pod="[${pod}]"
  fi
}

function show_logs() {
  local sync="$1"
  grep -E -f $sync $NAMED_PIPE &
}

function banner() {
  echo ""
  echo "==================================================="
  echo "+ Showing logs for:"
  echo "+ $1"
  echo "==================================================="
  echo ""
}

function start_log() {
  show_logs $sync 
  shl_pid=$!
  disown $shl_pid
  PIDS+=$shl_pid" "  
}

function usage() {
  echo "~~~~~~~~~~~"
  echo " U S A G E"
  echo "~~~~~~~~~~~"
  echo "Usage: kubectl demo [option]"
  echo "  options:"
  echo "    --no-colorize: no colorize [default: colorize]"
  echo "    -h: Show this help"
  echo ""
  echo "When running you can use the numbers 1 to N to filter N-pod and only show it's output."
  echo "0 resets and shows all outputs again"
  echo ""
}

NAMED_PIPE="/tmp/my_named_pipe"
PODS=$(kubectl get pods --no-headers=true -o=custom-columns=NAME:.metadata.name)
TOTAL_PODS=$(echo $PODS | sed -s 's/ /n/g' | wc -l)
colorize_output=true

if [[ $@ ]]; then
  case "$@" in
    "--no-colorize")
      colorize_output=false
      ;;
    "-h")
      usage
      exit 1
      ;;
    *) echo "Invalid option"
      exit 1
      ;;
  esac
fi


# create named pipe
if [ ! -p $NAMED_PIPE ]
then
    mkfifo $NAMED_PIPE
    chmod a+rw $NAMED_PIPE
fi

PIDS=()
declare -A pods_index
counter=1
for pod in $(echo $PODS)
do
  colour_pod=""
  colorize $pod $counter
  kubectl logs -f $pod | awk -v pod_name=$colour_pod '{print pod_name" "$0}' > $NAMED_PIPE &
  PIDS+=$!" " # save all PIDs
  pods_index[$counter]=$pod
  counter=$((counter+1))
done

# Trick: Shared memory segment for inter process comunication. 
sync=/dev/shm/syntest-$$  # allocate a shared memory segment name for this pid
echo '' > $sync           # init the shm

start_log
input="*"
re='^[0-9]+$' # we match only numbers

while true
do
  read -t 0.25 -N 1 input
  if  [[ $input =~ $re ]]  && [ "$input" -ge "0" ] && [ "$input" -le "$TOTAL_PODS" ]
  then
    if [ "$input" -eq "0" ]
    then
      banner "All Pods"
      echo $ > $sync    # grep everything
    else
      banner ${pods_index[$input]}
      echo ${pods_index[$input]} > $sync  # grep only pod name
    fi
    kill -SIGTERM $shl_pid
    PIDS=$(echo $PID | sed -e "s/$shl_pid//g" | tr -s " ")  # remove unused pid
    start_log
  fi
done

Second: Set execution permissions

chmod +x kubectl-demo

Third: And last step, we need our script to be accessible from our PATH.

ln -s $PWD/kubectl-demo /usr/local/bin/kubectl-demo

From now on, and with nothing else to do, we can make use of our newly created plugin.

We can check that kubectl is ready to use it by listing the plugins it finds:

$ kubectl plugin list
The following compatible plugins are available:

/home/user/kubectl-demo

Let’s test it!

We show the possible options:

$ kubectl demo -h
~~~~~~~~~~~
 U S A G E
~~~~~~~~~~~
Usage: kubectl demo [option]
  options:
    --no-colorize: no colorize [default: colorize]
    -h: Show this help

When running you can use the numbers 1 to N to filter N-pod and only show it's output.
0 resets and shows all outputs again

As an example of how we can pass parameters to our sub-commands, we have added the option to allow not to paint the output.

Outputs

Default:

[my-fake-logger-66dcffbccd-7q4tm] 214.216.236.37--[02/Jan/2021:10:17:17+0000] 
"GET/moderator/Reduced.cssHTTP/1.1"....
[my-fake-logger-66dcffbccd-6kx8b] 132.56.98.34--[02/Jan/2021:10:19:22+0000] 
"GET/standardization.svgHTTP/1.1"2001908 
[my-fake-logger-66dcffbccd-cff4s] 159.240.103.70--[02/Jan/2021:10:24:45+0000] 
"GET/Virtual.htmHTTP/1.1"2002288"-"....

We can choose which POD want to display with the numbers 1 to 3 in this case:

# 1
===================================================
+ Showing logs for:
+ my-fake-logger-66dcffbccd-6kx8b
===================================================
[my-fake-logger-66dcffbccd-6kx8b]  16.211.208.91 - - [02/Jan/2021:10:34:33 +0000]  "GET /bottom-line%20Advanced/Upgradable/intermediate.svg HTTP/1.1" ....

And we can reset too default view by using number 0 and have all the logs again:

===================================================
+ Showing logs for:
+ All Pods
===================================================
[my-fake-logger-66dcffbccd-7q4tm] 214.216.236.37--[02/Jan/2021:10:17:17+0000] "GET/moderator/Reduced.cssHTTP/1.1"30143"-"....
[my-fake-logger-66dcffbccd-6kx8b]  16.211.208.91 - - ....

As we can see, the script is pretty simple and I invite you to try all its options, modify it to your liking and extend its functionality.

Conclusion

We saw that it is extremely easy to create a kubectl plugin, it is enough to program what our imagination wants and comply with the executable name and location.

Because of this facility, there is no excuse for not making a plugin for practically every specific need. It is also necessary to comment that there is a large number of plugins provided by the community. It’s worth taking a look!

I hope this note has helped you to learn something new and continue to expand your knowledge.

I invite you if you need information about the DevOps world or Kubernetes, contact us and keep checking our blog to find other useful publications.

La entrada Kubectl Plugins in just 3 steps se publicó primero en Geko Cloud.

Nowadays is more and more common for companies to migrate some parts of their infrastructure —or even the entire company— to the cloud. There are two main approaches: Stay as close as possible to the previous architecture by using VMs, or bet for flexibility/scalability/availability and go for a new perspective by using a container orchestrator like Kubernetes. Focusing on this latter approximation (Kubernetes), what you previously had running on an Operative System (which had access to mostly all the conveniences an Operative System provides) now runs in containers. In addition, these containers don’t commonly run directly on specific targets (or machines), but on a cluster. Everything works cool and everything is awesome until you realize you’re not in a friendly and well-known environment anymore. Operative System’s tools like automated tasks or Cron don’t actually follow the containers’ best practices, as they are system-wide (which is the opposite to the app-isolation approach the containerization prays for). Embed those tools (or their behaviors) on your isolated application could become into a big pain full of workarounds and not-that-good practices.

As a full huge ecosystem, Kubernetes provides some of these functionalities so the day-to-day requirements could be addressed. However, brand-new operatives come into the scene when talking about clusters, containers, and a big variety of workflows to handle. Across the following lines, you will learn how Kubernetes have solved the Cron functionality, and most of its tricky, hidden features.

1. How Kubernetes CronJobs actually work

When a CronJob resource is created, what Kubernetes actually does is to register a schedule. Every 10 seconds the CronJob Controller checks if there are matching schedules to take care of. When the proper time arrives a new Job resource is created to handle the task for that specific run. Finally, every Job creates a Pod in order to run the task.

As you may notice this approach differs significantly from the OS one. What is actually happening here is a decoupling between cron-schedules’ handling and the task’s handling (Jobs). This allows the cluster (and also you) to handle ephemeral tasks without loosing control over them.

Moreover, Jobs can create one or more Pods (allowing concurrency/parallelism) and they also ensure the tasks are successfully accomplished. However, this last behaviour could create additional issues as the container also offer a restart-handling feature. This topic will be addressed on the following section.

2. How to configure the advanced functionalities

First of all it must be taken into account that in order to configure a CronJob, every underlying resource could be configured as well. This means a CronJob configuration aggregates its own parameters plus the Job’s properties and also the Pod/container specifications. As most of the common work-flows can be addressed just by having a quick look at the documentation, the aim of this section is to show how to achieve certain tricky functionalities through configuration.

Errors’ handling

When a container stops its execution (because of a failure or after a successful execution) there are a set of actions that could be taken just after, which are defined —as usually— by resource directives. Typical actions are restarting the container (always or only when a failure is detected) or doing nothing. Moreover, the Jobs add another complexity layer which ensures the task is successfully terminated. This means the restarting policy is guaranteed through two different layers that must be properly configured to achieve the desired behavior.

On the container side, the directive is called restart Policy. On the Job side, this policy is “handled” by the directive back-off limit, which specifies the number of allowed failures before giving up and stopping to restart the task. Keeping that in mind, setting up a CronJob able to fail without restarts is as easy as follows.

apiVersion: batch/v1beta1
kind: CronJob
...
spec:
  ...
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          containers:
            ...
          restartPolicy: Never

Overlapping vs. Sequential executions

When talking about a specific CronJob, multiple runs of it could coexist. Depending on the kind of flow the task is characterized for, concurrency could be a way to proceed in order to speed-up processing. There are three available ways to handle how the Jobs are run, which are controlled by the directive concurrency Policy.

1. Allow: Allow overlapping executions.
2. Replace: New executions terminate the previous ones before starting.
3. Forbid: New executions are discarded if a previous one is still running.

While the first one allows concurrence, the two last ones bet for sequential executions. Once again, to stay as close to old-fashioned crons the closer approach is to set the Allow policy. On the other hand, concurrent runs could cause undesired effects if they are not properly managed, and it’s something that should be kept in mind and also be handled with care.

Minimum execution time

As it was previously stated, new task executions could interfere with previous ones depending on how the concurrency directives are set. There’s a property from the container specification which could be useful to deal with the consequences of interrupted runs. A minimum execution time can be set through the terminationGracePeriodSeconds container-property so even if another new task causes an old one to finish, a graceful termination is guaranteed.

3. Operating the CronJobs as a Master

Once the configuration shows up what the CronJob was intended to do, the basic commands can be issued to retrieve the status (scheduling info, running state, logs, …). As on the previous section, the following operatives will cover how to achieve some uncommon features.

Enable/disable CronJobs

The CronJob’s specification has a property called suspend which allows to deactivate them. Temporarily or not, CronJobs can be defined but not being executed at certain times (as their schedule states).

# Disable a CronJob
CRONJOB_NAME=my-cronjob-1
kubectl patch cronjobs $CRONJOB_NAME -p '{"spec" : {"suspend" : true }}'

# Disable ALL CronJobs
kubectl get cronjobs | grep False | cut -d' ' -f 1 | xargs kubectl patch cronjobs -p '{"spec" : {"suspend" : true }}'

Have a look at the following section in order to get further details about the side effects this could cause.

Run CronJobs manually

It’s widely known testing is very useful when detecting undesirable effects. CronJobs can be run manually even when they are suspended (deactivated), so keeping them in that state and running them under testing circumstances could help to validate everything is correct.

CRONJOB_NAME=my-cronjob-1
kubectl create job --from=cronjobs/$CRONJOB_NAME $CRONJOB_NAME-manual-exec-01

4. Showing up the edge cases

After all, there would be no variety if everything were the same way. Every situation has its own particularities and specific characteristics, so when talking about CronJobs that will not be different. Across the following lines, some edge cases will be presented and addressed, so the solutions to them could be reused (or —at least— taken into consideration).

Maximum execution time

On previous sections the execution-time topic was addressed to guarantee a graceful time is conceded before termination. But hey! What about the tasks taking too much time to finish? Agnostic containers’ motivation (and also the Kubernetes one) is to run the tasks until the infinity and beyond. Containers can flow or they can crash, but they should never be terminated as a common practice. Following this principle, there is no way to manage timeouts from the CronJobs/Jobs/Containers specification. So, it’s impossible to handle a maximum execution time? — No, it isn’t! Hence is where the Linux tool-set comes to the rescue. There’s a command called timeout that could be used to run another command until a specific amount of time.

Even though the previous utility could limit the time, the exit code when it does it’s not a successful one so the container will enter into a failure status (that could escalate to a restart if it’s allowed to). On the other hand, the command status could be preserved but then it can’t be known if it was terminated or not. So, how to address them all? On the following snippet can be found a suggested approach that manages to always finish successfully while giving feedback about what actually happened.

containers:
  - name: "my-time-limited-to-10s-container"
    ...
    command: ["bash", "-c"]
    args:
      - /usr/bin/timeout 10 bash -c 'bash -c "comm arg1" && echo OK || echo KO-COMM' || echo KO-TIME

CronJobs not being scheduled after being disabled and enabled again

There is a side effect when a CronJob is disabled, which is that after 100 missed schedules the CronJob will no longer be scheduled. This is already on the docs, but it’s just mentioned as something else not very important. The solution here is to recreate the resource.

CronJobs being scheduled out of their schedule just after being enabled

Another unexpected effect you may find when dealing with CronJobs is that when reactivating one of them —despite that time doesn’t match the schedule— it is immediately executed. This happens because the execution is not just an isolated event, but a time-window until a deadline. This means every missed schedule (because of concurrency or because the CronJob is disabled) will increase a counter (up to a certain number, which is something that could be related to the previous edge case). Then, when the CronJob gets reactivated and it’s allowed to run, the controller realizes there are pending schedules. If the time window to the deadline is still not closed, the CronJob begins to run.

This behavior can be addressed by setting the startingDeadlineSeconds directive to a small value, so the execution window will not match the reactivation time.

CronJobs is not being scheduled

It comes the startingDeadlineSeconds directive could be set to any value, but not all of them are going to cause the desired effect. As previously said the CronJob‘s controller runs every 10 seconds, so every value below ten seconds will make the CronJobs never be scheduled. An issue has been submitted by us to the Kubernetes website project, in order to warn them about this effect. In the next versions you will probably find out it’s already documented, but not for now.

So don’t forget to set startingDeadlineSeconds to a value greater than 10.

Conclusion

As you may have seen, the difference between OS’ crons and Kubernetes‘ crons is bigger than it could be expected at first looking. There are several scenarios on a cluster and many situations to handle, so they are addressed. Sometimes we will be looking for a OS-like behavior, sometimes not, but probably all of them will be possible to achieve. On the other hand, being so multi-purpose could end (as on CronJobs) on more difficult configuration experience, which could be even more difficult when the docs are a little bit short and vague.

Thankfully, you can always count on Geko team -a high-skilled engineering team- who will dig on the topic until getting it easy for you. Don’t forget to come back to the Geko’s blog and check out what’s new in here! The Geko team will be always glad to see you back, and also you should contact us for further information!

Further reading

https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
https://kubernetes.io/docs/concepts/workloads/controllers/job/
https://www.magalix.com/blog/kubernetes-patterns-the-cron-job-pattern
https://medium.com/@bambash/kubernetes-docker-and-cron-8e92e3b5640f
https://medium.com/cloud-native-the-gathering/how-to-write-and-use-kubernetes-cronjobs-3fbb891f88b8
https://medium.com/@hengfeng/what-does-kubernetes-cronjobs-startingdeadlineseconds-exactly-mean-cc2117f9795f

La entrada Kubernetes CronJobs – A deeper look se publicó primero en Geko Cloud.

We are all familiar with the “big three” offer in managed Kubernetes: EKS, GKE and AKS.

But today, apart from the former and a few more who want to play in the same league (Oracle or IBM, for example), there are other less well-known providers that, unable to compete with the service catalog, try to make a name for themselves by offering simplicity and lower prices. One of the benefits of Kubernetes “explosion” is that with it and little more (a load balancer, some block and object storage and a managed database) you can go very far before needing what the big ones offer.

So, when we talk about “simple and cheap” Kubernetes, the formula is to offer the “managed Kubernetes” service (the master nodes are hosted and managed by the provider) without charging for it. And this is exactly what alternatives such as Scaleway, Linode, OVH or DigitalOcean provide.

Testing a managed cluster

At Geko Cloud we are developing a product based on Kubernetes that you will soon hear more about, so we decided to use it to test one of the services on the list (in this case, Digital Ocean Kubernetes Service).

Creating a cluster is very simple: just select the version, the region, the type of node for the pool and give it a name:

In our case, since this was a test, we chose to create basic droplets and rely on the cluster autoscaler in case the application would need more resources.

When the cluster was ready (the process can take about 10-15 minutes), we only had to download the kubeconfig file and launch the application installation process. And then the problems began.

Observed problems

After a while, the API server became unresponsive, responding very slowly or returning several errors. Symptoms:

• Very high response times, sometimes even more than a minute

  bash-5.0# time kubectl get pods -A
  NAMESPACE NAME READY STATUS RESTARTS AGE
  istio-system istio-init-crd-10-1.4.8-86xf9 1/1 Running 1 7m1s
  …
  real 1m14.507s
  user 0m0.174s
  sys 0m0.028s

• Connection errors

  bash-5.0# kubectl get pods -A
  Unable to connect to the server: unexpected EOF

• Timeouts in the TLS handshake

  bash-5.0# kubectl get pods -A
  Unable to connect to the server: net/http: TLS handshake timeout

The problem had to be related with the installation because, before that, everything worked perfectly.

DigitalOcean, like the other providers, does not offer any information about the resources dedicated to the master nodes (number of nodes, node size…) nor access to the control plane logs or monitoring, so there was no way to know what was happening.

A quick search made us realize that we were not alone with this kind of problems, although we also saw that many others were happy users of the service.

Solution

This made us suspect that not all master nodes were created equally, and that the only parameter in the creation process that could make the difference was the configured node pool.

In order to verify our theory, we first tried to create a cluster with a node pool with more nodes (from 3 to 6), with identical results.

So we tried to create a node pool with non-basic nodes (CPU-Optimized in this case). And there we hit the nail on the head: the installation process finished smoothly and the API server responded properly all the time.

Conclusion

It is obvious that nobody gives anything away and that you cannot expect to have a dedicated master node with 16 cores and 64 Gb of RAM paying 20 euros a month for a basic droplet. Anyway, it wouldn’t hurt if they were a little more transparent and informed about the capabilities and limitations of the service they are offering (even if it is free). In this way, more than one would avoid some headaches and frustrations.

So if you are planning to use this service to host an application that is going to make some intensive use of the API server, you’d better opt for non-basic droplets if you don’t want to have performance problems.

And remember that if you need anything we will be happy to listen to you, and you can also check our blog to find other useful posts like this one!

La entrada “Alternative” managed Kubernetes se publicó primero en Geko Cloud.

In this how-to we will cover how to install cert-manager in our cluster as well as how to perform HTTP validation. We will also learn how to create a new certificate for our host by just appending an annotation to our ingress object.

Sounds good! What do I need?

I expect you to meet the following requirements:

• You have a working k8s cluster with nginx ingress-controller
• You have helm installed (read how to install helm)
• You have a domain you own targeting your cluster
• Desire to learn new things

Let’s start!

First of all we need to add the helm chart repository for cert-manager:

helm repo add jetstack https://charts.jetstack.io

Now, we proceed to create the namespace and deploy cert-manager in it:

kubectl create ns cert-manager
helm upgrade --install cert-manager 
  --namespace cert-manager 
  --version v1.0.3 
  jetstack/cert-manager 
  --set installCRDs=true

Just give it a few seconds to finish and you should receive a success message.

Creating a ClusterIssuer

We can use either an Issuer or a ClusterIssuer (namespace vs cluster-scoped). It’s the same exact thing, only changes the scope. The Issuer/ClusterIssuer represents the CA from which we want to get the new certificate, in this case we are using LetsEncrypt.

Note that the ClusterIssuer is actually a pretty simple object. The name and the secret name can be anything:

apiVersion: cert-manager.io/v1alpha3
kind: ClusterIssuer
metadata:
  name: my-cluster-issuer
spec:
  acme:
    email: {{ your email }}
    privateKeySecretRef:
      name: my-cluster-issuer
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
      - http01:
         ingress:
           class: nginx

Issuing a new certificate

We are going to deploy a simple nginx, expose it as a service and then create an ingress for it. So, go ahead:

kubectl create deployment nginx --image nginx:alpine --namespace test
kubectl expose deployment nginx --port 80 --target-port 80 --namespace test

Create a new ingress object to make it accessible through the ingress controller. Remember to replace with your actual domain in all ocurrences:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  labels:
    app: nginx
  name: nginx
  namespace: test
spec:
  rules:
  - host: nginx.{{ your domain }}
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
        path: /
  tls:
  - hosts:
    - nginx.{{ your domain }}
    secretName: nginx-certificate

Apply the new manifest:

kubectl apply -f ingress.yaml

If you try to access now to the host defined in the ingress, you’ll see an error message stating that “Your connection is not private”. To fix this, we are going to issue a new certificate. Edit the ingress that you just created and add the following annotation, replacing the value for whichever was the name you put in your ClusterIssuer:

cert-manager.io/cluster-issuer: my-cluster-issuer

Apply the ingress again (or save changes if your editing it) and now try to check the “cert” objects. You should get something like this:

kubectl get cert -n test

NAME                READY   SECRET              AGE
nginx-certificate   False   nginx-certificate   21s

Now wait for about 1 or 2 minutes and when you check again…

kubectl get cert -n test

NAME                READY   SECRET              AGE
nginx-certificate   True    nginx-certificate   93s

…we got our certificate! Go test it by refreshing the page or just opening a new tab and you should get the Nginx welcome screen. If you inspect the certificate you’ll see that it was indeed issued by LetsEncrypt:

Summarizing

What we saw here is:

• How to install cert-manager
• How to setup a new ClusterIssuer
• How to easily issue a new certificate

It was pretty easy overall, don’t you think so? In fact, when you know what to do you can easily replicate this setup virtually to any cluster in a matter of very few minutes.

Cert-manager will take care of all your certificate renewals. You can also create other Issuers or ClusterIssuers that support other CA or use other validation methods (DNS validation) but this is a more advance topic for the future.

I hope that you enjoyed this how-to and that it will be useful to you. Remember that if you need anything we will be glad to hear from you! Also check our blog for more useful posts like this one!

La entrada How-to: Install cert-manager with HTTP validation in Kubernetes se publicó primero en Geko Cloud.