Here at Geko we’ve had to go through our share of operations, IT and security incidents. It’s something that is just bound to happen for a number of reasons (which we’ll get into later), and we’ve come to adopt a lot of habits and practices that makes these far easier to quickly detect, pinpoint, handle and resolve incidents of all kinds. It becomes a lot more manageable once you assume these will happen, and prepare for it. Can’t ever be too cautious when you’re talking about critical infrastructure for your entire team to work on, or your clients to access, it’s a “can’t afford to fail” scenario and you need to be ready for it.

There’s some countermeasures and checks you absolutely need to build into your infrastructure and ecosystem that makes this easier to handle:

• A robust monitoring platform
• A sensible alerting plan
• Service failover
• data snapshotting and backups
• A disaster recovery plan

Let’s go through each one of these and define why these are important.

How do I know when it’s down?

The very first step towards knowing your infrastructure has failed is knowing when it is not failing. You need to build a system that constantly checks every important part of your ecosystem so it notices deviations of that “working correctly” state. Status drift is that will absolutely be the first step of your fail state. Everything just works, then kinda works. Then one day, it doesn’t, and it’s deviated so much from the original state that you need to rebuild everything almost from scratch. You do not want to get here. So you monitor for system abnormalities. If something has to move, monitor it. If something doesn’t have to move, monitor it in case it moves.

Do I just sit someone looking at metrics?

Not much use of a monitoring stack if it doesn’t yell at you when something is going haywire. So as you set up the monitoring infrastructure, at the same pace you add alerting. How you do this is up to you, depends on the urgency of the task. It’s a server that has 70% of its disk full? maybe send a Slack message about it to your IT team. Is the company program server not responding to pings? You probably want it to call someone immediately to turn it on as soon as possible. Depends on the system’s urgency and how big the problem indicator is. Your environment, your priorities.

But that doesn’t keep the service running, does it?

Do you absolutely need a way to keep service up even on the event of failure? Keep a failover system. Maybe you can skip calling about the ping-failing server, or you can move that “change drive” down the priority list because you got another one on the RAID for now. Two is one, one is none. Keep a failover in basically anything important, even if it’s a manual failover system. Have something lined up to quickly change to and keep service.

What do I do when I get a fail alarm?

Let’s get into a disaster scenario for a second. Let’s say you ignored that S.M.A.R.T. alert for one day too long. Your main machine is gone, you can’t just press the “on” button for it to go back up and forget about it, and you somehow had your failover on that machine too, for example a scenario where your proxmox machine bit the dust. Congratulations, you’re facing an incident. So, for this case, which is bound to happen, you’ve prepared backups (hopefully) and you can swap that drive and restore it in. You may lose a day of work, but it’s nothing compared to losing everything and spending hundreds of hours rebuilding your company from scratch. It is especially important to remember the basic rule of backups generally known as the 3-2-1 rule:

• 3 copies of your data
• On 2 different types of storage media
• At least one of them in an offsite location

So that, even in case of an especially bad incident, like a fire, you can just restore from the offsite backup (even though that may take substantially more time depending on your solution). Also, check that your backups work. Please. A backup is not a backup until you test it.

Also, as an extra,  do not pull a Michael Scott on your team.

But I can’t plan for everything that’ll happen, can’t I?

There’s a lot of things that can happen, and unfortunately you can’t predict all of them. Anything can happen and the more complex your infrastructure setup is, the more points of failure there are on it, so you need to prepare as much as possible. Maybe you can’t get all of them nailed down, but the more you plan for, the better, so if something happens, your on-call staff can just walk through a runbook on your documentation and fix the issue without much of a problem. This plan usually includes examples of scenarios that you consider possible based on your infrastructure setup, the elements affected by it, and how to fix this issue.

Sounds like I need one of those.

If an important element of your infrastructure fails, would you get a call, an email, or a Slack notification? Are you sure your backups work? How resistant is your product to a drive failure? If these scenarios sound like a problem in your case, maybe you’d find it useful to Contact us and we’ll talk about setting you up in a better state. You just have to make a choice: are you doing it now, or are you waiting after your next IT incident? Remember Picard’s words on management:

La entrada Disaster recovery: expect the unexpected se publicó primero en Geko Cloud.

Most juniors that come into an interview have recently finished a degree in IT, whether it’s security, architecture or engineering. Class was mostly what they expected: data structures, some programming, low-level process cost calculation, building virtual machines, service administration, how a datacenter works, data recovery, maybe some security as red-team and blue-team…

Or maybe you’re not that junior anymore, but you’ve been sitting in your current position too long, looking at an unchanging Zabbix panel for a day too long, and you need some fresh career choices? turns out, times have changed. Times have changed a lot.

Exploring uncharted territory

So the first thing you may notice by scavenging LinkedIn is that sysadmins are out of style. You don’t want a systems administrator anymore, or at least not someone who’s only a sysadmin. That’s separated from the development team, what’s called a “silo”, and “silos” are so last-gen. Now, DevOps is in!

In this search for something new, you’ve probably come across the word DevOps. We’ve talked about it in the Geko blog before, but here’s the gist of it: DevOps is essentially a philosophy, a way of working in your team, that makes those “silos” merge together and make development and environment management a lot more agile and fast-moving. No more patch Tuesdays, no more “deploying on Fridays” panic!

Now if you’re like most cases that either try to break into an IT field or want a new challenge, you might find yourself wanting to follow the security path. It’s a growing field, there’s plenty of work, entails exciting experiences every day… seems like it’s all positive! (as long as your client listens to your recommendations, of course), but pretty quickly you notice that cybersecurity is one of the fastest moving fields in the industry. It needs to keep up with changing in all of the IT aspects of a company. You’re in charge of protecting every asset around, whatever it happens to do.

What do these two fields have in common anyway?

If you follow that lead and give the thought a spin, there’s a conclusion you may come to pretty fast: DevOps and security are a very good match! integrating everything for automated checks can take work out of your hands, defining security requirements in every part of the process… it just makes sense that if you choose to follow security culture in the entire development process, it will probably take work out of you after the fact.

That thought recently gave way to a new job opening space that integrated security inside the development culture of DevOps, which we named DevSecOps, because the IT industry is about getting work done, not making up fancy names. At least we didn’t let AWS name it so we aren’t stuck with AWS DSO or something… yet.

So, let’s dive into it. DevSecOps, as its easy to recognize name presents, is a culture, a way of working, where we integrate security into the process of DevOps. This is better explained with an example: Imagine you have a pipeline that tests your code before you build and publish it to test if it works as intended: unit testing, code analysis…

Well, you can do something similar with the rest of the environment, like for example analyzing if the infrastructure the code is gonna run on is well configured and has no security problems by analyzing your Terraform declarations for encryption missing in a volume, or an unrestricted security group. If the alert is raised before the infrastructure is built, you don’t have to fix it later!

I like this idea. How do I implement it?

The entire idea can be summarized into a simple concept: Security isn’t a toggle you can click. You don’t apply security efficiently by installing that software the vendor recommended to you. Security is work culture. A continuous process that is better built into your development pipeline, preferably with automated checks that tell your team that something needs changing. No deployment with security holes in your ops team’s watch. But even then, maybe you don’t need ops intervention, since your developer may just get notified of this security flaw that needs fixing before deployment. This entire thought process is about breaking the concept of separated teams and making departments more connected and cohesive.

Isn’t my infra setup a bit too old-school for this?

Even if your infrastructure seems too legacy to adapt into the DevOps way of working, DevSecOps could be important. Uploading some Ansible playbooks to GitHub? Maybe pass them through a security analysis module when they get pushed out as a pre-commit hook. Built your own docker images for some specific task? Maybe you want to give them a spin through an image analyzer like Snyk or docker scan (did you know Docker had its own vulnerability analyzer? I didn’t!). Are you moving some infrastructure into terraform code? Bridgecrew integrates into pretty much any provider to alert you of that slip of “allow 22/tcp from 0.0.0.0/0”.

Any small step is a big move towards making work easier for your developers in the long run in avoiding technical debt and lifting load from your ops and admin team from these janitor tasks so they can focus on the really important tasks in your organization. Also worth noticing these tools are far cheaper than an incident response spending. Certainly in money, maybe in public image. I’d gladly take some cheap AWS Security Hub alerts over a GDPR fine.

I’m in, let’s chat.

Are you new to all this and you want to learn? Do you want to bring your DevOps knowledge to a new challenging ground? Here at Geko we’re always looking for talent from any expertise level.

Have you been thinking about getting up to speed on your infrastructure security? Well, maybe you should. And we at Geko Cloud could help you with that. At Geko Cloud we are specialists in Internet platforms, cloud infrastructure management and microservices. Contact us without any commitment and we’ll have a chat about stepping up your operations game.

La entrada What is DevSecOps? se publicó primero en Geko Cloud.