Introduction
Recover accidentally deleted AWS elastic IP. It may not be the most common thing for the average AWS user but for those who work in the cloud industry it’s an everyday task to manage several AWS accounts. Either through a catalog of login links or through a list of account IDs (and their matching credentials), or through a role impersonation manager — the fact is that sooner or later it could happen that one account is mistakenly confused with another and an unwanted action is carried out.
That’s the case it’s being exposed today, in which some tasks were being done on a new account and the created infra there was disposable. It happened that by mistake this new account was mistook with the base account — the Geko’s one — where the elastic IP for the company’s bastion is hosted. Yes, indeed, that key point used to access other infrastructures in a secure way, and which these other infrastructures consider as a safe origin. In other words, that particular IP was used as one of the security measures when verifying the origin of the connections. An IP which was placed in many configurations, and which was not trivial to replace. Well, that IP was released by mistake. In AWS the word “release” is used to refer to the action of removing the IP from the account, leaving it available to the rest of the provider’s users.
It’s not necessary to describe the moments of maximum alert that followed the IP’s release but the possibility of having lost that address forever because it had been assigned to another user, began to take over the tension of the moment. However, there’s a reason to have crisis cabinets, and above all there’s a big importance to be able to count on a team composed by professionals in the area. Brainstorming was fogging up the windows when suddenly a lightning of hope made everything to stop. There was still a chance. AWS had foreseen this catastrophic event and Geko found the way to recover what had always been its own but which it had never valued that much.
Has it happened to you? Would you like to know how it was solved? Keep reading and don’t miss out how this epic story ended!
Recover the AWS elastic IP which was accidentally deleted
Recover accidentally deleted AWS elastic IP. As it was previously stated, once an elastic IP is released it then belongs to the pool of available IPs for the users, so recover it could be impossible in the case it was reassigned. Nonetheless, AWS realized this situation could cause troubles so they developed a procedure to recover from this scenario. Despite this is not something you can do through the web console, recover the lost IP can be achieved by running just one command.
MY_IP=48.151.62.342 aws ec2 allocate-address --domain vpc --address $MY_IP
Yes, you need to know the IP address you want to recover but — if it’s critical enough to make you want to get it again — it should be something you should be able to retrieve.
What else could go wrong?
It happens that in this case there was an additional factor that could add more complexity to the situation, and in fact it did. At Geko — in order to have credentials for the AWS CLI — temporary keys are generated after going through a multi-factor verification process (which can also be revoked at any time), and these credentials are retrieved from the aforementioned bastion. Connecting the dots it can be concluded that the situation had fallen into cyclic dependency since to fix the problem a credentials set needed to be generated while the problem itself was preventing us to do so. Of course there were alternative methods to get out of the loop, but in the end it was enough to use some not-yet-expired credentials we were able to obtain.
Conclusion
Throughout this short — but intense — story, the importance of having a team around you has been shown. Keeping aside pride and rising the alarm was what made the solution to appear in a matter of a few minutes. As it’s commonly said — Two pair of eyes can see more than one.
On the other hand, we learned and reinforced a set of good practices which could be the key next time. They are the following ones.
- Ensure you have some emergency credentials in case something happens in the bastion.
- Ensure the account you’re trying to modify is the one you actually want to modify. It’s better to spend a little bit more time checking twice than to be sorry later.
- Protect valuable resources at all cost, such as creating IAM policies to deny any harmful actions.
We hope this article has helped you to learn something new and continue to expand your knowledge. If you need information about Cloud and the DevOps world, we invite you to contact us and keep checking out our blog in order to find other useful publications. See you soon!
