Hi friends,

On July 15th, 2019, I messed up bad. Real bad. I wanted to finish a project quickly, and show a quick POC to a customer I was working with. To make a long story short, I pushed a container, to a public repo, containing admin credentials to an AWS account.

I thought of myself as a senior consultant, who delivers, fast, with no mistakes. Man I managed to break that reputation. The silver-lining however, is that I learned my lesson. So deeply so, that I’ve implemented these same principles in every project I’ve done since. So that neither me, or anyone else who works with me has to ever feel the same again.

Lastly, before we being: keep in mind that these principles can be looked at by some people like “over engineering”, or some CISO BS. Believe, it isn’t. Take it from someone who suffered the “burns”, more than one, to tell you that each of these are ESSENTIAL. And together, they make and almost unbreakable environment (everything is breakable, but you’re covering 95% of attach vectors here).

Let’s jump right in:

1. Network Structure: Security starts with how you structure your network. Incorporate both public and private subnets. Key resources should only be placed in private subnets, effectively isolating them from direct internet access and reducing vulnerability. don’t neglect this part! Misconfigurations are extremely vulnerable to human error making private resources easily exposable, avoid at all costs! If something like a NAT gateway as a service makes things too expensive, think of using something like FCK NAT​
   ​
2. Key Sharing: The sharing of SSH keys is a common security pitfall. These are easily leaked (run a small Github search for SSH keys and see how many you come up with). To improve security, consider abandoning SSH entirely in favor of managed backbone connections like AWS SSM Connect, which provide more secure, scalable, and controlled access mechanisms. If you’re interested in learning more about how to achieve this, respond to this email directly and let me know! I can create a dedicate issue explaining these steps!
   ​
3. Secrets Management: It’s vital to handle secrets like API keys or database credentials with care. Avoid committing them into your version control. Utilize a dedicated secret manager to securely store and handle access to these sensitive elements, ensuring they’re encrypted and accessible only to those who truly need them.
   ​
4. Scanning: Implement a routine where every merge commit is scanned for secret leaks and vulnerabilities using tools like gitleaks. Establish strict policies to halt deployments if issues are found in the codebase or in the container images during CI. Don’t have a CI in place yet? 1. Do it! 2. Run these locally before EVERY push.
   ​
5. Zero Trust or VPN: Move away from traditional firewall-based security for accessing internal systems remotely. Instead, adopt a VPN or, ideally, a Zero Trust framework, which authenticates and authorizes every access request to internal resources, no matter where it comes from. If you’ve implemented principle (1) in this list, and built a secure network structure, there’s not much of a way around these. The “easy hack” is a jumpbox / bastion server but these are just as problematic - no audit log and error prone, avoid at all costs, especially in production.
   ​
6. Reading the Bill: Regularly reviewing your cloud bills can help you identify unused or forgotten resources and even expose potential security threats. This is how I discovered a fleet of Monero coin miners running on a client’s forgotten region. This happened due to leaked keys in a public git repo and taken advantage of for almost 6 months before popping on our radar. Make it a habit to read the bill and drill down even to these with seemingly low monthly costs.
   ​
7. Web Application Firewall (WAF): Deploying a WAF can provide a critical defense layer against numerous web-based threats. The default set of rules can cover 80% of randomly sent malicious query attempts which you can then tweak over time to block additional potentially harmful requests.
   ​
8. No Shell Containers: To further minimize the risk of remote code execution attacks, consider deploying containers that lack any form of shell environment. Building your containers with containers starting with FROM: scratch ensures that only the essential application binaries are running, thereby hardening your containers against simple intrusion attempts.

That’s it. There’s a long list of additional steps to go through, but making sure these 8 principles are followed, at least to some extent, can make a world of a difference!

Thanks for reading, as always - feel free to reply to this email with feedback and questions!

Have a great weekend.