When an EKS cluster is created, the user that created the cluster is the owner and is granted permanent admin access to the cluster. I’m going to show you how to use IAM roles and AWS SSO to manage access to EKS. First, create an EKS cluster. Once you have an EKS cluster, you’ll to get kubectl access to it. Note that you’ll need to have aws-cli installed if you don’t have it already. Read more...

If you’re on Apple Silicon, it’s likely that the infrastructure that you deploy your code onto is using amd64 instead of arm. This means that docker images you build on your computer won’t be able to execute on your servers. You might run into an error that looks like this: exec user process caused: exec format error To fix it, you’ll have to rebuild your docker image for amd64. Instead of: Read more...

I learned recently that EKS Clusters will always grant admin privileges to the IAM entity that created the cluster: When you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster’s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn’t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster. Read more...

I just spent many hours over the course of several days figuring out how to set up a VPN between AWS and Digital Ocean with strongSwan. I had to piece together various blog and forum posts to get this to work. Why might you want a Site-to-Site VPN between Digital Ocean and AWS? First, let’s go over why a VPN connection between AWS and Digital Ocean might be useful for you. Read more...

Table of Contents Background Our Stack GitHub Actions Exporter Making Metrics Useful with GHA Supervisor Alerting on Builds that could use Smaller Runners Monitoring the GitHub Actions Queue Synthetic Tests Background I led the migration from Jenkins to GitHub Actions (GHA) at Venmo. We were required to run CI/CD pipelines on our infrastructure. One pain point I had with GitHub Actions self-hosted runners was the complete lack of ability to monitor anything related to GHA on the runners. Read more...

Volumes and bind mounts let you share files between the host machine and container so that you can persist data even after the container is stopped. If you’re running Docker on Linux, you have a third option: tmpfs mounts. When you create a container with a tmpfs mount, the container can create files outside the container’s writable layer. As opposed to volumes and bind mounts, a tmpfs mount is temporary, and only persisted in the host memory. Read more...

The remediation of the recent log4j security vulnerability has solidified an idea in my mind. Services and infrastructure must be regularly deployed to be considered healthy long-term. I’ve read through several Reddit threads where people in companies with a strong devops culture had an easier time doing the dependency update and deploy as opposed to others. The issue people tend to have were updating services that have had infrequent and/or manual deploys. Read more...

Hey everybody, thanks for signing up for my email list. I’m going to try to send something out at least weekly. This first newsletter I want to highlight a very useful Kubernetes feature that I started leveraging recently: port forwarding. You can bind a port from a kubernetes cluster to your development machine you can hit private endpoints on your kubernetes cluster. In the past week I’ve used it to access private endpoints and leverage a datadog agent that was on a cluster but not on my local development machine. Read more...

Table of Contents Background Gotchas with Actions on GitHub Enterprise Server Caching isn’t available GitHub Enterprise Server is behind GitHub Enterprise Cloud Using Public GitHub.com Actions Dockerhub pull rate limiting General Limitations No dropdowns for manually triggered jobs Self-hosted runner default labels You can’t restart a single job of a workflow Slow log output You can’t have actions that call other actions Metrics and observability Workflow YAML Syntax can confusing Closing Background I’ve been leading a team at my dayjob to migrate every engineering team to GitHub Actions (GHA) Read more...

Say you’re a developer working on a feature, and you want to get early feedback from other teammates. How would you go about it? Most companies don’t have a good process for this, and it ends up crippling their ability to ship features and iterate. The process at companies without review app environments: An engineer works on a UI feature that she wants to get some early feedback on how it looks and feels. Read more...