Graceful Shutdowns for Services on Kubernetes
If you’re running services in the cloud or on Kubernetes, your services need to be able to handle shutdowns.
A pod can shut down for many reasons, such as being evicted from scaling down nodes.
There’s a grace period that you can configure with terminationGracePeriodSeconds. It defaults to 30 seconds. You can read more about pod lifecycles here.
You want requests to your API services to be completed within the grace period. Otherwise, the pod will be forcefully shut down and return an error to the caller.
The general pattern is to design your applications and services to handle SIGTERM. One of the first things Kubernetes does is send SIGTERM when it tries to stop a pod. Once the grace period ends, it’ll send a SIGKILL.
Once you get the SIGTERM, your service should stop accepting new requests, complete in-flight requests, and close any stateful resources, such as database connections.
Some major frameworks, such as Spring Boot, already have a flag you can enable to shut down gracefully.
Resources
Master GitHub Actions with a Senior Infrastructure Engineer
As a senior staff infrastructure engineer, I share exclusive, behind-the-scenes insights that you won't find anywhere else. Get the strategies and techniques I've used to save companies $500k in CI costs and transform teams with GitOps best practices—delivered straight to your inbox.
Not sure yet? Check out the archive.
Unsubscribe at any time.