Deployment Checklist
To ensure smooth deployments and minimize potential issues, it's crucial to follow a thorough checklist. Here's a high-level overview:
Know what is going to get deployed
Excessive testing; preferably using a pipeline with automated tests
Know when is going to get deployed
By adhering to this checklist, you can enhance control over deployments and reduce the likelihood of complications.
Know what is put in version control
Testing
Pipelines
1. Application Development
Health checks
Containers have Readiness probes
Containers crash when there's a fatal error
Configure a passive Liveness probe
Liveness probes values aren't the same as the Readiness
Apps are independent
The Readiness probes are independent
The app retries connecting to dependent services
Graceful shutdown
The app doesn't shut down on SIGTERM, but it gracefully terminates connections
The app still processes incoming requests in the grace period
The CMD in the Dockerfile
forwards the SIGTERM to the process
Close all idle keep-alive sockets
Fault tolerance
Run more than one replica for your Deployment
Avoid Pods being placed into a single node
Set Pod disruption budgets
Resources utilisation
Set memory limits and requests for all containers
Set CPU request to 1 CPU or below
Disable CPU limits — unless you have a good use case
The namespace has a LimitRange
Set an appropriate Quality of Service (QoS) for Pods
Tagging resources
Resources have technical labels defined
Resources have business labels defined
Resources have security labels defined
Logging
The application logs to stdout
and stderr
Avoid sidecars for logging (if you can)
Scaling
Containers do not store any state in their local filesystem
Use the Horizontal Pod Autoscaler for apps with variable usage patterns
Don't use the Vertical Pod Autoscaler while it's still in beta
Use the Cluster Autoscaler if you have highly varying workloads
Configuration and secrets
Externalise all configuration
Mount Secrets as volumes, not environment variables
2. Governance
Namespace limits
Namespaces have LimitRange
Namespaces have ResourceQuotas
Pod security policies
Enable Pod Security Policies
Disable privileged containers
Use a read-only filesystem in containers
Prevent containers from running as root
Prevent privilege escalation
Network policies
There's a conservative NetworkPolicy in every namespace
Role-Based Access Control (RBAC) policies
Disable auto-mounting of the default ServiceAccount
RBAC policies are set to the least amount of privileges necessary
RBAC policies are granular and not shared
Custom policies
Allow deploying containers only from known registries
Enforce uniqueness in Ingress hostnames
Only use approved domain names in the Ingress hostnames
3. Cluster Configuration
Approved Kubernetes configuration
The cluster passes the CIS benchmark
Disable metadata cloud providers metada API