Best Practices for Developers

Patterns, antipatterns and best practices for working with Kubernetes.

Managing manifests #

See also: developer quick start, delivery via Argo CD.

Versioning #

• Store manifests in Git - all changes should be trackable. Always used: both for internal services and for production.
• Use version tags for images instead of latest (e.g. v1.0.0, git-sha). The latest tag does not make it clear which version was deployed during an incident and makes rollback more difficult.
• Apply changes via GitOps (Argo CD) for production environments - when the team has a code review process and you need to avoid manual edits in production.

See also: preparing images and manifests, GitOps change flow.

Repository structure #

Choose a structure based on the size of the application and the number of environments.

my-app/
├── k8s/
│   ├── base/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ingress.yaml
│   └── overlays/
│       ├── dev/
│       ├── stage/
│       └── prod/
├── Dockerfile
└── .gitlab-ci.yml

my-app/
├── charts/
│   └── my-app/
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
└── Dockerfile

See also: shturval-cd, import manifests.

Container resources #

Requests and Limits #

Always specify requests and limits:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Why is this important:

• allows the Kubernetes scheduler to correctly place pods;
• necessary for HPA operation;
• prevents resource exhaustion of the node.

See also: requests/limits for deployment, quotas and limits.

Value recommendations #

• Requests - minimum resources guaranteed by the pod. Limits - the maximum that the pod can use. The limits/requests ratio is usually 2–3×.
• Guidelines by type of workload (per one sub):
  • Lightweight API / web server (Go, Node, quick responses): requests: cpu: 50–100m, memory: 128–256Mi, limits: cpu: 200–500m, memory: 256–512Mi.
  • Loaded backend (lots of CPU or memory per request): requests: cpu: 200–500m, memory: 256–512Mi, limits: cpu: 1–2, memory: 1–2Gi.
  • Background workers/queues (constant load): measure peak consumption in dev/stage and set requests slightly below average, limits to the peak with a margin of ~20%.
  • Init containers and one-time jobs: set requests/limits for actual consumption so as not to block nodes.
• If you don’t know the numbers: startwith requests: cpu: 100m, memory: 128Mi and limits: cpu: 500m, memory: 512Mi, then look at usage metrics and raise with stable utilization >70%.

See also: namespace metrics, resource troubleshooting.

Health Checks #

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

Security #

apiVersion: v1
kind: Secret
metadata:
  name: registry-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded>

# Bad
env:
- name: DB_PASSWORD
  value: "my-password"

# Good
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-secret
      key: password

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL

Configuration #

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  config.yaml: |
    database:
      host: db.example.com
      port: 5432    

volumes:
- name: config
  configMap:
    name: app-config
containers:
- name: app
  volumeMounts:
  - name: config
    mountPath: /etc/config

env:
- name: ENVIRONMENT
  value: "production"
- name: LOG_LEVEL
  valueFrom:
    configMapKeyRef:
      name: app-config
      key: log-level

Scaling #

HPA for automatic scaling #

When to use: the load on the service is variable (web, API, workers in turn), and you want to automatically increase/decrease the number of pods based on CPU or custom metrics. For a stable load with a fixed number of pods, HPA may not be used.

Example for a typical web service:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70

See also: HPA on deployment.

How to choose minReplicas and maxReplicas #

• minReplicas: minimum pods that are always running.
  • 1 — valid only for dev/test (if one pod crashes, the service is unavailable).
  • 2 is a typical minimum for production (availability if one pod fails or is updated).
  • 3 or more - if you need high availability or uniform distribution across zones.
• maxReplicas: upper bound to avoid exhausting cluster resources.
  • Evaluate: “maximum pods × limits of one pod” should not exceed the available resources of the nodes. For example, with a limit of 500m CPU and 512Mi of memory on the pods and 10 nodes with 4 CPUs each, you can aim for maxReplicas of the order of 50–60 (with a margin for system pods).
  • For a small cluster, maxReplicas: 10–20 is often sufficient; for high-load services, calculate based on actual capacity.
• target averageUtilization: 70 - typical value: scaling will begin before full load, there is a reserve for bursts. For more “aggressive” scaling, you can use 80; for a smoother one - 60.

Deployment #

Deployment Strategies #

• Rolling Update (by default in Deployment) - one/several pods are replaced with new ones. When to use: almost always by default; low risk, does not require additional resources. It is important to have a readiness probe, otherwise traffic may go to pods that are not yet ready.
• Blue-Green - two identical environments, switching traffic entirely from one to another. When to use: when you need a quick “one-button” rollback and have the resources for two full sets of pods.
• Canary - the new version receives a small share of traffic (5–10%), then the share increases. When to use: when you need a smooth transition to a new version and checking metrics/errors before a complete switchover (often through Ingress or service mesh).

See also: Argo Rollouts strategies.

Recommendations #

• Be sure to configure readiness probes: with a rolling update, new pods do not receive traffic until they become Ready, which reduces the risk of errors during deployment.
• First run the same version of the image in dev/stage (for example, the same tag), then deploy to production - this way you will check manifests and policies, and not just the code.
• If you use Argo CD or a GitOps tool, configure automatic rollback when the application’s health drops (if the tool supports it).

Monitoring and diagnostics #

Check manifests before deployment, and after release, look not only at the status of the pods, but also at the actual resource consumption, restarts and Kubernetes events.

# Validate manifests before deployment
kubectl apply --dry-run=server -f deployment.yaml

# View actual resource usage
kubectl top pods -n production
kubectl top nodes

# Check restarts and HPA status
kubectl get pods -n production -o wide
kubectl get hpa -n production -w

# Diagnose pod that fails to start
kubectl describe pod <pod-name> -n production
kubectl logs <pod-name> -n production --previous
kubectl get events -n production --sort-by='.lastTimestamp'

See also: alerts, viewing logs, preparing for deployment.

Storage #

PersistentVolumes for stateful applications #

When to use: data must survive a restart or re-creation of a pod (database, disk caches, queues with persistent storage). For temporary data and memory cache, PVCs are not needed.

Example for a StatefulSet (for example, a queue or an application with one database per pod):

volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes: ["ReadWriteOnce"]
    resources:
      requests:
        storage: 10Gi

• ReadWriteOnce - the volume is attached to one node, the volume can only be moved to this node. Suitable for most databases and stateful applications.
• ReadWriteMany - the volume can be mounted on several pods at the same time. Only needed if several pods actually write to one volume (shared files, NFS shares). Often slower and more expensive; Don’t use “just in case”.
• Storage size: set based on a realistic estimate of data growth (for example, logs for 7 days, database volume + 20–30% reserve). The underestimated request will then have to be increased (depending on the StorageClass and cluster policies).

See also: PVC creation, StatefulSet during deployment.

Network #

Services for internal access #

When to use: one service accesses another within the cluster (for example, API → DB, frontend → backend). ClusterIP gives a stable DNS name and port, traffic does not go beyond the cluster.

apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  type: ClusterIP
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080

Accessing the service from another pod: http://my-app or http://my-app.<namespace>.svc.cluster.local (default port 80).

See also: creating a Service, diagnosing network problems.

Ingress for external access #

When to use: users or external systems access the application via HTTP/HTTPS from outside the cluster. One Ingress can serve many hosts or paths (different applications behind one balancer).

• TLS: for production, be sure to use TLS (Secret with a certificate is indicated in Ingress). Otherwise, passwords and tokens are transmitted in clear text.
• Domain names: set the host in Ingress (for example, app.example.com), rather than relying on IP - when changing environments or clusters, the configuration remains the same.
• Rate limiting and WAF are configured at the level of the Ingress controller or provider (depending on the platform).

See also: creating Ingress, Gateway API routes.

Antipatterns #

# Bad
image: my-app:latest

# Good
image: my-app:v1.0.0

# Bad
containers:
- name: app
  image: my-app:v1.0.0

# Good
containers:
- name: app
  image: my-app:v1.0.0
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"

# Bad
env:
- name: PASSWORD
  value: "secret-password"

# Good
env:
- name: PASSWORD
  valueFrom:
    secretKeyRef:
      name: app-secret
      key: password

Checklist before production #

• All manifests are stored in Git and versioned.
• Specifies requests and limits for all containers.
• Readiness and liveness probes are configured.
• Secrets are stored in Kubernetes Secrets, not in manifests.
• Configured TLS for all external services.
• Deployment tested in dev/stage.
• Monitoring, log collection and alerts are configured.
• Configured backup for critical data.
• Rollback command prepared: kubectl rollout undo deployment/<name>.
• Application dependencies and requirements are documented.