Kubernetes in Production: Hard-Won Lessons from 50+ Deployments

Kubernetes solves real problems — service orchestration, scaling, self-healing, and declarative infrastructure. But it also introduces an enormous amount of operational complexity. The learning curve is practically vertical, and the defaults are rarely suitable for production.

A harsh reality check: If you are running fewer than 10 microservices, you probably don't need Kubernetes. Managed services like AWS App Runner, Google Cloud Run, or even Heroku are often a better, more cost-effective choice for smaller deployments.

That said, when you *do* reach the scale where Kubernetes is necessary, there is nothing else that comes close to its power and flexibility. Here is what we've learned from running it in production across industries from fintech to healthcare.

The single most impactful practice for cluster stability is proper resource management. Every container in your cluster *must* have resource `requests` and `limits` explicitly defined. Without them, you are playing Russian roulette. A single memory-leaking application can starve the entire node, bringing down other critical services.

Set `requests` based on actual historical usage (use metrics, not guesses). Set `limits` slightly higher (e.g., 2-3x requests) to handle legitimate spikes, but tight enough to kill runaway processes before they cause collateral damage.

# Production-Ready Resource Configuration
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: api-service
        resources:
          requests: # Guaranteed resources
            cpu: "250m"
            memory: "512Mi"
          limits:   # Maximum allowed
            cpu: "750m"
            memory: "1Gi"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Notice the probes in the YAML above. Kubernetes needs to know when your application is ready to receive traffic (Readiness) and when it has crashed and needs to be restarted (Liveness).

If you don't define these, Kubernetes assumes your app is ready the moment the container starts, leading to dropped requests during startup. Conversely, if an app deadlocks but the process doesn't exit, Kubernetes won't know to restart it without a liveness probe.

By default, Kubernetes has a 'flat' network: any pod can talk to any other pod. This is terrible for security. If an attacker compromises a frontend pod, they can easily probe your backend databases.

Implement NetworkPolicies to restrict traffic. Start with a 'default deny' policy that blocks all ingress and egress traffic, then explicitly whitelist the communication paths that are actually required by your architecture.

You cannot operate what you cannot observe. In a distributed system, traditional logging is insufficient. Our standard observability stack for Kubernetes includes:

* Metrics: Prometheus (often via Prometheus Operator) scraping node and application metrics, visualized in Grafana. * Logs: Fluent Bit running as a DaemonSet to collect and ship structured JSON logs to a centralized store like Elasticsearch or Datadog. * Traces: OpenTelemetry instrumentation to track requests as they flow across multiple microservices. * Alerting: Alertmanager routing critical issues to PagerDuty.

Never manually apply YAML files to a production cluster using `kubectl`. Your cluster state should be entirely defined in code (Terraform for the underlying infrastructure, Helm or Kustomize for the applications) and stored in version control.

We strongly recommend adopting a GitOps workflow using tools like ArgoCD or Flux. When a PR is merged to the `main` branch, the GitOps controller automatically synchronizes the cluster to match the declared state. This provides a complete audit trail and makes disaster recovery trivial.