Advanced Docker & Kubernetes: A Sysadmin’s Guide to Scaling, Security, and Optimization

Introduction

Modern infrastructure demands sysadmins to go beyond basic container orchestration. Docker and Kubernetes (K8s) are powerful, but misconfigurations can lead to security breaches, wasted resources, or downtime. This guide dives into advanced techniques for optimizing images, hardening clusters, debugging at scale, and cutting costs without sacrificing performance.

---

1. Docker Deep Dive: Beyond the Basics

Optimizing Images for Production

Problem: Bloated images slow deployments and increase attack surfaces.

Solutions:

• Multi-stage builds: Remove build dependencies in the final image.

dockerfile

Copy

Download

# Stage 1: Build

FROM golang:1.21 as builder

WORKDIR /app

COPY . .

RUN go build -o myapp .

# Stage 2: Runtime

FROM alpine:latest

COPY –from=builder /app/myapp /usr/local/bin/

CMD [“myapp”]

• Minimal base images: Prefer distroless or alpine over ubuntu.
• Layer caching: Order Dockerfile commands from least to most frequently changed.

Pro Tip: Use dive to analyze image layers:

bash

Copy

Download

dive my-image:latest

Advanced Storage & Networking

• Overlay2 Driver: Default for Docker, but tune dm.basesize for disk space limits.
• Custom Bridge Networks: Isolate containers and control traffic.

bash

Copy

Download

docker network create –driver=bridge –subnet=192.168.100.0/24 my_network

• Volume Optimization: Use tmpfs for ephemeral data to reduce I/O overhead.

yaml

Copy

Download

# docker-compose.yml

services:

  redis:

    volumes:

      – type: tmpfs

        target: /data

Security Hardening

• Rootless Containers: Run Docker daemon as a non-root user (requires uidmap).

bash

Copy

Download

dockerd-rootless-setuptool.sh install

• Seccomp Profiles: Restrict syscalls. Use the default profile or customize:

json

Copy

Download

{

  “defaultAction”: “SCMP_ACT_ERRNO”,

  “syscalls”: [{“name”: “mkdir”, “action”: “SCMP_ACT_ALLOW”}]

}

• Image Signing: Verify integrity with Docker Content Trust (DCT):

bash

Copy

Download

export DOCKER_CONTENT_TRUST=1

docker push my-repo/image:signed

---

2. Kubernetes Mastery for Sysadmins

Advanced Scheduling

• Taints & Tolerations: Dedicate nodes for specific workloads (e.g., GPU nodes).

yaml

Copy

Download

# Taint a node

kubectl taint nodes node1 gpu=true:NoSchedule

# Pod toleration

tolerations:

– key: “gpu”

  operator: “Equal”

  value: “true”

effect: “NoSchedule”

• Topology Spread Constraints: Prevent zone outages by spreading pods evenly.

yaml

Copy

Download

topologySpreadConstraints:

– maxSkew: 1

  topologyKey: topology.kubernetes.io/zone

  whenUnsatisfiable: DoNotSchedule

Resource Management

• LimitRanges: Enforce default CPU/memory requests per namespace.

yaml

Copy

Download

apiVersion: v1

kind: LimitRange

metadata:

  name: cpu-limiter

spec:

  limits:

  – defaultRequest:

      cpu: “500m”

    type: Container

• Autoscaling: Combine HPA (Horizontal Pod Autoscaler) and VPA (Vertical Pod Autoscaler).

bash

Copy

Download

# Install VPA

kubectl apply -f https://github.com/kubernetes/autoscaler/releases/download/vertical-pod-autoscaler-0.14.0/vertical-pod-autoscaler.yaml

Debugging Tricks

• Ephemeral Containers: Inspect running pods without restarting.

bash

Copy

Download

kubectl debug -it my-pod –image=busybox –target=my-pod

• Logging Pipelines: Send logs to Elasticsearch via Fluentd.

yaml

Copy

Download

# fluentd-configmap.yaml

data:

  fluent.conf: |

    <source>

      @type tail

      path /var/log/containers/*.log

      read_from_head true

      <parse>

        @type json

      </parse>

    </source>

    <match **>

      @type elasticsearch

      host elasticsearch.default.svc.cluster.local

    </match>

---

3. Security at Scale

Pod Security Policies (Replacement)

K8s deprecated PSPs in 1.25. Use Pod Security Admission (PSA) or OPA Gatekeeper:

yaml

Copy

Download

# PSA example (enforce baseline policy)

apiVersion: v1

kind: Namespace

metadata:

  name: secure-ns

  labels:

    pod-security.kubernetes.io/enforce: baseline

Network Policies

• Zero-Trust with Calico: Deny all traffic by default, then allow specific flows.

yaml

Copy

Download

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: deny-all

spec:

  podSelector: {}

  policyTypes:

  – Ingress

  – Egress

Secrets Management

• HashiCorp Vault + CSI Driver: Inject secrets dynamically.

yaml

Copy

Download

apiVersion: secrets-store.csi.x-k8s.io/v1

kind: SecretProviderClass

metadata:

  name: vault-db-creds

spec:

  provider: vault

  parameters:

    vaultAddress: “http://vault:8200”

    roleName: “k8s-role”

    objects: |

      – objectPath: “secret/database”

   secretKey: “password”

---

4. Performance & Cost Optimization

Cluster Sizing

• Spot Instances: Save 90% with intelligent node pools (AWS Spot, GCP Preemptible).

yaml

Copy

Download

# AWS EKS node group with spot

apiVersion: eksctl.io/v1alpha5

nodeGroups:

– name: ng-spot

  instanceTypes: [“t3.large”, “t3a.large”]

  spot: true

Monitoring

• Prometheus Alerts: Detect memory leaks early.

yaml

Copy

Download

– alert: HighMemoryUsage

  expr: container_memory_usage_bytes{pod=”my-pod”} > 1GB

  for: 10m

Waste Reduction

• Kube-cost: Track spending by namespace/deployment.

bash

Copy

Download

kubectl cost –window 7d –show-all-resources

---

Conclusion

Mastering advanced Docker and Kubernetes requires a balance of security, efficiency, and observability. Key takeaways:

1. Docker: Optimize images, harden runtime, and use rootless mode.
2. Kubernetes: Leverage advanced scheduling, autoscaling, and OPA for governance.
3. Costs: Right-size clusters and automate scaling policies.

Next Steps:

• Experiment in a sandbox (e.g., Kind or Minikube).
• Join the K8s Slack (#security, #autoscaling channels).
• Read the CIS Benchmarks for compliance.