Episode 67 — Mitigate Hypervisor and Container Security Weaknesses
Side channels ride the physics of shared silicon, which means scheduling policy can strengthen or weaken your blast radius. Use strict scheduling and Non-Uniform Memory Access (N U M A) pinning where needed to corral high-sensitivity workloads onto dedicated cores and memory banks, limiting cache-based cross-tenant observations. Reserve nodes or sockets for regulated data classes, and prefer no-co-tenancy for jobs with sustained cryptographic operations that could amplify leakage. Where performance allows, enable features that randomize or partition cache behavior, and log placements so an after-action can reconstruct which guests shared which dies, cores, and banks at a given time. This is not fear; it is scope control. When you can say “these tenants never shared L3 cache during the window,” an entire category of speculation quiets down.
Containers shrink blast radius by design, but only if the runtime is stingy. Harden runtimes to least privilege: run as non-root, mount read-only filesystems, and lock down write points to the few paths processes truly need. Apply system call filters with secure computing mode (seccomp) and mandatory access control like AppArmor (A p p A r m o r) or SELinux, starting from restrictive profiles and punching holes only for documented requirements. Remove setuid binaries from images; disable automatic device mounts; and keep ephemeral storage small to make abuse noisy. Evidence matters here: profiles live in version control, pods or services reference them by name, and denials show up in logs that engineers actually read. Over time, this converts “containers are light” into “containers are disciplined,” which is the only lightness that sustains.
Some knobs are so dangerous that the default answer is “no.” Prohibit privileged containers and host Process Identifier (P I D) or Inter-Process Communication (I P C) namespace sharing unless a narrow, ticketed justification exists, and then confine the exception with layers—node isolation, network quarantine, and time-boxed lifetimes. Disallow hostPath mounts to sensitive directories; where a mount is unavoidable, scope it read-only and subpath-specific. Collapse inherited capabilities to a minimal set: most workloads do not need NET_ADMIN, SYS_ADMIN, or the ability to load kernel modules. Build admission controls that reject risky specs at the door, attach the rejection reason, and point requesters to the sanctioned pattern. This is not friction for its own sake; it is a safety rail that keeps a one-line YAML change from becoming a host compromise.
Supply chain is part of isolation because a hostile binary can bypass clean boundaries. Scan images before deploy for vulnerabilities, embedded secrets, and base-image provenance, and capture a Software Bill of Materials (S B O M) for every build. Pin image digests rather than tags in deployments, and require signatures with verification in the cluster so only attested images run. Rotate base images on a cadence, refuse builds from unapproved registries, and fail pipelines when scans exceed severity budgets or S B O M gaps appear. Keep tight evidence: which commit produced which digest, which scanner saw which issues, and which exception—if any—allowed a short-term risk with an expiration. When an issue lands in the news, you can ask your registry “where does library X appear” and get a list in seconds, not days.
Linux namespaces, control groups (cgroup), and capability sets are the kernel’s isolation primitives; apply them deliberately to contain reach and resource abuse. Namespaces split visibility for mounts, processes, networks, and users; ensure each workload lives in its own worlds and avoid host networking except for infrastructure primitives with their own moats. Constrain cgroup CPU, memory, P I D counts, and I/O so noisy neighbors cannot starve peers or stage fork bombs; align those limits to service-level objectives so enforcement is predictable, not punitive. Strip ambient capabilities down to the few a process requires, and prefer per-binary rules over blanket allowances. Instrument breaches of these fences as high-signal events: when a task hits a limit or requests a forbidden capability, you want both telemetry and a human-readable breadcrumb that speeds triage.
Not all workloads are created equal; isolate by sensitivity as if it were your network. Use namespaces and projects to separate teams; apply network policies that explicitly state who may talk to whom; and dedicate nodes or pools for high-risk or regulated data so they never mix with casual compute. Tie scheduling to node labels—“crypto-isolated,” “no-passthrough,” “strict-seccomp”—so placement carries policy, not just capacity. Keep taints and tolerations for the few agents that must run everywhere, and validate that nothing else can co-locate on the wrong iron. This segmentation reads like a map: when a request asks for a new service, the answer includes where it can live, who it can speak to, and which controls prove those claims minute by minute.
Isolation decays when platforms drift. Monitor for mutating webhooks that change traffic or inject sidecars without approval, for policy bypasses that add wildcards to roles, and for unsafe mounts or capabilities that creep back after refactors. Alert on admission controller outages, image-policy failures, and clusters running with the wrong Pod Security Standards; treat each as a red light, not a suggestion. Record every policy change with the actor, diff, and reason so a noisy day can be reconstructed without folklore. Drift is not personal; it is entropy. The fix is visibility plus small, frequent corrections that keep guardrails where you promised they would be.
Containment must be rehearsed or it will be clumsy when it counts. Plan escape-response playbooks that isolate nodes at the switch or orchestrator, rotate cluster credentials, recycle affected workloads, and reimage hosts that crossed a trust boundary. Keep a clean path to cordon and drain suspect nodes, and a script that validates image signatures and policy hooks before letting them rejoin. Pair every isolation with evidence preservation: capture process lists, network sockets, container metadata, and a time-boxed packet trace before the iron goes to the scrubber. Practice these moves quarterly on staging so the muscle memory exists. The point is not perfection—it is making the dangerous day short, bounded, and documented.
A concrete example shows hardening paying off. A team ships a service that originally requested the CAP_SYS_ADMIN capability to mount a temporary filesystem. Admission policy rejects the spec with a clear error that CAP_SYS_ADMIN is disallowed and suggests CAP_SYS_CHOWN and CAP_MKNOD as likely mistaken asks. The engineer removes the risky capability, switches to a read-only root with a writable /tmp, and adds a minimal seccomp profile that permits only the handful of system calls the service genuinely needs. A week later, a known container breakout technique attempts a mount call that the seccomp profile denies; the event logs show the block with the pod, namespace, and rule identifier, and no host-level artifacts appear. One tightened capability and a small profile turned a would-be breakout into a harmless, auditable blip.
We close with two immediate, measurable actions. First, tighten runtime policy: enforce non-root, read-only filesystems, minimal capabilities, and a baseline seccomp/AppArmor pair in admission so unsafe pods never schedule, and publish the small list of sanctioned exceptions with owners and expiry. Second, run an image provenance audit across your top services: require signatures, pin digests, capture S B O Ms, and refuse builds from unvetted bases; produce a one-page gap list with teams and dates. When these are done and logged, your platform will not only be harder to escape—it will be able to prove, quickly and calmly, why attempts failed and why tenants remained safely apart.
