A Kubernetes cluster admin account has been compromised for 45 minutes. Do you clean the cluster or burn it to the ground, and why?
Evaluate the candidate's crisis management and technical recovery skills in a high-stakes scenario involving a total compromise of Kubernetes administrative control. Use this security answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects incident response to the point an interviewer is testing.
What the interviewer is scoring
- Whether they immediately contain the blast radius without tipping off the adversary.
- Does the candidate treat secret and certificate rotation as a mandatory, non-negotiable step rather than an afterthought?
- That they call for thorough forensics to identify persistence mechanisms like rogue daemonsets before anything is trusted again.
- Whether the candidate rebuilds the cluster from known-good IaC rather than trying to clean a deeply compromised state.
Answer
Short answer
Evaluate the candidate's crisis management and technical recovery skills in a high-stakes scenario involving a total compromise of Kubernetes administrative control.
Why remediation in place cannot work
When a Kubernetes cluster-admin account is compromised, the naive approach is to attempt remediation in place. Incident responders often try to hunt down rogue DaemonSets, delete malicious ClusterRoles, and play a futile game of whack-a-mole with an adversary who has had unrestricted API access. This is a catastrophic error. A cluster-admin compromise is a "game over" scenario. The attacker can easily bury backdoors in Mutating Webhooks, modify core control plane components, or deploy stealthy sidecars. An environment breached at this level of privilege can never be trusted again.
Surgical containment and forensics
Before taking destructive actions, immediate and surgical containment is required to halt the blast radius without tipping off the adversary. This involves immediately revoking the compromised cloud IAM credentials and initiating a rotation of the Kubernetes API server's certificate authority (CA) to invalidate active client certificates.
Rather than shutting down the cluster and causing a massive business outage, control plane networking must be isolated, restricting API server access exclusively to incident response jump boxes. Simultaneously, memory dumps must be captured, and the underlying worker nodes and etcd datastore must be snapshotted to preserve volatile forensic evidence before the environment is destroyed.
The mandate for immutable recovery
Because the integrity of the cluster is fundamentally broken, the only authoritative recovery strategy is to abandon it. Disaster recovery must rely on Infrastructure as Code (IaC) and GitOps pipelines to instantly provision a parallel, pristine Kubernetes cluster in a clean VPC. Workloads must be redeployed exclusively from cryptographically signed, known-good container images pulled from a secure registry.
Total cryptographic rotation
In a complete compromise scenario, every single secret residing in the cluster must be considered stolen. A comprehensive, aggressive secret rotation event must be executed before cutting traffic over. This includes rotating all database passwords, API keys, third-party integration tokens, and TLS certificates, injecting the fresh credentials exclusively into the new cluster.
Post-incident, structural changes are mandatory: static kubeconfig files with admin privileges must be entirely eradicated in favor of ephemeral, just-in-time access tied to a central Identity Provider with strict device posture checks.
flowchart TD
A["Detect Admin Compromise"] --> B["Revoke Compromised IAM/Kubeconfig"]
B --> C["Snapshot API Server & Node Logs"]
C --> D{"Assess Cluster State"}
D -- "Heavily Modified" --> E["Spin up Parallel Clean Cluster via IaC"]
D -- "Lightly Modified" --> F["Aggressive In-Place Key Rotation"]
E --> G["Shift Traffic to New Cluster"]
F --> H["Hunt for Persistence (DaemonSets, Webhooks)"]
G --> I["Forensic Analysis of Old Cluster"]When a Kubernetes cluster administrator account is compromised, the environment's integrity is entirely voided; the most secure recovery strategy relies on immutable infrastructure principles, abandoning the tainted cluster in favour of rebuilding from known-good code while aggressively rotating all cryptographic material.
© 2026 Preptima. Originally published at preptima.com.
Likely follow-ups
- How would your response differ if the compromised account only had access to a single namespace rather than full cluster-admin?
- What compensating controls would you put in place so a single stolen kubeconfig can never grant cluster-admin again?
- How do you prove to auditors and customers that the rebuilt cluster is clean, given you cannot fully trust anything from the old environment?
Related questions
- Would you use iptables or eBPF for network policy enforcement in a massive multi-tenant Kubernetes cluster, and what are the operational trade-offs?hardAlso on kubernetes and security2 min
- A customer reports seeing another company's records in your admin console. Walk me through the first hour, and then tell me what you change so this class of bug cannot happen again.hardAlso on security4 min
- Pods are being evicted during node pressure even though your CPU dashboards look fine. Where do you look, and what would you change so it stops?hardAlso on kubernetes4 min
- An upstream feed resets ten thousand of your prices to a penny and orders start arriving. What should have stopped it, and what do you do with the orders that got through?hardAlso on incident-response6 min