Why is state the hard part of Terraform, and how do you manage drift, imports and locking around it?

State is an identity mapping, and that is the whole problem

Terraform's language describes what should exist. It does not describe which real object corresponds to which block, and it cannot derive that from a cloud API, because there is nothing in a virtual network that says "I am module.network.azurerm_virtual_network.main". State holds exactly that mapping, plus a snapshot of the attributes as last observed, plus dependency ordering.

Every hard thing about Terraform follows. plan reads the config, refreshes the recorded resources against the provider, and diffs the three-way relationship between config, state and reality. If a resource is in state, Terraform will try to update it. If it is absent from state, Terraform will try to create it, even when an identical object already exists and the create will fail with a conflict. If it is in state but gone from the config, Terraform will destroy it. There is no fourth behaviour, so a surprising plan is nearly always a state problem wearing a config costume.

Two consequences are worth saying out loud in an interview because they are where real damage happens. State contains resource attributes in plain text, including generated passwords, connection strings and certificate material, so it is a secret in itself and must live in an encrypted, access-controlled backend rather than a repository. And removing something from state does not touch the infrastructure: terraform state rm makes Terraform forget the resource, leaving it running and unmanaged, which is a legitimate tool and also the fastest way to create an orphan nobody can find in six months.

Drift is a decision, not an error

Drift is any change made to a managed resource outside Terraform: a console edit during an incident, an autoscaler adjusting a count, a platform team applying a tag policy. The refresh step surfaces it, and a plan that proposes changes you did not write is the signal.

# Show drift without proposing any config-driven changes.
terraform plan -refresh-only

# Accept observed reality into state, changing nothing in the infrastructure.
terraform apply -refresh-only

The mistake is treating the resulting plan as noise and applying to make it go away. That reverts whatever the change was, and during an incident the console edit may well be the thing keeping the service up. So the response is a choice with three branches. If the change was wrong, apply and revert it. If the change was right, codify it and then apply, so the code becomes the truth again. If the attribute is genuinely owned by something else, tell Terraform to stop caring about it with lifecycle { ignore_changes = [...] }, which is the honest way to model an autoscaled desired_count or a tag another system stamps on.

The structural fix is to reduce how much drift can happen at all: no human write access to production outside the pipeline, and a scheduled -refresh-only plan that reports rather than applies, so drift is detected within a day instead of at the next unrelated deployment when it is tangled up with real changes.

Imports, and why a rename is a destroy

Adoption of existing infrastructure runs through import. Config-driven import blocks, available since Terraform 1.5, are preferable to the older terraform import command because they are reviewable, plannable and idempotent: you commit the block, the plan shows what will be adopted, and the apply records it in state without creating anything.

import {
  to = aws_s3_bucket.artefacts
  id = "acme-build-artefacts-eu-west-1"
}

resource "aws_s3_bucket" "artefacts" {
  bucket = "acme-build-artefacts-eu-west-1"
}

The related and more commonly encountered problem is refactoring. Because state keys on the resource address, renaming a resource or moving it into a module changes its identity as far as Terraform is concerned. The old address vanishes from the config and the new one is unknown to state, so the plan is a destroy followed by a create. On a security group that is a blip; on a database or a static IP it is an outage. A moved block, available since Terraform 1.1, tells Terraform the new address is the old resource and turns that plan into a no-op.

# Refactoring only. Terraform updates state; nothing is created or destroyed.
moved {
  from = aws_db_instance.primary
  to   = module.data.aws_db_instance.primary
}

The same identity trap bites in a subtler way with count. Resources created with count are addressed by index, so removing the second element from a five-element list re-indexes everything after it and Terraform destroys and recreates four resources to shuffle them down. for_each over a map or set addresses each instance by a stable key, so removing one entry removes exactly one resource. For anything stateful, or anything whose recreation is user-visible, for_each is not a style preference; it is the difference between a targeted change and a cascade.

Locking, and what a corrupted state costs

Two applies running against one state file at the same time will interleave writes and leave a state that describes neither the infrastructure nor the config. Recovery means reconstructing the mapping by hand against a live environment, which is the worst afternoon in this job.

So the backend must support locking and you must use one that does: S3 with a DynamoDB lock table historically, and since Terraform 1.10 a native lock file in the bucket via use_lockfile; a blob lease on Azure Storage; the object generation on GCS. Local state files are unacceptable for anything shared, both for locking and because they end up in a repository with their secrets.

In practice, add -lock-timeout=5m in CI so a queued job waits for a running one rather than failing, and treat terraform force-unlock as a break-glass command that requires you to have confirmed the holding process is genuinely dead. The pipeline should also be the only place applies happen, because a lock protects concurrent applies from each other but nothing protects you from a developer applying an out-of-date branch.

Structuring modules and environments without copying files

The instinct to copy an environment directory is what IaC exists to prevent, and yet the most common Terraform layout in the wild is three near-identical folders that have quietly diverged. The way out is to separate the composition from the parameters.

A module should encapsulate a meaningful unit with a decision inside it: "a service with its load balancer, DNS record, autoscaling group and alarms" is a module because it embeds choices worth making once. A module that wraps a single aws_s3_bucket and passes six variables straight through is worse than the resource, because it adds an indirection, a version to bump and a leaky interface without removing any decision. Version modules with git tags or a registry and pin the version in each caller, so upgrading staging and production is two deliberate commits rather than one accidental one.

Environments should then be root modules that differ only in their variable values and their backend configuration, each with its own state. Splitting state further, by blast radius and change frequency, is the more important call: the network and the shared data stores change rarely and are catastrophic to break, while application infrastructure changes daily. In one state, every application change plans against the database and every plan takes minutes. In separate states, an application apply cannot touch the network, and the coupling between them is an explicit data source lookup at the boundary.

Workspaces are frequently proposed here and are usually the wrong tool for environments. They share one backend, one set of credentials and one configuration, differing only by a name you can interpolate, which is exactly what you do not want between development and production: separate accounts, separate credentials and separate blast radius. They suit many near-identical instances of the same thing, such as per-tenant or per-developer stacks, and little else.

Terraform's mental model is three-way: config, state, reality. Almost every confusing plan, destructive refactor and unrecoverable afternoon comes from a change to one of those three that the other two do not know about.