df says the filesystem is 100% full but du only accounts for half of it. What is going on?

Start from what each command asks

df queries the filesystem's own accounting via statfs: total blocks, free blocks, blocks reserved. It is authoritative about allocation because the filesystem is the thing doing the allocating. du does something entirely different — it walks a directory tree, stats each entry it finds, and sums the blocks those entries report. Anything not reachable by walking the tree is invisible to it.

So the question is never "which one is lying". It is "what is allocated but not reachable from the path I walked", and there are four answers worth knowing.

Deleted, but still open

Unlinking a file removes a directory entry. The inode and its data blocks survive as long as any process holds an open descriptor, which is why a log you deleted at 3am is still consuming forty gigabytes at 9am. du cannot see it because there is no name left to walk to; df still counts every block.

# Open files whose link count has dropped to zero.
sudo lsof +L1

# Or scan every process's descriptors for unlinked regular files.
sudo lsof -nP | awk '$0 ~ /deleted/ {print $1, $2, $7, $NF}'

The signature is a large SIZE/OFF value against a path ending in (deleted). The process holding it is almost always something that had its log rotated out from under it without being told to reopen — an application logging to a file directly, or a rotation configured to create a new file while the writer still has the old one.

Inodes, not blocks

A filesystem allocates its inode table at creation time on ext4, so a directory full of millions of tiny files can exhaust inodes while leaving most of the disk unwritten. Writes then fail with ENOSPC, and every tool reports "no space left on device", which sends you looking at the wrong number.

df -h /var    # blocks
df -i /var    # inodes: check both, always

If inodes are the problem, find where the count lives rather than where the bytes live: sudo find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head counts entries per directory. Mail spools, session files, per-request temp directories and unrotated small logs are the usual sources.

A mount point standing on top of files

If a process wrote to /var/log before a separate volume was mounted there, those files still exist on the root filesystem, underneath the mount. Nothing can reach them by path any more, so du /var/log reports the new volume's contents and the old ones are unaccounted for. The root filesystem, meanwhile, is still carrying them.

# Expose the underlying root filesystem so the shadowed files become walkable.
sudo mount --bind / /mnt
sudo du -shx /mnt/var/log
sudo umount /mnt

The reverse mistake matters too, and is the cheapest of all these to rule out. Without -x or --one-file-system, du descends into other mounted filesystems and reports space that belongs to none of the volume df is talking about. Comparing an unbounded du against a per-filesystem df is not a comparison at all.

The reserve, and the gap it explains

On ext4, five percent of blocks are reserved for the superuser by default. Ordinary processes hit ENOSPC while df still shows a few percent free, and root can happily write in the same moment, which makes the fault look intermittent or permission-related. The reserve exists to leave room for the administrator to recover and to limit fragmentation, so it is not simply waste, but on a large data volume five percent is a lot of blocks to hold back.

sudo tune2fs -l /dev/sda1 | grep -i 'reserved block'
sudo tune2fs -m 1 /dev/sda1   # reduce the reserve to 1% on a data volume

XFS has no equivalent knob, so if the filesystem is XFS this branch of the diagnosis does not apply; sparse files and preallocation are the more likely source of a gap there.

Reclaiming it without a restart

For the deleted-but-open case, restarting the process works and is often unacceptable. You can instead truncate the file through the descriptor itself, since /proc/<pid>/fd/<n> is a path to the same inode:

sudo ls -l /proc/2841/fd | grep deleted     # find the descriptor number
sudo truncate -s 0 /proc/2841/fd/3          # frees the blocks immediately

Here is the caveat that makes this worth understanding rather than copying. If the writer did not open the file with O_APPEND, its file offset is unchanged by the truncation, so the next write lands at the old offset and the kernel creates a sparse file — apparent size back to forty gigabytes, actual allocation only what was written after the truncation. That is usually fine, because the blocks are genuinely free, but ls -l will now disagree with du on that file and you should know why rather than be alarmed by it. The durable fix is different anyway: log to stdout and let the supervisor handle rotation, or configure rotation to signal the writer to reopen, and reserve copytruncate for programs that cannot be signalled.