You read a four gigabyte CSV into pandas and the process needs far more memory than that. Where did it go?

A column is not a block of bytes unless its dtype says so

A pandas DataFrame stores each column as an array, and how much memory that array needs depends entirely on its dtype. A numeric column is genuinely compact: int64 is eight bytes per row and nothing else. A string column, in the default NumPy-backed representation, is object dtype, which means the array holds pointers and every value is a separate Python str living somewhere on the heap with its own object header and its own reference count.

That is the first and largest multiplier. Eight bytes of pointer plus a full Python string object per value means a short code like a two-letter country identifier can cost an order of magnitude more in memory than it does in the file, and a file that is mostly text can expand several times over on the way in. It also means the total is invisible to any accounting that only looks at the array, because the array is just the pointers.

The promotion nobody asks for

The second multiplier is quieter. In the NumPy-backed dtypes there is no missing-value marker for integers, so as soon as a column that should be int32 contains one empty field, pandas represents the whole column as float64 and stores NaN in the gaps. A column you expected to be four bytes per row is now eight, silently, because of one blank cell somewhere in ten million rows. The same logic makes a boolean column with any missing value into object, which puts it back in the pointer case above.

pandas' nullable extension dtypes and the Arrow-backed dtypes both have a proper missing-value representation and avoid the promotion, and since pandas 2.0 the readers accept a dtype_backend argument so you can ask for Arrow-backed columns directly. Arrow-backed strings are the more significant win of the two, because they store the characters contiguously with an offsets array instead of one Python object per value.

Measure it rather than reasoning about it

The number df.info() reports by default undercounts exactly the case that hurts, because it does not follow the pointers in an object column. Ask for deep accounting and the picture changes:

# shallow: counts the pointer array only, so object columns look cheap
usage = df.memory_usage(deep=True).sort_values(ascending=False)
print(usage / 1e6)          # megabytes per column, largest first
print(df.dtypes[usage.index[:5]])   # the dtypes doing the damage

Two or three columns almost always account for most of the total, and they are almost always the free-text ones and the ones that were promoted. That is worth doing before optimising anything, because the fix for a wide numeric frame is completely different from the fix for one long comment field.

There is a third effect that measurement of the final frame cannot show you: the parser's own peak. The C parser reads the file in blocks and assembles the columns, so resident memory during the read is higher than the frame that survives it, and a process that dies partway through a read_csv may have been sized correctly for the result. Relatedly, because the default reader infers each column's type from a chunk at a time, a column whose values change shape halfway down the file is inferred inconsistently and falls back to object with a DtypeWarning.

Fix it at read time, not afterwards

Converting a column after the fact still requires having held the expensive version, so the cheap version has to be requested up front.

df = pd.read_csv(
    path,
    usecols=["order_id", "country", "status", "amount"],  # never parse what you drop
    dtype={
        "order_id": "int32",
        "country": "category",   # int codes plus one shared categories index
        "status": "category",
        "amount": "float32",
    },
    parse_dates=["created_at"],
)

category is the highest-leverage of these. It replaces one Python string per row with a small integer code per row plus a single index of the distinct values, so a column with a few dozen distinct values across millions of rows collapses to nearly nothing. The condition is low cardinality relative to row count: applied to a column of mostly unique values it stores the codes and the full set of values, so it costs more than the plain column did.

The underlying point, though, is that CSV is the wrong format for anything you read more than once. It carries no types, so every read pays for inference and every reader has to be told the dtypes again; it cannot be read column-wise, so usecols still means scanning the whole file; and it is uncompressed text. Writing the data once to Parquet gives you the declared dtypes, genuine column selection, per-column compression and row-group statistics, and it usually removes the memory question rather than mitigating it.

When a smaller frame is the wrong goal

The trap in this question is treating it purely as a dtype-tuning exercise when the honest answer is sometimes that the whole file does not need to be resident. If the work is an aggregation, a chunksize loop that accumulates partial results, or a query engine that streams over the file, uses bounded memory regardless of how large the input grows, and it keeps working next quarter when the file has doubled. Halving the frame buys you one doubling of the data; not holding the frame buys you all of them.

The memory is in the dtypes, not the row count: pointers behind object columns and a float64 promotion caused by one missing value. Declare the types at read time, and if the operation is an aggregation, ask why the file is being materialised at all.