mediumConceptCodingMidSenior
You read a four gigabyte CSV into pandas and the process needs far more memory than that. Where did it go?
4 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
Object-dtype string columns hold pointers to individual Python objects rather than packed bytes, any missing value promotes an integer column to float64, and the parser's peak exceeds the final frame.
Was this answer helpful?
Likely follow-ups
- Which columns are worth converting to categorical, and when does that make things worse?
- How would you compute a monthly aggregate over a file that will not fit in memory at all?
- What changes if the file is Parquet rather than CSV, and what does that buy you at read time?
- Why does deleting a large frame sometimes not reduce the process's resident memory?
Related questions
- How would you rewrite a row-by-row pandas transformation, and what is SettingWithCopyWarning telling you?mediumAlso on pandas and memory5 min
- Why would you stream a large file rather than read it into memory, and what is backpressure?mediumAlso on memory4 min
- What happens when you append to a slice, and when do two slices stop sharing memory?mediumAlso on memory5 min
- How do generators reduce memory use, and when are they the wrong choice?mediumAlso on memory5 min