mediumCodingConceptEntryMidSenior
How would you rewrite a row-by-row pandas transformation, and what is SettingWithCopyWarning telling you?
5 min readUpdated 2026-08-15
Answer
Short answer
loc assignment, and move to DuckDB, Polars or a cluster once the frame plus its intermediate copies stop fitting in memory.
Was this answer helpful?
Likely follow-ups
- Your transformation is genuinely recursive - each row depends on the previous result. What now?
- How would you find which line of a slow pandas script is responsible?
- When does converting a string column to categorical dtype pay for itself, and when does it cost you?
- You have 400 daily Parquet files and need one monthly aggregate. Walk me through the read.
Related questions
- You read a four gigabyte CSV into pandas and the process needs far more memory than that. Where did it go?mediumAlso on pandas and memory4 min
- What happens when you append to a slice, and when do two slices stop sharing memory?mediumAlso on memory5 min
- A list of a million small objects uses far more memory than the data inside them. Where is it going?mediumAlso on numpy5 min
- Why would you stream a large file rather than read it into memory, and what is backpressure?mediumAlso on memory4 min