mediumConceptCodingMidSeniorStaff
How do generators reduce memory use, and when are they the wrong choice?
5 min readUpdated 2026-08-15
Answer
Short answer
A generator produces items on demand and keeps only the current item plus its suspended frame, so memory stays flat as the input grows; it is the wrong tool when you need a length, indexing, or more than one pass, and when per-item resume overhead outweighs the space saved.
Was this answer helpful?
Likely follow-ups
- What happens to a with block inside a generator if the consumer stops iterating half way through?
- Why can a generator not be sent to a worker with multiprocessing, and what do you do instead?
- When is itertools.tee worse for memory than simply building a list?
- How does contextlib.contextmanager use the fact that a generator suspends at yield?
Related questions
- A context manager written with @contextmanager leaves a lock held whenever the with body raises. What is missing from the generator?mediumAlso on contextlib4 min
- You read a four gigabyte CSV into pandas and the process needs far more memory than that. Where did it go?mediumAlso on memory4 min
- How do you remove elements from a collection while iterating it, and what makes an iterator fail-fast?mediumAlso on iterators5 min
- How would you rewrite a row-by-row pandas transformation, and what is SettingWithCopyWarning telling you?mediumAlso on memory5 min