Why would you stream a large file rather than read it into memory, and what is backpressure?

Short answer

Respect backpressure by pausing writes when write() returns false, waiting for drain, or using pipeline() so Node coordinates reads, writes, errors and cleanup. Ignoring it turns slow downstream I/O into memory growth.

Use streams as the boundary in the explanation: data is not being copied magically, it is being pulled or pushed through a contract with explicit flow control. A strong streams answer names where buffering happens and why ignoring that boundary turns a fast producer into a memory problem.

The memory argument is about concurrency

Reading a 500 MB file into a buffer and sending it works fine in development. The reason it is wrong is not that 500 MB is large — it is that the memory cost is per concurrent request.

Ten simultaneous downloads is 5 GB. The process either dies or spends its time in garbage collection, and the failure arrives precisely when the service is busiest. Worse, in Node the buffer must be allocated contiguously, so as the heap fragments you can fail to allocate while nominally having free memory.

Streaming inverts the relationship. Memory becomes a function of the chunk size and the number of chunks in flight, not of the file size, so a 5 GB file costs the same as a 5 MB one. That property — capacity that does not depend on the largest input a user might supply — is the actual benefit, and it is a stronger statement than "uses less memory".

The second benefit is latency. The first byte reaches the client as soon as the first chunk is read, rather than after the whole file has been loaded.

What backpressure is

Streaming only bounds memory if the fast side slows down for the slow side. Reading from an SSD is far quicker than writing to a network socket, and without a brake the reader would pull data as fast as the disk allows while the writer's internal buffer grew without limit — reproducing exactly the memory problem you were avoiding, just less visibly.

Backpressure is the signal that prevents it. Every writable stream has a high-water mark, and write() returns false once the buffered amount exceeds it. That return value means stop sending; I will emit drain when I am ready.

The mechanism only works if you honour it, which is why this is the classic Node bug:

// Broken: the return value is discarded, so nothing ever pauses.
// On a slow client this buffers the entire file in the socket's queue.
readable.on("data", (chunk) => {
  writable.write(chunk);
});

The code is correct in the sense that every byte arrives. It just does so having buffered the whole file in memory, so it behaves identically to the non-streaming version while looking like a stream. This is a genuinely common production bug, and it is invisible in testing because a fast local client drains as quickly as the disk can supply.

Let the library do it

The correct answer is not to write the pause and resume handling yourself. pipe implements it, and pipeline implements it and fixes pipe's error handling.

import { pipeline } from "node:stream/promises";

// Honours backpressure, and destroys every stream in the chain if any
// of them fails - which is the part `pipe` does not do.
await pipeline(
  createReadStream(inputPath),
  createGzip(),
  createWriteStream(outputPath),
);

pipe's specific defect is that an error in one stream does not clean up the others, so a failed write leaves the read stream open and its file descriptor held. Under load that becomes descriptor exhaustion, which presents as an unrelated failure elsewhere in the process — one of the least pleasant things to diagnose. pipeline destroys the whole chain on any failure and gives you one place to handle the error.

Chunks do not respect your record boundaries

The mistake in almost every first attempt at parsing a stream. A chunk is however many bytes the underlying resource happened to deliver, and it has no relationship to lines, JSON documents, or CSV rows. A record will be split across two chunks, and it will not be the record you tested with.

// Broken: a line straddling two chunks is silently split in half.
readable.on("data", (chunk) => {
  for (const line of chunk.toString().split("\n")) handle(JSON.parse(line));
});

The fix is to keep the incomplete tail and prepend it to the next chunk, which is what readline and every line-splitting transform do internally. There is a second, subtler version of the same bug: chunk.toString() on a boundary that falls inside a multi-byte UTF-8 character produces a replacement character. StringDecoder, or setting an encoding on the stream, holds the partial bytes until the character is complete.

Async iteration, and where it stops

Modern Node lets you consume a readable with for await, which reads much better than event handlers and honours backpressure automatically, since the loop does not request the next chunk until the body has finished.

for await (const chunk of readable) {
  await writeSomewhere(chunk);   // the loop waits; the source pauses
}

The limit is that this consumes one stream at a time. As soon as you have a chain of transforms and need failures anywhere in it to tear down everything, pipeline is still the right tool. Combining them — a transform written as an async generator, composed with pipeline — is often the most readable arrangement available.

Streaming bounds memory only while the brake is connected. Discard the return value of write and you have written the buffering version with extra steps.

---

© 2026 Preptima. Originally published at preptima.com.