Node.js Core ModulesLesson 3.4

Node.js streams: reading and writing large data efficiently

Readable streams, Writable streams, Transform streams, pipe, pipeline, stream events, backpressure, fs.createReadStream, fs.createWriteStream

Streams Process Data in Chunks

Loading a 1GB file with fs.readFile puts it all in memory. Streams process data in chunks, keeping memory usage flat regardless of file size.

const fs = require('fs');
const { pipeline } = require('stream/promises');
const zlib = require('zlib');

await pipeline(
  fs.createReadStream('./large.txt'),
  zlib.createGzip(),
  fs.createWriteStream('./large.txt.gz')
);
console.log('Compressed');

Manual Stream Events

const readable = fs.createReadStream('./data.csv', { encoding: 'utf8' });

readable.on('data', (chunk) => {
  process.stdout.write(chunk);
});
readable.on('end', () => console.log('Done'));
readable.on('error', (err) => console.error(err));

Backpressure

When you write faster than the destination can consume, data buffers in memory. Use pipeline from stream/promises rather than the older pipe method — it handles backpressure automatically and propagates errors through the chain. The deprecated pipe does not forward errors; one stream error can leave others open, leaking file handles.