zlib Streaming API

zlib is a C library that implements DEFLATE and its common wrappers. Its streaming API uses z_stream, so an application can process input and output in bounded buffers instead of assuming that one call consumes or produces a whole message.

Stream State and Initialization #

The application supplies next_in/avail_in and next_out/avail_out. After a call, zlib advances the pointers and decreases the available counts; total_in and total_out provide cumulative counters. The application must consume produced bytes and refill buffers before the next call, while leaving the other state fields to zlib.

deflateInit() produces a zlib-wrapped stream. deflateInit2() selects raw DEFLATE with a negative windowBits, or gzip with windowBits + 16. Initialize allocator fields before initialization and call deflateEnd() on every successfully initialized stream, including error paths. A compressor can preload a shared history with deflateSetDictionary(). The zlib wrapper carries an Adler-32 dictionary identifier and lets inflate() return Z_NEED_DICT; raw DEFLATE requires the enclosing protocol to identify the dictionary. The gzip wrapper does not define preset-dictionary signaling.

Correct Deflate Loops #

Use Z_NO_FLUSH for ordinary input; frequent flushing reduces compression. Continue calling deflate() while input remains or output is pending. To finish, stop supplying input and call deflate(..., Z_FINISH) until it returns Z_STREAM_END, growing or draining the output buffer between calls. This bounded-input sketch assumes input_len fits in zlib’s uInt count:

z_stream strm = {0};
if (deflateInit(&strm, Z_DEFAULT_COMPRESSION) != Z_OK) fail();
strm.next_in = input;
strm.avail_in = (uInt)input_len;

for (;;) {
    strm.next_out = out;
    strm.avail_out = sizeof out;
    uInt input_before = strm.avail_in;
    int flush = strm.avail_in == 0 ? Z_FINISH : Z_NO_FLUSH;
    int rc = deflate(&strm, flush);
    size_t produced = sizeof out - strm.avail_out;
    write_bytes(out, produced);
    if (rc == Z_STREAM_END) break;
    if (rc != Z_OK) fail(rc);
    if (input_before == strm.avail_in && produced == 0) fail_no_progress();
}
deflateEnd(&strm);

Real code must refill next_in before the next Z_NO_FLUSH call and track progress across both cursors. Z_BUF_ERROR can mean that no progress was possible because a buffer was empty or full; it is not by itself evidence of corruption. Z_SYNC_FLUSH makes data available to a decoder without ending the stream, while Z_FINISH ends it.

For a network or file reader, assign each newly received chunk to next_in/avail_in, call until that count reaches zero or the output buffer fills, then drain output and continue. Do not overwrite a chunk while zlib still reports it in avail_in; the cursor is the ownership boundary.

Inflating with Bounds #

inflateInit() accepts a zlib wrapper. inflateInit2() with -15 accepts raw DEFLATE, 15 + 16 accepts gzip, and 15 + 32 enables zlib-or-gzip autodetection. Supply output space and call inflate() until input is consumed, output is full, or zlib returns a terminal status. Handle Z_NEED_DICT by supplying the preset dictionary identified by the wrapper through inflateSetDictionary(); do not guess a dictionary from unrelated input.

On Z_STREAM_END, avail_in can be nonzero: those bytes follow the completed stream and must not be silently discarded. For concatenated gzip members, record the remaining input, reset or reinitialize the inflater, and process each member while validating each member trailer. A single successful first member does not prove that trailing bytes are valid gzip data. Raw DEFLATE has no wrapper checksum; zlib validates Adler-32, and gzip validates CRC-32 and ISIZE.

Decompression can expand dramatically. Enforce output, allocation, ratio or work, and deadline limits before accepting untrusted data. Reject Z_DATA_ERROR, truncated input, invalid distances, and output-limit violations. Do not treat bytes produced before an error as complete or authenticated application data.

Cleanup and Higher-Level APIs #

Call inflateEnd() or deflateEnd() exactly once for each successful initialization. Those functions release internal allocations but do not flush pending output. compress() and uncompress() are convenient for bounded one-shot buffers; gzopen() and gzread() provide gzip file access. Use the streaming API when input size is unknown, output must be bounded, or a protocol needs explicit flush boundaries.

zlib Streaming API

Stream State and Initialization #

Correct Deflate Loops #

Inflating with Bounds #

Cleanup and Higher-Level APIs #

References #

Comments