What is HTTP Chunked Encoding? How and When is it Used?

What is Chunked Encoding?

Introduction

Chunked encoding, provided under HTTP 1.1, involves dividing data into smaller blocks, or chunks. Crucially, chunks are sent independently, usually through a single persistent connection. The first chunk specifies the type in the Transfer-Encoding header.

The receiver doesn't see the entire file, as it might not be completely available in the first place. For example, a server might read and send a large file to a client, or generate a table of results from a database.

An everyday use case for HTTP chunked encoding is in video streaming applications. Breaking up data-heavy video files into small pieces allows for much faster load times than a browser loading a whole video file before it renders a single frame. Video streaming also takes advantage of the encoding format to deliver chunks (video blocks) as they arrive at the server.

Pausing and resuming

Chunked encoding isn't exclusive to video streaming or file downloads. It's used to transmit any large amount of data to a client. There is one caveat to chunked encoding, though. Due to the use of persistent connections, and because the total size is known to the user at the end of a download, pausing and resuming isn't possible.

How does chunked encoding work?

Generally, the client sees the first chunk almost immediately. The user experience is pretty good. Chunked encoding can work for any type of data, such as text:

`6\r\n
Hello,\r\n
6\r\n
world!\r\n
\r\n`

The 6\r\n lines indicate the length of the ensuing content, while the \r and \n elements are control characters, signaling the end of a line. Data can also be compressed and chunked simultaneously, though the compression algorithm varies. Gzip and deflate are commonly used compression formats.

Note: Servers never compress blocks individually. They compress data first, then split it up.

Finally, the server (or sender) transmits a final empty block:

The final block consists of two control characters (\r\n), marking the end of a chunked stream.