From what I understand, in HTTP/1.1, headers and always sent in plaintext, and message bodies can optionally be compressed using an off-the-shelf algorithm like gzip.
In HTTP/2.0, a dedicated header compression algorithm was introduced to compress headers (HPACK).
This is a naïve question, but I'm wondering why an HTTP-specific was necessary. Wouldn't it make more sense to simply apply gzip over everything sent over TCP? (i.e. compress together all headers and bodies of all messages sharing a connection, using a compression algorithm that can encode/decode in a streaming fashion)
Are the compression gains of a custom-built solution really that significant, or am I not seeing some additional constraint making a naïve solution infeasible?
I believe the main reason for using HPACK instead of gzip is that gzip is vulnerable to the CRIME attack. An attacker can infer the characters you send by observing the compressed data's length because gzip uses DEFLATE for compression, which eliminates duplicate strings. However, HPACK significantly reduces the risk of this type of attack. You can find more details in the RFC.