pythonjsonpython-requestscompressionzstd

How to detect zstd compression?

I am currently working on a python application, that works with facebook api's. As we all know, facebook loves their own technology and is working with zstd for data compression.

The problem: facebook is returning either a uncompressed response with normal json or if the response is longer, it is responding with a zstd compressed json.

My current code is something like this:

import zstd
import json


def handle_response(response)
    json = None
    try:
        json = json.loads(zstd.decompress(response.content))
    except:
        json = json.loads(response.text)

    return json

I am currently wondering, if there is a more clean way to do this and even detect zstd.

Solution

  • What you're doing is fine.

    You could, I suppose, check to see if the stream starts with the four bytes 28 b5 2f fd. If it doesn't, it's not a zstd stream. If it does, it may be a zstd stream. In the latter case, you would try to decompress and if it fails, you would fall back to just copying the input.

    That turns out to be exactly the same as what you're already doing, since the first thing that zstd.decompress is doing is to look for that signature.