jsonrustactix-webreqwestserde-json

Can't parse JSON from StackAPI: expected value", line: 1, column: 1


I'm trying to parse the JSON from this endpoint: https://api.stackexchange.com/2.2/users/13029516?&site=stackoverflow.

It looks like this:

{
"items": [
{
"badge_counts": {
"bronze": 27,
"silver": 14,
"gold": 0
},
"account_id": 17932441,
"is_employee": false,
"last_modified_date": 1677242400,
"last_access_date": 1682981080,
"reputation_change_year": 313,
"reputation_change_quarter": 29,
"reputation_change_month": 0,
"reputation_change_week": 10,
"reputation_change_day": 0,
"reputation": 2234,
"creation_date": 1583703647,
"user_type": "registered",
"user_id": 13029516,
"website_url": "https://matthewtrent.me/links",
"link": "https://stackoverflow.com/users/13029516/matthew-trent",
"profile_image": "https://lh3.googleusercontent.com/a/AATXAJwHVeuBCVo52xa3k_IryQ1llGDNWkHRYiO0R2xh=k-s256",
"display_name": "Matthew Trent"
}
],
"has_more": false,
"quota_max": 300,
"quota_remaining": 271
}

but I can't. I'm using the reqwest package. Here is my endpoint where I'm using it (simplified):

#[get("/test")]
async fn my_function(username: web::Query<QueryParams>) -> impl Responder {
    println!("recieved!");

    match reqwest
        ::get("https://api.stackexchange.com/2.2/users/13029516?&site=stackoverflow")
    .await
    {
        Ok(response) => {
            if response.status() != 200 {
                return HttpResponse::InternalServerError().finish();
            } else {
                let res = response.json::<serde_json::Value>().await.unwrap(); // <-- where it panics
                println!("Success! {:?}", res);
            }
        }
        Err(_) => return HttpResponse::InternalServerError().finish(),
    };
    HttpResponse::Ok().finish()
}

It panics at the line let res = response.json::<serde_json::Value>().await.unwrap();. What's odd is that I've hit different endpoints (ex: https://pokeapi.co/api/v2/pokemon/ditto) and it works perfectly and lists out the JSON results. My only guess it that the Stack API isn't properly formatted JSON?

Here's the output for when I run it against https://api.stackexchange.com/2.2/users/13029516?&site=stackoverflow:

recieved!
thread 'actix-rt|system:0|arbiter:0' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
recieved!
thread 'actix-rt|system:0|arbiter:1' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:2' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:3' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:4' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:5' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:6' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:7' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70
recieved!
thread 'actix-rt|system:0|arbiter:0' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }', src/services/stack_overflow.rs:42:70

Any help would be great, thanks.

1


Solution

  • This fails because the Stack Exchange API does not follow the HTTP spec exactly.

    While our API is HTTP based, we chose to diverge from standard HTTP in one particular area. During normal operation, we guarantee that all responses are compressed, either with GZIP or DEFLATE.

    If you inspect the request and response headers, you will see that the API returns a Content-Encoding: gzip despite reqwest not sending an Accept-Encoding containing gzip.

    Reqwest decides to ignore the unknown encoding and try to read it anyway, resulting in junk data.

    Fortunately, you can enable the gzip feature of reqwest so that it can both send the Accept-Encoding: gzip header and decode GZIP-encoded responses. There is also a deflate feature if you would like to use DEFLATE.

    [dependencies]
    reqwest = { verson = "0.11", features = ["json", "gzip", "deflate"] }
    

    You can also turn off this behavior at runtime with the ClientBuilder::gzip method.