[SOLVED] Why do I get a ChunkedEncodingError in Python when using requests module?

Why do I get a ChunkedEncodingError in Python when using requests module?

I try to query this API https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi with these params ?db=mesh&id=68016019 So the whole URL is https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=mesh&id=68016019 When executing this in Postman everything's fine and the output is as expected:

1: Survival Analysis
A class of statistical procedures for estimating the survival function (function 
of time, starting with a population 100% well at a given time and providing the
percentage of the population still well at later times). The survival analysis is
then used for making inferences about the effects of treatments, prognostic
etc.
*
*
*

BUT when I try to query this via requests module in Python I receive this error:

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

So what am I doing exactly? I just fire this command:

response2 = requests.get(
        "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=mesh&id=" + str(id))

(The id is converted to a string as I got some issues when I don't do it.) For 99.9% of the cases this approach is absolutely fine but then there comes this 0.1% cases I am lost. I do not get what this ChunkedEncodingError is about.

Is this error coming from the API or from my script? Can please someone help?

I tried to query this in Postman -> it works I tried to execute the same query from my script in a complete new script -> it works (???)

but why not in my original script?

Solution

If you are occasionally getting a ChunkedEncodingError, you can implement some retry logic in that case:

Something like this would retry a given URL 3 times (with a 1 second pause between each retry):

import requests
import time

article_id=68016019

for attempt in range(3):
    try:
        res = requests.get(
            f"https://eutils.ncbi.nlm.nih.gov/entre/eutils/efetch.fcgi?db=mesh&id={article_id}")
        break
    except requests.exceptions.ChunkedEncodingError:
        time.sleep(1)
else:
    print("Failed to retrieve url")

You would obviously want to implement better error handling than simply printing a messsage and continuing.