Backblaze Slow Upload Speed

I'm using Backblaze for an e-commerce like website I'm building where I allow the user to upload a set of photos for a given product, I upload the image along with a scaled down thumbnail to BackBlaze, and store the URL in Firestore.

The issue I'm running into is the INCREDIBLY slow upload speeds. When a user uploads a < 1MB photo it takes 23-27 seconds. Weirdly enough, a 7-10MB photo takes roughly 15-17 seconds. Both of these seem incredibly slow, and on top of this, download speeds to displaying these can also sometimes be quite slow. I put Cloudflare infront of the Backblaze bucket to try to speed up load times, but every few page loads there is one or two thumbnails (less than .5MB) that take 5+ seconds to load.

This is my route:

@router.post('/productImage', response_model=UploadImageResponse, status_code=201)
def upload_product_image_route(file: UploadFile, Authorize: AuthJWT = Depends()):
    Authorize.jwt_required()
    image_content = file.file.read()
    img = PIL_Image.open(io.BytesIO(image_content))

    try:
        name: str = str(uuid.uuid4())
        thumbnail_url = create_thumbnail_for_image(img, name)
        image_url = upload_image_to_cloud_storage(img, name)
        return UploadImageResponse(thumbnail=thumbnail_url, image=image_url)
    except FileUploadError as e:
        raise HTTPException(status_code=500, detail='Failed to process image.')
    finally:
        img.close()
        file.file.close()

I added some metrics to see which task is taking the longest and create_thumbnail_for_image takes less than 5 seconds usually but my upload_image_to_cloud_storage takes 15-30 seconds depending on the photo.

def upload_image_to_cloud_storage(image: PIL_Image, image_name: str) -> str:
    s3 = _get_storage_resource()

    buffer = io.BytesIO()
    image.save(buffer, format=image.format)
    buffer.seek(0)

    try:
        s3.Bucket(BUCKET_NAME).upload_fileobj(buffer, image_name)
        return f'{BASE_HOST_URL}/{image_name}'
    except Exception as e:
        raise FileUploadError(e)


def create_thumbnail_for_image(image: PIL_Image, image_name: str, size: [int, int] = None) -> str:
    if size is None:
        size = [350, 350]

    image_name = f'{size[0]}-{size[1]}-{image_name}'

    f = image.format
    img: PIL_Image = image.copy()
    img.thumbnail((size[0], size[1]))
    img.format = image.format

    r = upload_image_to_cloud_storage(img, image_name)
    img.close()
    return r

Any idea on why these upload speeds may be taking so long?

Solution

I tried to reproduce your results, so I boiled your code down to this:

def upload_image_to_cloud_storage(image_name: str, size: int):
    s3 = _get_storage_resource()

    # Generate random data so results aren't skewed by compression
    buffer = io.BytesIO(bytearray(map(random.getrandbits, (8,) * size)))

    try:
        s3.Bucket(BUCKET_NAME).upload_fileobj(buffer, image_name)
    except Exception as e:
        raise FileUploadError(e)


SIZES = [
    500 * 1024,         # 500 kB
    10 * 1024 * 1024    # 10 MB
]
TRIALS = 10

times = [[], []]

for i in range(len(SIZES)):
    for j in range(TRIALS):
        name: str = str(uuid.uuid4())
        start = time.perf_counter_ns()
        print('.', end='', flush=True)
        upload_image_to_cloud_storage(name, SIZES[i])
        times[i].append(time.perf_counter_ns() - start)
    print('', flush=True)

for i in range(len(SIZES)):
    mean = sum(times[i]) / len(times[i])
    print(f'Size: {SIZES[i]}, time: {mean // 1000000} ms')

These are the results:

..........
..........
Size: 512000, time: 1489.0 ms
Size: 10485760, time: 15186.0 ms

BTW - in case you're wondering if I have a particularly speedy network connection, my broadband is out, and my laptop is tethered to my phone, so this is via 5G.

What happens when you try the above code?

Also, where are you located, and which region is your Backblaze account in?