pythontiffsatellite

how to read a large image by lines with python

I am trying to process some satellite photo with the size over 1G. I code in python, and I have tried in these ways.

image = np.array(io.imread(each_dir))
image = np.array(TiffImagePlugin.open(each_dir))
image = np.array(Image.open(each_dir))

Only io.read works with photos with size of 200Mb, when the size is larger than 1G, there's an error MemoryError: Unable to allocate 17.1 GiB for an array with shape (156105, 39136, 3) and data type uint8

I have changed my RAM from 16g to 32g, but this doesn't work. So I 'd like to ask is there a way to read the imgae by lines but not read the whole image in a time ? Thank you very much.

Solution

  • I would suggest you tile your rather large TIFF outside of Python before you start. Drawing heavily on John's (@jcupitt) answer here, you can do that with vips.

    I created a 39100x156100 TIFF in my Terminal, and tiled it using vips' "DeepZoom Save" feature:

    vips dzsave big.tif outdir --depth one --tile-size 8192 --overlap 0 --suffix .tif

    It took 38s and 2GB of RAM, so it is pretty frugal. Below is a list of the tiles it produced and their metadata.

    Results

    0_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
0_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
1_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
2_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.010u 0:00.000
3_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
3_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
4_0.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_10.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_11.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_12.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_13.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_14.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_15.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_16.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_17.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_18.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_19.tif TIFF 6332x452 6332x452+0+0 8-bit sRGB 8.18867MiB 0.000u 0:00.000
4_1.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_2.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_3.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_4.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_5.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_6.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_7.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_8.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_9.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000

    Another, completely different approach, might be to use the stream command which is part of the ImageMagick suite - also outside of Python. So, I made a 50,000x50,000 pixel RGB TIFF called big.tif and then streamed off the bottom 1/5 (i.e. rows 40,000 onwards) into an RGB888 raw file like this:

    magick stream -map rgb -storage-type char -extract 50000x10000+0+40000 big.tif bottom.dat

    That command runs in around 18s on my Mac. You can then read that into a Numpy array in Python like this:

    import numpy as np

# Load RGB888 file and reshape back to its correct shape
im = np.fromfile('bottom.dat', dtype=np.uint8).reshape((10000,50000,3))

    In case you are unfamiliar with ImageMagick's geometry specifications, the 50000x10000+0+40000 part in the command above means "extract a 50,000 pixel wide by 10,000 pixel tall region whose top-left corner is 0 pixels in from the left edge and 40,000 pixels down from the top".