bigdatavisualizationpoint-cloudsholoviewsdatashader

interactive big 2D point cloud data visualization on map with python

I have a 2D point cloud data time series. The shape of the data is [N, T]. N is a big number（millions) and T is the number of 2D point cloud images and generally less than 300. I also have the coordinates of these N points with shape of [N, 2] (the longitudes and the latitudes). I want to interactively visualize these data in a map.

Specifically, I want: 1.) interactively zoom in or out to show both the specific region and the big picture; 2.) I can click one point and plot the time series of this point; 3.) I can switch which image to be shown among the T images.

I know it is impossible and unnecessary to plot that much of points. A dynamic aggregation and rasterization would be a better choice. https://egms.land.copernicus.eu/ is a very good example. It meets all of my requirement. It only shows all points when zoom into a small region. Otherwise, only a small number of points are shown. It could be the reason of its efficiency.

The only similar technique I found in Python ecosystem is Holoviews+Datashader. But I find it is still not very efficient. The reason could be that datashader rasterizes the point cloud on the fly, but that website rasterizes the point cloud previously and only on several specific zoom level.

I want to know if there are any python packages that can visualize such large dataset efficiently, or if there are any better ways to utilize holoviews and datashader. Thanks!

Solution

  • I just post what I have implemented in my open source package Moraine which focus on radar interferometry.

    I implemented a function ras_pyramid which save images of different resolutions into the disk. Here is a piece of the log of this function:

    2024-07-17 21:50:07 - darr_info - INFO - downsampled ras in level 0 dask array shape, chunksize, dtype: (2500, 1834, 17), (256, 256, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 1 dask array shape, chunksize, dtype: (1250, 917, 17), (256, 256, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 2 dask array shape, chunksize, dtype: (625, 459, 17), (256, 256, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 3 dask array shape, chunksize, dtype: (313, 230, 17), (256, 230, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 4 dask array shape, chunksize, dtype: (157, 115, 17), (157, 115, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 5 dask array shape, chunksize, dtype: (79, 58, 17), (79, 58, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 6 dask array shape, chunksize, dtype: (40, 29, 17), (40, 29, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 7 dask array shape, chunksize, dtype: (20, 15, 17), (20, 15, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 8 dask array shape, chunksize, dtype: (10, 8, 17), (10, 8, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 9 dask array shape, chunksize, dtype: (5, 4, 17), (5, 4, 1), complex64
2024-07-17 21:50:08 - darr_info - INFO - downsampled ras in level 10 dask array shape, chunksize, dtype: (3, 2, 17), (3, 2, 1), complex64

    Here the shape of the image stack is (2500, 1834, 17). I downsample it to (1250, 917, 17), (625, 459, 17),... (3, 2, 17). And the downsampled image is save to the disk in zarr format with chunk size of (256,256,1).

    I also create the visualization function ras_plot based on holoviews which automatically decide the current zoom level and extent to read needed chunks from disk. E.g.: enter image description here Note the resolution of this gif is reduced to meet stackoverflow's requirement. Please refer to https://kanglcn.github.io/moraine/CLI/plot.html for better resolution.

    I also implemented the functions for pyramid generation and plotting for point cloud data. The visualization of point cloud data: enter image description here The mechanism is, the point cloud data are rasterized into different resolutions. The raw point cloud data and rasterized data are saved to disk by the function pc_pyramid. Then just like ras_plot, pc_plot also automatically read the needed data from disk for plotting. I.e., when looking at large region, a coarse resolution raster image is loaded. When looking at small resion, the raw point cloud data is loaded.