pythongoogle-earth-engine

Import multiple CSV files from local computer into the google earth engine in python


I have multiple CSV files contain different points. I want to import these files into the google earth engine using python instead of importing them one-by-one. I have written the below code but it shows the following error:

EEException: Collection.loadTable: Collection asset 'D:/.../2022-8-20.csv' not found.

I appreciate if somebody help me to fix it.

import ee
from ee.batch import Export

# Check authentication
ee.Authenticate()
ee.Initialize(project='water-quality')
print(ee.String('Hello from the Earth Engine servers!').getInfo())

file_path = 'D:/.../2022-8-20.csv'

gee_asset_id = 'users/.../CSV'

csv_fc = ee.FeatureCollection(file_path)

task = ee.batch.Export.table.toAsset(collection = csv_fc, description = 'testimport',  assetId = gee_asset_id)
task.start()

In the below link, there is a suggestion to use ee.data.startTableIngestion(request_id, params, allow_overwrite=False) but I do not know how to use it.

gis.stackexchange


Solution

  • I found the solution (with a slight modification) from the question asked in the following link:

    how to upload big files to GEE as assets with the API?

    import ee
    import geemap
    import pandas as pd 
    import geopandas as gpd
    import json
    import os
    import shutil
    
    # Check authentication
    ee.Authenticate()
    ee.Initialize(project='water-quality')
    print(ee.String('Hello from the Earth Engine servers!').getInfo())
      
    # specify path of the files
    base_dir_file = 'D:/.../CSV folder'
    fnames_file = [os.path.join(base_dir_file, fname) for fname in os.listdir(base_dir_file)]
    dfs_names_file = (os.listdir(base_dir_file))
    
    for i in range(len(fnames_file)):
        print(i)
        df = pd.read_csv(fnames_file[i], sep=None, engine='python')
        gdf = gpd.GeoDataFrame(df, crs='EPSG:4326', geometry = gpd.points_from_xy(df['X'], df['Y']))
            
        # convert it into geo-json 
        json_df = json.loads(gdf.to_json())
            
        # create a gee object with geemap
        ee_object = geemap.geojson_to_ee(json_df)
        
        # split filename and the extention
        file, ext = os.path.splitext(dfs_names_file[i]) 
                        
        task = ee.batch.Export.table.toAsset(collection = ee_object, description = file , assetId = 'users/.../%s'%file)
        task.start()