pythonpandasaws-data-wrangler

cast list type value of dictionary column in pandas


I am trying to write a complex type to dynamodb, but getting into the error:

import pandas as pd

df = pd.DataFrame.from_records([{"col_w_status": {"ACTIVE":["ABC100-01"],"INACTIVE":["ABC100"]}}])
col_list = ["col_w_status"]

display(df)

for col in col_list:
    df[col] = df[col].apply(lambda x: dict(x))
    # wr.dynamodb.put_df(df, <tableName>) 

display(df)

Error out with below message:

Unexpected err=TypeError('Unsupported type "<class \'numpy.ndarray\'>" for value "[\'ABC100-01\']"'), type(err)=<class 'TypeError'>

I think nested lambda might work, but I am not sure how to implement it.


Solution

  • In the example provided you have lists inside of the dictionary, I think that should work with DynamoDB. I guess instead of list you have numpy arrays. So probably it would work if you convert arrays to Python lists.

    import pandas as pd
    
    df = pd.DataFrame.from_records([{"col_w_status": {"ACTIVE": np.array(["ABC100-01"]),"INACTIVE": np.array(["ABC100"])}}])
    col_list = ["col_w_status"]
    
    display(df)
    
    def convert_to_native_types(row):
        row = dict(row)
        for key in row:
            row[key] = list(row[key])
        return row
    
    for col in col_list:
        df[col] = df[col].apply(lambda x: convert_to_native_types(x))
        # wr.dynamodb.put_df(df, <tableName>) 
    
    display(df)