pythondaskdask-ml

Problems implementing Dask MinMaxScaler

I am having problems normalizing a dask.dataframe.core.DataFrame using Dask.dask_ml.preprocessing.MinMaxScaler, I am able to use sklearn.preprocessing.MinMaxScaler however I wish to use dask to scale up.

Minimal, Reproducible Example:

# Get data
ddf = dd.read_csv('test.csv') # See below
ddf = ddf.set_index('index')

# Pivot
ddf = ddf.categorize(columns=['item', 'name'])
ddf_p = ddf.pivot_table(index='item', columns='name', values='value', aggfunc='mean')
col = ddf_p.columns.to_list()

# sklearn verison
from sklearn.preprocessing import MinMaxScaler

scaler_s = MinMaxScaler()
scaled_ddf_s = scaler_s.fit_transform(ddf_p[col]) # Works!

# dask verison
from dask_ml.preprocessing import MinMaxScaler

scaler_d = MinMaxScaler()
scaled_values_d = scaler_d.fit_transform(ddf_p[col]) # Doesn't work

Error message:

TypeError: Categorical is not ordered for operation min
you can use .as_ordered() to change the Categorical to an ordered one

Not sure what the 'Categorical' is in the pivoted table, but I have tried to .as_ordered() the index:

from dask_ml.preprocessing import MinMaxScaler

scaler_d = MinMaxScaler()
ddf_p = ddf_p.index.cat.as_ordered()
scaled_values_d = scaler_d.fit_transform(ddf_p[col])

But I get the error message:

NotImplementedError: Series getitem in only supported for other series objects with matching partition structure

Additional information

test.csv:

index,item,name,value
2015-01-01,item_1,A,1
2015-01-01,item_1,B,2
2015-01-01,item_1,C,3
2015-01-01,item_1,D,4
2015-01-01,item_1,E,5
2015-01-02,item_2,A,10
2015-01-02,item_2,B,20
2015-01-02,item_2,C,30
2015-01-02,item_2,D,40
2015-01-02,item_2,E,50
Solution

  • Looking at this answer:

    pivot_table produces a column index which is categorical because you made the original column "Field" categorical. Writing the index to parquet calls reset_index on the data-frame, and pandas cannot add a new value to the columns index, because it is categorical. You can avoid this using ddf.columns = list(ddf.columns).

    Therefore adding ddf_p.columns = list(ddf_p.columns) solved the problem:

    # dask verison
from dask_ml.preprocessing import MinMaxScaler

scaler_d = MinMaxScaler()
ddf_p.columns = list(ddf_p.columns)
scaled_values_d = scaler_d.fit_transform(ddf_p[col]) # Works!