pythonpandasnumpytensorflownormalize

How can I cleanly normalize data and then denormalize it later?


I am using Anaconda with a Tensorflow neural network. Most of my data is stored with pandas.
I am attempting to predict cryptocurrency markets. I am aware that this lots of people are probably doing this and it is most likely not going to be very effective, I'm mostly doing it to familiarize myself with Tensorflow and Anaconda tools.
I am fairly new to this, so if I am doing something wrong or suboptimally please let me know.

Here is how I aquire and handle the data:

  1. Download datasets from quandl.com into pandas DataFrames
  2. Select the desired columns from each downloaded dataset
  3. Concatenate the DataFrames
  4. Drop all NaNs from the new, merged DataFrame
  5. Normalize each column (independently) to 0.0-1.0 in the new DataFrame using the code
    df = (df - df.min()) / (df.max() - df.min())
  6. Feed the normalized data into my neural network
  7. Denormalize the data (This is the part that I haven't implemented)

Now, my question is, how can I cleanly normalize and then denormalize this data? I realize that if I want to denormalize data, I'm going to need to store the initial df.min() and df.max() values, but this looks ugly and feels cumbersome.
I am aware that I can normalize data with sklearn.preprocessing.MinMaxScaler, but as far as I know I can't denormalize data using this.

It might be that I'm doing something fundamentally wrong here, but I'll be very surprised if there isn't a clean way to normalize and denormalize data with Anaconda or other libraries.


Solution

  • All the scalers in sklearn.preprocessing have inverse_transform method designed just for that.

    For example, to scale and un-scale your DataFrame with MinMaxScaler you could do:

    from sklearn.preprocessing import MinMaxScaler
    scaler = MinMaxScaler()
    scaled = scaler.fit_transform(df)
    unscaled = scaler.inverse_transform(scaled)
    

    Just bear in mind that the transform function (and fit_transform as well) return a numpy.array, and not a pandas.Dataframe.