[SOLVED] Converting Pandas DF to Numpy Array gives me a # of features error when trying to predict?

I have a TPOT regressor set up to predict stock prices on a dataset (after some feature engineering), and I ran into an issue when the XGBoost regressor was involved, I would receive an error that said:

feature_names mismatch:

...and it would then display the list of column names for my dataset. A solution was brought up for this issue over at Github that suggested to convert the dataframes of the X features and Y label to a Numpy Array during the train_test_split to take care of it, so that's what I did, but now I receive an error:

X_train, X_test, Y_train, Y_test = train_test_split(X.values, Y.values, test_size = test_size, random_state = seed)
print('[INFO] Printing the shapes of the training/testing feature/label sets...')
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)


[INFO] Printing the shapes of the training/testing feature/label sets...
(1374, 68)
(459, 68)
(1374,)
(459,)

Best pipeline: ExtraTreesRegressor(DecisionTreeRegressor(input_matrix, max_depth=1, min_samples_leaf=9, min_samples_split=11), bootstrap=False, max_features=0.8500000000000001, min_samples_leaf=1, min_samples_split=9, n_estimators=100)

Traceback (most recent call last):
  File "main2.py", line 656, in <module>
    predictions = best_model.predict(X_test)

File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\tpot\base.py", line 921, in predict
        return self.fitted_pipeline_.predict(features)
      File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\utils\metaestimators.py", line 116, in <lambda>
        out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs)
      File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\pipeline.py", line 422, in predict
        return self.steps[-1][-1].predict(Xt, **predict_params)
      File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\ensemble\forest.py", line 693, in predict
        X = self._validate_X_predict(X)
      File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\ensemble\forest.py", line 359, in _validate_X_predict
        return self.estimators_[0]._validate_X_predict(X, check_input=True)
      File "C:\Users\windowshopr\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\tree\tree.py", line 402, in _validate_X_predict
        % (self.n_features_, n_features))
    ValueError: Number of features of the model must match the input. Model n_features is 68 and input n_features is 69

The issue at Github is now closed, but I'm hoping someone here can explain what I'm missing here? As you can see there's 68 feature columns and 1 label column. And you'll also see that the model this time doesn't even use XGBoost, but I'd like to be able to have any model it comes up with work with the .predict() function.

UPDATE WTIH CODE

Ok I'm seriously stuck here. I've posted a working code below to duplicate the error. Let me know what you see. Input stock ticker CLVS. I've added print shapes of the dataframes and arrays throughout the entire process and it still says the shapes are fine so what am I not seeing? You'll need Pandas 0.23 (yes old version) and TPOT and DASK installed. Thanks:

def main():


    # 1. Input a stock ticker
    ticker_input = input('Which stock ticker would you like to predict?') # Start with CLVS for testing
    print('Getting the historical data for: ',ticker_input)










    # 2. Download the historical daily data
    # Import dependencies
    from datetime import datetime
    from pandas_datareader import data as web
    import pandas as pd
    pd.options.display.float_format = '{:,.2f}'.format
    import seaborn as sns
    import matplotlib.pyplot as plt
    import random
    import os
    import numpy as np
    import time
    # Downloading historical data as dataframe
    ex = 'yahoo'
    start = datetime(2000, 1, 1)
    end = datetime.now()
    dataset = web.DataReader(ticker_input, ex, start, end) #.reset_index()









    # 3. Construct the dataframe from the historical data
    # Only use the Adj Close, and use the open price
    # of the current day. Then shift all the other
    # data 1 day to make the dataset include the 
    # previous day's values for each. 

    # (This is because on the trading day, we won't know what the 
    # High or Low or Close or Volume is, but we would
    # know the Open.)
    dataset = dataset.drop(['Close'],axis=1)
    dataset['PrevOpen'] = dataset['Open'].shift(1)
    dataset['PrevHigh'] = dataset['High'].shift(1)
    dataset['PrevLow'] = dataset['Low'].shift(1)
    dataset['PrevAdjClose'] = dataset['Adj Close'].shift(1)
    dataset['PrevVol'] = dataset['Volume'].shift(1)

    dataset = dataset.drop(['High'],axis=1)
    dataset = dataset.drop(['Low'],axis=1)
    dataset = dataset.drop(['Volume'],axis=1)

    # Add in moving averages based on Opening prices
    dataset['9MA'] = dataset['Open'].rolling(window=9).mean()
    dataset['20MA'] = dataset['Open'].rolling(window=20).mean()



    # Get which industry the stock is in to get the industry performance data
    from bs4 import BeautifulSoup
    import requests
    headers = requests.utils.default_headers() 
    headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'
    # Get the industry name of the stock
    url = 'https://finance.yahoo.com/quote/' + ticker_input + '/profile'
    page = requests.get(url)
    soup = BeautifulSoup(page.text, 'html.parser')
    table = soup.find('p', {'class' :'D(ib) Va(t)'})
    industry = table.findAll('span')
    indust = industry[3].text
    print(indust)
    print('Getting Industry ETF historical data...')
    # Then get historical data for that industry's ETF
    if indust == "Biotechnology":
        etf_ticker = "IBB"
    elif indust == "Specialty Retail":
        etf_ticker = "XRT"
    elif indust == "Oil & Gas E&P":
        etf_ticker = "XOP"
    ex = 'yahoo'
    etf_df = web.DataReader(etf_ticker, ex, start, end)
    dataset['PrevIndOpen'] = etf_df['Open'].shift(1)
    dataset['PrevIndHigh'] = etf_df['High'].shift(1)
    dataset['PrevIndLow'] = etf_df['Low'].shift(1)
    dataset['PrevIndClose'] = etf_df['Adj Close'].shift(1)
    dataset['PrevIndVol'] = etf_df['Volume'].shift(1)





    # Reshape the dataframe to put Adj Close at the far right
    # so when we export the predictions dataset, the predictions
    # column will be right next to it for easier analysis
    dataset = dataset[['Open','9MA','20MA','PrevOpen','PrevHigh','PrevLow','PrevAdjClose','PrevVol','PrevIndOpen','PrevIndHigh','PrevIndLow','PrevIndClose','PrevIndVol','Adj Close']]










    # Disable the Future Warnings that repeat "needlessly" (for now)
    import warnings
    warnings.simplefilter(action='ignore', category=FutureWarning)
    warnings.filterwarnings("ignore")









    # 5. Explore the inital dataset
    # Show the shape of the dataset
    print("[INFO] features shape : {}".format(dataset.shape))

    # Print the feature names
    print("[INFO] dataset names : {}".format(dataset.columns))

    # Convert the dataframe into a Pandas dataframe and print the first 5 rows
    df = pd.DataFrame(dataset)
    print("[INFO] df type : {}".format(type(df)))
    print("[INFO] df shape: {}".format(df.shape))
    print(df.head())

    # Specify the column names and print
    df.columns = dataset.columns
    #print('[INFO] df shape with features:')
    #print(df.head())
    # This prints the same as above

    # Find any columns with missing values? If you find them, you either have to:
    # 1. Replace the missing value with a large negative number (e.g. -999).
    # 2. Replace the missing value with mean of the column.
    # 3. Replace the missing value with median of the column.
    # Because of our 1 day shift, the first row will have empty values,
    # so we'll drop them as one day won't make much difference in our entire model
    print('[INFO] Checking for any columns with no values...')
    df = df.dropna(how='any')
    print(pd.isnull(df).any())


    # Ensure numeric datatypes of the dataframe.
    # If a column has different datatype such as string or character, 
    # we need to map that column to a numeric datatype such as integer 
    # or float. For this dataset, the Date index column is one.
    print('[INFO] Feature types:')
    print(df.dtypes)

    # Print a statistical summary of the dataset for reference
    print('[INFO] Print a statistical summary of dataset:')
    print(df.describe())




    # # Reset the index column for FeatureTools to use Date as the index, then it'll revert it back after feature stuff is done
    # df = df.reset_index()


    # This is not  good way to drop the rows here because if there are any
    # nan values in the middle of the dataset, those will get lost too.
    # Need to work with this
    df = df.dropna()
    print(df)


    # 4. Hold out a prediction dataset to predict on later
    prediction_df = df.tail(90).copy()
    df = df.iloc[:-90,:].copy() # subtracting 90 rows/days from the dataset to use as the predictions dataset later








    # 7. Split the dataset into features (X) and target (Y)
    # Split into features (x) and target (y) and print the shapes of them
    X = df.drop("Adj Close", axis=1)
    Y = df["Adj Close"]
    print('Shape of features: ', X.shape)
    print('Shape of target: ', Y.shape)
    # Standardize the data. Commenting this out until you can figure out how to
    # unscale the prediction dataset for review
    #from sklearn.preprocessing import StandardScaler, MinMaxScaler
    #scaler = MinMaxScaler().fit(X)
    #scaled_X = scaler.transform(X)

    print('Printing X and Y shape :')
    print(X.shape)
    print(Y.shape)







    # 8. Split dataset into training and validation data
    # Split the data into training and testing data and print their shapes
    from sklearn.model_selection import train_test_split
    seed = 9
    test_size = 0.25
    X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = test_size, random_state = seed)
    print('[INFO] Printing the shapes of the training/testing feature/label sets...')
    print(X_train.shape)
    print(X_test.shape)
    print(Y_train.shape)
    print(Y_test.shape)

    X_train=X_train.values
    X_test=X_test.values
    Y_train=Y_train.values
    Y_test=Y_test.values
    print('[INFO] Printing the arrays of the training/testing feature/label sets...')
    print(X_train.shape)
    print(X_test.shape)
    print(Y_train.shape)
    print(Y_test.shape)







    # 9. Start a TPOT Auto Regression to find the best Regression model and export feature importances
    from sklearn.metrics import mean_squared_error, r2_score, explained_variance_score
    from tpot import TPOTRegressor
    import os
    # Create a custom config dictionary for TPOT to use.
    # I've made this list full of Regressors that use the
    # .feature_importances_ attribute. How to implement XGBoost
    # into the plotting of feature importances below? IF XGBOOST is 
    # present in the final model, then plot one way, ELSE, plot the
    # way it is now?
    tpot_config = {



        'sklearn.ensemble.ExtraTreesRegressor': {
            'n_estimators': [100],
            'max_features': np.arange(0.05, 1.01, 0.05),
            'min_samples_split': range(2, 21),
            'min_samples_leaf': range(1, 21),
            'bootstrap': [True, False]
        },



        'sklearn.tree.DecisionTreeRegressor': {
            'max_depth': range(1, 11),
            'min_samples_split': range(2, 21),
            'min_samples_leaf': range(1, 21)
        },

        'sklearn.ensemble.RandomForestRegressor': {
            'n_estimators': [100],
            'max_features': np.arange(0.05, 1.01, 0.05),
            'min_samples_split': range(2, 21),
            'min_samples_leaf': range(1, 21),
            'bootstrap': [True, False]
        },


        # Preprocesssors
        'sklearn.preprocessing.Binarizer': {
            'threshold': np.arange(0.0, 1.01, 0.05)
        },

        'sklearn.decomposition.FastICA': {
            'tol': np.arange(0.0, 1.01, 0.05)
        },

        'sklearn.cluster.FeatureAgglomeration': {
            'linkage': ['ward', 'complete', 'average'],
            'affinity': ['euclidean', 'l1', 'l2', 'manhattan', 'cosine']
        },

        'sklearn.preprocessing.MaxAbsScaler': {
        },

        'sklearn.preprocessing.MinMaxScaler': {
        },

        'sklearn.preprocessing.Normalizer': {
            'norm': ['l1', 'l2', 'max']
        },

        'sklearn.kernel_approximation.Nystroem': {
            'kernel': ['rbf', 'cosine', 'chi2', 'laplacian', 'polynomial', 'poly', 'linear', 'additive_chi2', 'sigmoid'],
            'gamma': np.arange(0.0, 1.01, 0.05),
            'n_components': range(1, 11)
        },

        'sklearn.decomposition.PCA': {
            'svd_solver': ['randomized'],
            'iterated_power': range(1, 11)
        },

        'sklearn.preprocessing.PolynomialFeatures': {
            'degree': [2],
            'include_bias': [False],
            'interaction_only': [False]
        },

        'sklearn.kernel_approximation.RBFSampler': {
            'gamma': np.arange(0.0, 1.01, 0.05)
        },

        'sklearn.preprocessing.RobustScaler': {
        },

        'sklearn.preprocessing.StandardScaler': {
        },

        'tpot.builtins.ZeroCount': {
        },

        'tpot.builtins.OneHotEncoder': {
            'minimum_fraction': [0.05, 0.1, 0.15, 0.2, 0.25],
            'sparse': [False],
            'threshold': [10]
        },


        # Selectors
        'sklearn.feature_selection.SelectFwe': {
            'alpha': np.arange(0, 0.05, 0.001),
            'score_func': {
                'sklearn.feature_selection.f_regression': None
            }
        },

        'sklearn.feature_selection.SelectPercentile': {
            'percentile': range(1, 100),
            'score_func': {
                'sklearn.feature_selection.f_regression': None
            }
        },

        'sklearn.feature_selection.VarianceThreshold': {
            'threshold': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2]
        },

        'sklearn.feature_selection.SelectFromModel': {
            'threshold': np.arange(0, 1.01, 0.05),
            'estimator': {
                'sklearn.ensemble.ExtraTreesRegressor': {
                    'n_estimators': [100],
                    'max_features': np.arange(0.05, 1.01, 0.05)
                }
            }
        }

    }

    # Cross Validation folds to run
    folds   = 10
    # Start the TPOT regression
    best_model = TPOTRegressor(use_dask=True,n_jobs=-1,config_dict=tpot_config, cv=folds, 
                               generations=5, population_size=20, verbosity=2, random_state=seed) #memory='./PipelineCache',       memory='auto',
    best_model.fit(X_train, Y_train)

    # Export the TPOT pipeline if you want to use it for anything later
    if os.path.exists('./Exported Pipelines'):
        pass
    else:
        os.mkdir('./Exported Pipelines')
    best_model.export('./Exported Pipelines/' + ticker_input + '-prediction-pipeline.py')

    # Extract what the best pipeline was and fit it to the training set
    # to get an idea of the most important features used by the model
    exctracted_best_model = best_model.fitted_pipeline_.steps[-1][1]
    # Train the `exctracted_best_model` using the training/vildation set.
    # You need to use the whole dataset in order to get feature importance for all the
    # features in your dataset.
    exctracted_best_model.fit(X_train, Y_train)

    # plot model's feature importance and save the plot for later
    feature_importance = exctracted_best_model.feature_importances_
    feature_importance = 100.0 * (feature_importance / feature_importance.max())
    sorted_idx = np.argsort(feature_importance)
    pos        = np.arange(sorted_idx.shape[0]) + .5
    plt.barh(pos, feature_importance[sorted_idx], align='center')
    plt.yticks(pos, df.columns[sorted_idx])
    plt.xlabel('Relative Importance')
    plt.title('Variable Importance')
    plt.savefig("feature_importance.png")
    plt.clf()
    plt.close()






print(X_test.shape)



    # 10. See the stats of the validation predictions from the tuned model and export more plots
    # Make predictions using the tuned model and display error metrics
    # R2 and Explained Variance, best is 1
    predictions = best_model.predict(X_test)
    print('=============================')
    print("TPOT's final score on testing dataset is : ", best_model.score(X_test, Y_test))
    print('=============================')
    print("[INFO] MSE on test set : {}".format(round(mean_squared_error(Y_test, predictions), 3)))
    print('[INFO] R2 Score on test set : {}'.format(round(r2_score(Y_test, predictions), 3)))
    print('[INFO] Explained Variance Score on test set : {}'.format(round(explained_variance_score(Y_test, predictions), 3)))

    # Plot between predictions and Y_test
    x_axis = np.array(range(0, predictions.shape[0]))
    plt.plot(x_axis, predictions, linestyle="--", marker="o", alpha=0.7, color='r', label="predictions")
    plt.plot(x_axis, Y_test, linestyle="--", marker="o", alpha=0.7, color='g', label="Y_test")
    plt.xlabel('Row number')
    plt.ylabel('PRICE')
    plt.title('Predictions vs Y_test')
    plt.legend(loc='lower right')
    plt.savefig("predictions_vs_ytest.png")
    plt.clf()
    plt.close()










    # 11. Use the model on the held-out prediction dataset
    # Now, run the model on the prediction dataset
    features = prediction_df.drop(['Adj Close'], axis=1)
    labels = prediction_df['Adj Close']
    # Fit the model to the prediction_df and predict the labels
    #tpot.fit(features, labels)
    results = best_model.predict(features)
    predictions_list = []
    for preds in results:
        predictions_list.append(preds)
    prediction_df['Predictions'] = predictions_list
    prediction_df.to_csv('Final Predictions Performance.csv', index=True)
    print('============================')
    print("[INFO] MSE on prediction set : {}".format(round(mean_squared_error(labels, results), 3)))
    print('[INFO] R2 Score on prediction set : {}'.format(round(r2_score(labels, results), 3)))
    print('[INFO] Explained Variance Score on prediction set : {}'.format(round(explained_variance_score(labels, results), 3)))










    # 12. Review the exported .csv file of the predictions, and review all your plots
    print('DONE!')


if __name__ == "__main__":
    main()