I have recently attempted to complete a neural network to predict fluctuations within the prices of individual stocks on the stock market, utilising Keras as the framework for the network and Quandl to retrieve the historic adjusted stock prices; within running this program, I primarily utilised the program paradigm and information displayed within a singular tutorial, a link to which is displayed below:
https://www.youtube.com/watch?v=EYnC4ACIt2g&t=2079s
However, the tutorial utilised the "sklearn" Linear Regression module; I altered the program to utilise Keras, which possesses a greater capability for customisation. The program is displayed below:
import tensorflow as tf
import keras
import numpy as np
import quandl
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
df = quandl.get("WIKI/FB")
df = df[['Adj. Close']]
forecast_out = 1
df['Prediction'] = df[['Adj. Close']].shift(-(forecast_out))
X = np.array(df.drop(['Prediction'], 1))
X = X[:-forecast_out]
y = np.array(df['Prediction'])
y = y[:-forecast_out]
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
model = keras.models.Sequential()
model.add(keras.layers.Dense(units = 64, activation = 'relu'))
model.add(keras.layers.Dense(units = 1, activation = 'linear'))
model.compile(loss='mean_absolute_error',
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5, batch_size=32, validation_split = 0.2)
x_forecast = np.array(df.drop(['Prediction'], 1))[-forecast_out:]
print(x_forecast)
prediction = model.predict(x_train)
However, upon running the model with the provided testing information via the model.fit() command, I received this display of the loss and accuracy for each epoch:
Train on 940 samples, validate on 236 samples
Epoch 1/5
940/940 [==============================] - 1s 831us/step - loss: 85.4464 - acc: 0.0000e+00 - val_loss: 76.7483 - val_acc: 0.0000e+00
Epoch 2/5
940/940 [==============================] - 0s 51us/step - loss: 65.6871 - acc: 0.0000e+00 - val_loss: 55.4325 - val_acc: 0.0000e+00
Epoch 3/5
940/940 [==============================] - 0s 52us/step - loss: 43.3484 - acc: 0.0000e+00 - val_loss: 30.5538 - val_acc: 0.0000e+00
Epoch 4/5
940/940 [==============================] - 0s 47us/step - loss: 16.5076 - acc: 0.0011 - val_loss: 1.3096 - val_acc: 0.0042
Epoch 5/5
940/940 [==============================] - 0s 47us/step - loss: 2.0529 - acc: 0.0043 - val_loss: 1.1567 - val_acc: 0.0000e+00
<keras.callbacks.History at 0x7ff1dfa19470>
Provided my relatively small quantity of experience in testing such paradigms, I would prefer to know if this accuracy is satisfactory or not; are the loss and accuracy parameters indicating that the model is running perfectly? What is the variation between them and how may one read them? Lastly, how does Keras characterise them? The documentation for the module did not appear to provide a sufficient quantity of information; however, it may be my examination of them to blame. Thank you for your assistance.
You might get better answers about neural networks/ML on CrossValidated, but I can try to help you here.
In general, it is very very difficult to tell if a neural network is running "properly" — hence in my experience, ML development is a very iterative process with trial and error informed by educated statistical/mathematical guesses.
Let's do a high-level overview of the metrics first:
Loss = how far "off" the model's prediction is from your data.
Accuracy = % of predictions that your model got "right"; i.e. if your model is a function, model(x) = y for a particular data point.
Satisfactory "accuracy" is subjective and depends widely by application/model/your data. However, since you are trying to predict stock price; i.e. a continuous variable, you are doing regression, it doesn't make much sense to me to use a metric like accuracy. I can tell you are doing regression both by your problem formulation — and the linear activation is a strong hint as well.
To see why accuracy doesn't make sense, if I'm predicting the house price based on certain factors, I probably don't care how many predictions I got exactly right, but more how close my predictions overall were. If my regression model $1 off on every house price, I still have an accuracy of 0, but I could potentially still have a good model.
Instead, minimizing the loss function is probably a better way to think of things. Think of it this way: overall, you want to fit some function of your input variables that is "close" to the ground-truth output. For linear regression, the loss function is LMS (Least Mean Squares), which essentially the average squared distance of the residuals. Here you use mean absolute error, which is just the average absolute value of the difference. Both loss functions have pros and cons and I'd encourage you to look into this for your application.
The fact that your error is decreasing is good: this means that the function your model approximates is getting closer and closer to the training data (the residuals are decreasing). Your validation loss is also no greater than the training data, which demonstrates that you are not overfitting on your data either. I would encourage you to simply keep experimenting.