Delphi: EInvalidOp in neural network class (TD-lambda)

I have the following draft for a neural network class. This neural network should learn with TD-lambda. It is started by calling the getRating() function.

But unfortunately, there is an EInvalidOp (invalid floading point operation) error after about 1000 iterations in the following lines:

neuronsHidden[j] := neuronsHidden[j]+neuronsInput[t][i]*weightsInput[i][j]; // input -> hidden

weightsHidden[j][k] := weightsHidden[j][k]+LEARNING_RATE_HIDDEN*tdError[k]*eligibilityTraceOutput[j][k]; // adjust hidden->output weights according to TD-lambda

Why is this error? I can't find the mistake in my code :( Can you help me? Thank you very much in advance!

learningMode: Boolean; // does the network learn and change its weights?
neuronsInput: Array[1..MAX_TIMESTEPS] of Array[1..NEURONS_INPUT] of Extended;
neuronsHidden: Array[1..NEURONS_HIDDEN] of Extended;
neuronsOutput: Array[1..NEURONS_OUTPUT] of Extended;
weightsInput: Array[1..NEURONS_INPUT] of Array[1..NEURONS_HIDDEN] of Extended;
weightsHidden: Array[1..NEURONS_HIDDEN] of Array[1..NEURONS_OUTPUT] of Extended;

[...]

function HyperbolicTangent;
begin
  if x > 5500 then // prevent overflow
    result := 1
  else
    result := (Exp(2*x)-1)/(Exp(2*x)+1);
end;
[...]

The most likely reason for you error is that you are overflowing the accumulators neuronsHidden or weightsHidden. I know nothing about neural networks so I can't offer any explanation as to why this is so.

As a side issue, I question the use of Extended floating point variables. Normally this simply results in extremely slow performance, much slower than when using Double. You may think it gives you more headroom for large numbers, but in reality, if this is a run-away overflow of the nature that I suspect, then using Extended would never save you.

UPDATE OP points out that overflow results in a different exception class. So for EInvalidOp I suspect some square root or trig failure or something like that. Or perhaps a signaling NaN, but since you don't obviously use uninitialised data I'll not persue that.

I can see now that you have been affected by Embarcadero's bizarre decision to break their implementation of Tanh. This used to work perfectly on older versions of Delphi (e.g. D6), but was broken recently. The version you use isn't quite right for large negative input, and uses two calls to Exp when one suffices. I use this version:

const
  MaxTanhDomain = 5678.22249441322; // Ln(MaxExtended)/2

function Tanh(const X: Extended): Extended;
begin
  if X>MaxTanhDomain then begin
    Result := 1.0
  end else if X<-MaxTanhDomain then begin
    Result := -1.0
  end else begin
    Result := Exp(X);
    Result := Result*Result;
    Result := (Result-1.0)/(Result+1.0);
  end;
end;