The update rule TD(0) Q-Learning:
Q(t-1) = (1-alpha) * Q(t-1) + (alpha) * (Reward(t-1) + gamma* Max( Q(t) ) )
Then take either the current best action (to optimize) or a random action (to explorer)
Where MaxNextQ is the maximum Q that can be got in the next state...
But in TD(1) I think update rule will be:
Q(t-2) = (1-alpha) * Q(t-2) + (alpha) * (Reward(t-2) + gamma * Reward(t-1) + gamma * gamma * Max( Q(t) ) )
My question:
The term gamma * Reward(t-1) means that I will always take my best action at t-1 .. which I think will prevent exploring..
Can someone give me a hint?
Thanks
You are talking about "eligibility traces" usage, right? See the equations and the algorithm.
Notice the e_t(s, a) equation there. No penalty is applied when using an exploration step.