For instance consider this function, that could be used in a WM to allow moving a window from one desktop to another on a given display,

moveWindowSTM :: Display -> Window -> Desktop -> Desktop -> STM ()
moveWindowSTM disp win a b = do
  wa <- readTVar ma
  wb <- readTVar mb
  writeTVar ma (Set.delete win wa)
  writeTVar mb (Set.insert win wb)
 where
  ma = disp ! a
  mb = disp ! b

and obviously its IO wrapper,

moveWindow :: Display -> Window -> Desktop -> Desktop -> IO ()
moveWindow disp win a b = atomically $ moveWindowSTM disp win a b

and then assume that

our transaction succeeds at the third attempt,
because at the first attempt it gets invalidated by another concurrent transaction being commited that changes the value inside ma right after the wa <- readTVar ma of our transaction,
and at the second attempt it gets invalidated by another concurrent transaction being committed that changes the contents of either or both of ma and mb right after writeTVar ma (Set.delete win wa) of our transaction.

How would the transaction log evolve in this case, and at which point would the validation fail?

The relevant excerpt from Parallel and Concurrent Programming in Haskell by Simon Marlow is below (but similar information is available in the paper Beautiful concurrency by Simon Peyton Jones):

An STM transaction works by accumulating a log of readTVar and writeTVar operations that have happened so far during the transaction. The log is used in three ways:

By storing writeTVar operations in the log rather than applying them to main memory immediately, discarding the effects of a transaction is easy; we just throw away the log. Hence, aborting a transaction has a fixed small cost.

Each readTVar must traverse the log to check whether the TVar was written by an earlier writeTVar. Hence, readTVar is an O(n) operation in the length of the log.

Because the log contains a record of all the readTVar operations, it can be used to discover the full set of TVars read during the transaction, which we need to know in order to implement retry.

When a transaction reaches the end, the STM implementation compares the log against the contents of memory. If the current contents of memory match the values read by readTVar, the effects of the transaction are committed to memory, and if not, the log is discarded and the transaction runs again from the beginning. This process takes place atomically by locking all the TVars involved in the transaction for the duration. The STM implementation in GHC does not use global locks; only the TVars involved in the transaction are locked during commit, so transactions operating on disjoint sets of TVars can proceed without interference.