According to Does annotations order matter? the order shouldn't matter, however for these two it seems that the order does matter.
If the transaction occurs within the retry, then the transaction could roll back, throw an exception, then spring will retry it. However if the retry occurs within the transaction, there won't be a rollback between each retry as the retry will catch the exception and go again.
There could be also situations like the database details changing between retries.
There was a recent change to the order; see https://github.com/spring-projects/spring-retry/pull/335 and https://github.com/spring-projects/spring-retry/pull/333#pullrequestreview-1330977780
It will be available in 2.0.1 (currently in 2.0.1-SNAPSHOT).