What does the Datomic exception `read-transactor-location-failed` mean?

I'm getting the following exception when deleting or creating a Datomic database (with Datomic Pro 0.9.4899):

Exception in thread "main" java.lang.IllegalArgumentException: :db.error/read-transactor-location-failed Could not read transactor location from storage
    at datomic.error$arg.invoke(error.clj:55)
    at datomic.coordination$check_peer_version.invoke(coordination.clj:138)
    at datomic.coordination$lookup_compatible_transactor_endpoint.invoke(coordination.clj:149)
    at datomic.peer$send_admin_request$fn__8594.invoke(peer.clj:713)
    at datomic.peer$send_admin_request.invoke(peer.clj:707)
    at datomic.peer$delete_database.invoke(peer.clj:745)
    at clojure.lang.Var.invoke(Var.java:379)
    at datomic.Peer.deleteDatabase(Peer.java:142)
    at datomic.api$delete_database.invoke(api.clj:23)
...

What does this exception mean? What are some common reasons it can occur?

Solution

When transactors launch, they write their location to storage. They actually write their location with every heartbeat message. If you see this message when a peer tries to connect, it means that either no transactor location has been written, or that they are otherwise unable to read from storage (and hit this error first in attempting to do so).

A cause for this for an AWS config -- say with AWS CloudFormation and DynamoDB as your storage -- would be launching with a transactor properties file that is not correctly configured to write to storage. When the peers attempt to connect to storage they won't be able to read the transactor's location. This could be because the transactor and peers are pointed at different storages. You could also run into permissions issues, e.g. the transactor does not have permissions to write to the storage you've specified (security groups config, etc.)

Either way, the general cause of this is what the error message says - that a transactor location can't be read from storage. It looks like the missing detail here is with how this happens when the system is healthy and correctly configured: that peers read the transactor location from storage and that the transactor writes its location as part of heartbeat (and on a failover event, the standby transactor will write its location when it takes over). There are various system configuration issues which might contribute to this.