Imagine we have an access to continuous memory region with an alignment of 8 and size of the region being a multiple of 8.
Now, let's say I want to use a particular segment of that region, but I want an exclusive access to it, so I use first 8 bytes of that region to set a "busy" flag along with the length of the segment that I'm using.
The problem is that I need to set and subsequently clear this flag (after I'm done with using it) in an atomic fashion to avoid race conditions, if multiple threads periodically allocate some memory from the same region.
Using a bit of unsafe we can transmute those initial 8 bytes into AtomicUsize (assuming 64 bit processor) and then use atomic load/store/CAS operations on it, which should solve the problem with parallel access, at least in theory.
Now the question, would I be violating some compiler invariants and introduce UB with such an approach, or is it a completely sound thing to do?
Assumptions: segment "allocations" always happen at properly (8 byte) aligned addresses, initial 8 bytes are never accidentally overwritten or treated in non-atomic fashion.
This touches some subtle parts of the memory model.
First of all arises the question of whether there will be both atomic and non-atomic reads/writes to the memory region at the same time. Mixed atomic and non-atomic reads (which are UB in C++) were just defined to not be UB in Rust. However, when any of them is a write (either the atomic or the non-atomic), this becomes UB. So you have to ensure all accesses are atomic.
Then there is the question of how exactly you obtained the pointer. If you have a mutable reference to the region (&mut [u8] or something alike), you are allowed to convert this into &mut Atomic (and there is even safe, although unstable, API for that). But then, no other reference is allowed to be used to access this region (because of the mutable reference).
If you have a shared reference (&[u8]), the story becomes more complicated. Under Stacked Borrows, it is immediate Undefined Behavior to transmute a shared reference to a reference to UnsafeCell (which atomics contain). Tree Borrows, on the other hand, allow this, but it does not allow writes into this UnsafeCell, only reads. This is even what of the very few parts of the memory model that is final in some way, because LLVM's noalias mandates this, at least for shared references that are function parameters (and therefore, it may be possible to trigger an actual miscompilation with this).
If you have a raw pointer that was never a reference (e.g. all usages of the region use this pointer, then it doesn't matter if previously it was a mutable reference), then it is fine to transmute it to a &Atomic (not &mut Atomic!), as long as you make sure no references are created for the non-atomic type.