google-app-enginepython-2.7app-engine-ndb

How can I use 64 bit unsigned integers as entity keys in App Engine's datastore and retain sort order?


I have unsigned 64 bit integers that I would like to use as App Engine datastore keys. Unfortunately the NDB datastore API only seems to allow signed 64 bit integers with a max size of pow(2, 63).

I could two's complement the integers but I rely on the sort order in the datastore for queries so I want 0xffffffffffffffff coming after 0x0 in ascending sort order not before it . I believe this would happen if I two's complement my 64 bit unsigned integers as 0xffffffffffffffff would be interpreted as -1 by the datastore index and 0x0 would be interpreted as 0 for example.

I could change each unsigned 64 bit key from an integer to a string or even a hex string but that would take up a lot more space in the datastore than keeping the keys as 64 bit integers.


Solution

  • Just encode your unsigned values as signed values with a simple constant offset of pow(2, 63).

    This "shifts" all values "downwards" without messing up sort order, from the [0, 2^64) range representable by an unsigned 64-bit integer into the range [-2^63, 2^63) representable by a (typical implementation of a) signed 64-bit integer.

    def encode(unsigned_64_bit_int):
        return unsigned_64_bit_int - pow(2, 63)
    
    def decode(signed_64_bit_int):
        return signed_64_bit_int + pow(2, 63)
    

    (You can also replace pow(2, 63) with the equivalent constant 0x8000000000000000, if you find it clearer or effecting performance in a way that matters.)