Is this a bug of TokyoCabinet?

It's basically a binary tree which first searches against hash to decide whether it's left or right:

if(hash > rec.hash){
  off = rec.left;
  entoff = rec.off + (sizeof(uint8_t) + sizeof(uint8_t));
} else if(hash < rec.hash){
  off = rec.right;
  entoff = rec.off + (sizeof(uint8_t) + sizeof(uint8_t)) +
    (hdb->ba64 ? sizeof(uint64_t) : sizeof(uint32_t));
} else {
  if(!rec.kbuf && !tchdbreadrecbody(hdb, &rec)) return false;
  int kcmp = tcreckeycmp(kbuf, ksiz, rec.kbuf, rec.ksiz);
  if(kcmp > 0){
    off = rec.left;
    ...
  } else if(kcmp < 0){
    off = rec.right;
    ...

Here's how hash calculated:

static uint64_t tchdbbidx(TCHDB *hdb, const char *kbuf, int ksiz, uint8_t *hp){
  ...
  uint32_t hash = 751;
  const char *rp = kbuf + ksiz;
  while(ksiz--){
    ...
    hash = (hash * 31) ^ *(uint8_t *)--rp;
  }
  *hp = hash;
  ...
}

But it seems the way the hash calculated can't ensure the orderness of keys,

is it a bug?

It's not trying to order the keys by the value of the keys themselves. It's ordering them first by hash, and then by key value in the case of a hash collision.

So no, it is not a bug. Unless you can cite documentation saying that this type of table orders by key value.