python-3.xnumpyhashminhash

ufunc 'bitwise_and' not supported for the input types Minhash


I am using Python 3.7.1 for making minhash a list of string. The code is as follows.

import mmh3
import random
import string
import itertools
from datasketch import MinHash

def grouper(iterable,n=2):
    return ["".join(x) for x in list(itertools.permutations(iterable, n))]

def _hash_func(d):
    return mmh3.hash(d)

def _run_minhash(data, seed):

    m = MinHash(num_perm=128 ,hashfunc=_hash_func)
    for d in data:
        m.update(d.encode('utf8'))
    return m.count()

if __name__ == '__main__':
    st = string.ascii_uppercase
    ngrams  = grouper(st,n=4)
    print(_run_minhash(ngrams,seed=12))

To get speedup I am using mmh3.hash as mentioned in datasketch.Minhash documentation (mmh3.hash32 not available anymore) which produces the following error

TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Traceback (most recent call last):
  File "hashes.py", line 41, in 
    print(_run_minhash(ngrams,seed=12))
  File "hashes.py", line 35, in _run_minhash
    m.update(d.encode('utf8'))
  File "/home/nithin/miniconda3/lib/python3.7/site-packages/datasketch/minhash.py", line 134, in update
    phv = np.bitwise_and((a * hv + b) % _mersenne_prime, np.uint64(_max_hash))
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

I did not find a specific reason the reason and solution for this. How should I resolve this?. Or is there any other way to speed up the minhash calculation. Thanks in advance for your time – if I’ve missed out anything, over- or under-emphasized a specific point let me know in the comments.


Solution

  • same problem here and problem solved by adding signed=False to the call to the mmh3.hash function

    def _hash_func(b):
        return mmh3.hash(b,signed=False)