pythongoogle-cloud-platformbigtablegoogle-cloud-bigtablehappybase

Bigtable Google Happybase Python KeyError when trying to do `counter_inc`


I'm not sure why but counter_inc method is not working, no emulator, pointing to a development real instance.

Snippet:

from google.cloud import bigtable
from google.cloud import happybase

client = bigtable.Client(project='robbie-ai', admin=True)
instance = client.instance('visio-bt-staging')
connection = happybase.Connection(instance=instance)
connection.create_table('commons_TestBTModelsTable', {'family': None, 'counters': None})
table = connection.table('commons_TestBTModelsTable')
table.put('row-key1', {'family:surname': 'Trump'})
print("Getting row 'row-key1': {}".format(table.row(b'row-key1')))
table.counter_inc(b'row1', b'counters:qual1')

If I do table.counter_inc(b'row1', 'counters:qual1') is exactly the same.

When executed as script:

root@2d21638ea17f:/app# python scripts/counters.py
Getting row 'row-key1': {b'family:surname': b'Trump'}
Traceback (most recent call last):
  File "scripts/counters.py", line 28, in <module>
    table.counter_inc(b'row1', b'counters:qual1')
  File "/usr/local/lib/python3.5/dist-packages/google/cloud/happybase/table.py", line 591, in counter_inc
    column_cells = modified_cells[column_family_id][column_qualifier]
KeyError: 'qual1'

Is it a bug or this snippet has a problem?


Solution

  • Definitively a bug, they have different types for the column id qualifier and the returned dict key type, the one that it is returned by commit(), while the one you got from split was decoded into str,

    here: https://github.com/GoogleCloudPlatform/google-cloud-python-happybase/blob/master/src/google/cloud/happybase/table.py#L577

    here https://github.com/GoogleCloudPlatform/google-cloud-python-happybase/blob/master/src/google/cloud/happybase/table.py#L582

    and here https://github.com/GoogleCloudPlatform/google-cloud-python-happybase/blob/master/src/google/cloud/happybase/table.py#L591

    if you go lower level:

    # As it is in the source code
    row = table._low_level_table.row(b'row-key1', append=True)
    column_family_id, column_qualifier = 'qual1'.split(':')
    row.increment_cell_value(column_family_id, column_qualifier, 1)
    row.commit()
    """outputs
    {'counters': {b'qual1': [(b'\x00\x00\x00\x00\x00\x00\x00\x01',
    datetime.datetime(2017, 6, 12, 14, 2, 28, 858000, tzinfo=<UTC>))]}}
    """
    # of course if using the original portion code it breaks with KeyError as the returned key type is bytes
    # Get the cells in the modified column,
    column_cells = modified_cells[column_family_id][column_qualifier]
    """outputs
    KeyError                                  Traceback (most recent call last)
    <ipython-input-24-fe55fb8c47c5> in <module>()
          1 # Get the cells in the modified column,
    ----> 2 column_cells = modified_cells[column_family_id][column_qualifier]
    
    KeyError: 'counters'
    """
    # But it works if re-encoded the column 
    # Get the cells in the modified column,
    column_cells = modified_cells[column_family_id][column_qualifier.encode()]
    column_cells
    [(b'\x00\x00\x00\x00\x00\x00\x00\x04',
      datetime.datetime(2017, 6, 12, 14, 16, 18, 150000, tzinfo=<UTC>))]
    

    This is for Python 3.5.2, anyway is strange that the commit response is returning heterogeneous key types, what seems wrong to me.

    Workaround until they fix it in pypi: Override the counter_inc method with the above encoding