I have a dynamodb table with a number of global secondary indices on it. When I query one particular index I get a response that is missing slightly over 400 items, but only for one specific value of the range key.
How do I troubleshoot this?
>>> resp = table.query(IndexName='group_index', KeyConditionExpression=Key('group').eq('CLOUD') & Key('gsi_artificial_pk').eq(1))
>>> [x for x in resp['Items'] if x['id'] == 18692]
[]
>>> item = table.get_item(Key={'id': '18962'})['Item']
>>> item['group']
'CLOUD'
>>> item['gsi_artificial_pk']
Decimal('1')
To add to my confusion, if I run the same query on the same index with the same key conditions from the console, id 18962 is present in the response.
For every other value of 'group' that we use, I get a complete set of responses from the same query.
Even my rubber duck hasn't been able to help here
Probably your query call is returning the first page of results and you’re not noticing the LastEvaluatedKey in the response and the need to pull the subsequent pages.
The console and CLI will pull additional pages for you implicitly. That’s why you see the data there.