My Snowplow enricher loads data to a Kinesis Data Stream but Kinesis responds with error 400 and the message Member must have length less than or equal to 500 (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException and I cannot find this exception in the AmazonKinesis documentation. I also do not understand the message, 500 what, bytes? When I look at Kinesis within the AWS console, there is no setting set to 500 and also I do not see anything related to maximum sizes.
Can someone tell me what this 500 is about?
The error means that the Enrich app is trying to write more than 500 records to Kinesis at once. The PutRequests action of the Kinesis API has a limit of 500 entries. This is documented here.
In your Enrich config file, you have buffer section, where you can specify:
byteLimit = 10000000
recordLimit = 480
timeLimit = 60000
A flush is triggered when one of those limits is hit, whichever occurs first.