[SOLVED] Handling mapper_parsing_exception in OpenSearch for dynamic data types from Amazon EventBridge

Handling mapper_parsing_exception in OpenSearch for dynamic data types from Amazon EventBridge

I'm trying to store data obtained from Amazon EventBridge into OpenSearch. While it generally works as intended, it appears that EventBridge can send data with the same parameters but in different types. When this happens, OpenSearch throws a mapper_parsing_exception because different shapes of data are sent to the same field.

For example, the following requests will cause an error on the second request due to type mismatch.

// First request
{
  "foo": false
}

// Second request
{
  "foo": "bar"
}

To circumvent this, I have considered the following strategies:

A) Including type information in the field name:

{
  "foo.boolean": false,
  "foo_text": "bar"
}

B) Creating different indices for different types: I have not tried this yet, but I am concerned it might lead to an excessive number of indices.

C) Storing everything as strings: I would prefer to avoid this if possible, as I need to be able to search by any field name.

Are there any other good ideas or best practices for handling this situation in OpenSearch, especially when dealing with dynamic data types from Amazon EventBridge? Any advice or suggestions would be greatly appreciated. Thank you.

Solution