logstash

How to convert a string field to array using Logstash


How can I convert a string field to array using Logstash ?

I have a field like below:

"language" : "english"

I need it to be converted to:

"language" : ["english"]

Thanks in advance.


Solution

  • Assuming your field language is already mapped to a datatype (which usually would be text/keyword), converting would cause issues at index time because the array type would not match the text/keyword type.

    Notice that due to dynamic mapping, Elasticsearch will automatically create a mapping to text/keyword for the language field (assuming the first indexed value was a String like "english") even if you haven't set an explicit mapping for that index.

    So depending on your mappings I would suggest to create a new field for that array.

    You can do that with the ruby filter plugin like in the following (untested though):

    filter{
      ruby{
        code => '
          langValue = event.get("language")
    
          langArray = []
          langArray << langValue
    
          event.set("name_of_the_new_field", langArray)
        '
      }
    }
    

    Code explanation:

    In line #4 you create a variable with the value of the language field. In your example this would be english. Please refer to the Event API for detailed usage.

    In line #6 you initialize a new array. In the following line you add the variable with value english to that array. Refer to this page to get more information about ruby arrays.

    Finally in the last line of code you set a new field to the event with the array as its value. Note that the first argument in the paranthesis should be the name of the newly created field.

    I hope I could get you some hints/food for thought to solve your issue.