logstashelastic-stacklogstash-groklogparser

How to parse log with "dynamically" double quotes


I have a similar to following logs

4294967295,"adult,low-risk",6564ec78-4995-45b7-b73d-44ee50851dcb,"everything,lost,bla",0

The value inside double quotes should be on the same field so i would get something like

field1 => 4294967295
field2 => "adult,low-risk"
field3 => 36564ec78-4995-45b7-b73d-44ee50851dcb
field4 => "everything,lost,bla"
field5 => 0

But, if the value empty or has a single value the double quotes would not present like:

4294967295,,6564ec78-4995-45b7-b73d-44ee50851dcb,everything,0

Then if i place my dissect/grok like:

%{field1},%{field2},%{field3},%{field4},%{field5}

it would return:

field1 => 4294967295
field2 => "adult
field3 => low-risk"
field4 => 36564ec78-4995-45b7-b73d-44ee50851dcb
field5 => "everything,lost,bla",0

and if i place my dissect/grok like:

%{field1},"%{field2}",%{field3},"%{field4}",%{field5}

it would work but once the value empty or has a single value like i mentioned above it would return _grokparsefailure or __dissectfailure

How do i solve this? Any help would be appreciated, thanks


Solution

  • Using dissect in preference to grok is often a good idea because it has limited functionality, which means it is cheaper. However, dissect does not know about the quoting conventions for commas in csv files. A csv filter does, so if you use

    csv { columns => [ "field1", "field2", "field3", "field4", "field5" ] }
    

    you will get the result you want.