logstashkibanalogstash-grokbro

Grok filter isn't matching to the bro httplog data

I am trying to use ELK to visualize the BRO log data. I found multiple grok filters online and it keeps failing to match the pattern to the data. One of the filters I tried using is:

grok {
          match => [ "message", "(?<ts>(.*?))\t(?<uid>(.*?))\t(?<id.orig_h>(.*?))\t(?<id.orig_p>(.*?))\t(?<id.resp_h>(.*?))\t(?<id.resp_p>(.*?))\t(?<trans_depth>(.*?))\t(?<method>(.*?))\t(?<bro_host>(.*?))\t(?<uri>(.*?))\t(?<referrer>(.*?))\t(?<user_agent>(.*?))\t(?<request_body_len>(.*?))\t(?<response_body_len>(.*?))\t(?<status_code>(.*?))\t(?<status_msg>(.*?))\t(?<info_code>(.*?))\t(?<info_msg>(.*?))\t(?<filename>(.*?))\t(?<http_tags>(.*?))\t(?<username>(.*?))\t(?<password>(.*?))\t(?<proxied>(.*?))\t(?<orig_fuids>(.*?))\t(?<orig_mime_types>(.*?))\t(?<resp_fuids>(.*?))\t(?<resp_mime_types>(.*))" ]
        }

The bro data I am trying to ingest is as follows:

#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   http
#open   2018-11-27-18-31-02
#fields ts  uid id.orig_h   id.orig_p   id.resp_h   id.resp_p   trans_depth method  host    uri referrer    version user_agent  request_body_len    response_body_len   status_code status_msg  info_code   info_msg    tags    username    password    proxied orig_fuids  orig_filenames  orig_mime_types resp_fuids  resp_filenames  resp_mime_types
#types  time    string  addr    port    addr    port    count   string  string  string  string  string  string  count   count   count   string  count   string  set[enum]   string  string  set[string] vector[string]  vector[string]  vector[string]  vector[string]  vector[string]  vector[string]
1543343462.308603   CrJmZi31EU3tUXba3c  10.100.130.72   38396   216.58.217.110  80  1   -   -   -   -   1.1 -   0   219 301 Moved Permanently   -   -   (empty) -   -   -   -   -   -   FXhQ5K1ydVhFnz9Agi  -   text/html
1543344229.051726   CLj9eD4BcFR42BRHV1  10.100.130.72   37452   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FO0Zko4uvyxeC8LDx4  -   text/plain
1543345395.827176   C6Kdv49oODjjkgeFk   10.100.130.72   37464   169.254.169.254 80  1   -   -   -   -   1.0 -   0   345 404 Not Found   -   -   (empty) -   -   -   -   -   -   FW4NGDCyMNR43J4Hf   -   text/html
1543345691.165771   CNaObqkLN9imdehl4   10.100.130.72   37466   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FmUSygO8ocHKTN8L3   -   text/plain
1543347316.900516   Ck5CsV2hr56axo3rzl  10.100.130.72   37486   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FXKDmj3kllpKuJnSkg  -   text/plain
1543348718.870063   CFBClg1jRpmBp4ElYb  10.100.130.72   37506   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   F02j4T12ssIF2tYFF5  -   text/plain
1543348995.827387   CPMwHt2g13sPqdiXE1  10.100.130.72   37508   169.254.169.254 80  1   -   -   -   -   1.0 -   0   345 404 Not Found   -   -   (empty) -   -   -   -   -   -   FsbLPY8A3gpuBkM7l   -   text/html
1543350095.640070   CObHQk2ARejHIWBcgc  10.100.130.72   37518   169.254.169.254 80  1   -   -   -   -   1.0 -   0   13  200 OK  -   -   (empty) -   -   -   -   -   -   FxCY9C2fOP4dHO2Dkj  -   text/plain

Thanks -JP

Solution
    1. Ignore the lines starting with #.
    2. I used this Grok parser given and it worked. You can use the grokdebugger to check the parser. https://grokdebug.herokuapp.com/
    3. It can be an issue that the data source is using spaces rather than tabs. Verify the data.

    4. I have used => instead of using a comma(,). Change the code to 

      grok {
  match => [ "message" => "(?<ts>(.*?))\t(?<uid>(.*?))\t(?<id.orig_h>(.*?))\t(?<id.orig_p>(.*?))\t(?<id.resp_h>(.*?))\t(?<id.resp_p>(.*?))\t(?<trans_depth>(.*?))\t(?<method>(.*?))\t(?<bro_host>(.*?))\t(?<uri>(.*?))\t(?<referrer>(.*?))\t(?<user_agent>(.*?))\t(?<request_body_len>(.*?))\t(?<response_body_len>(.*?))\t(?<status_code>(.*?))\t(?<status_msg>(.*?))\t(?<info_code>(.*?))\t(?<info_msg>(.*?))\t(?<filename>(.*?))\t(?<http_tags>(.*?))\t(?<username>(.*?))\t(?<password>(.*?))\t(?<proxied>(.*?))\t(?<orig_fuids>(.*?))\t(?<orig_mime_types>(.*?))\t(?<resp_fuids>(.*?))\t(?<resp_mime_types>(.*))" ]
}

    5. You can also use csv parser rather than Grok if the error still persists.