kqlazure-log-analyticsazure-sentinelfortigate

Writing parser with | parse kind=regex operator


I`m trying to make a parser for FortiAnalyzer. I recieve log in next format (example from Fortinet documentation) : 2020-05-12 17:01:16 log_id=0001010018 type=event subtype=system pri=information desc="User login/logout successful" user="admin" userfrom="JSON(10.100.55.254)" msg="user 'admin' with profile 'Super_User' logout from JSON(10.100.55.254)" session_id=5108 adminprof="Super_User"

So variable=value . There are about 30-40 variables and they order may vary.

So i parse information in next way : | parse kind=regex (name_of_my_column) with * "log_id" * "=" Log_ID: string "type=" Type: string "subtype=" Subtype: string and so on.... When i write 17 variables in one query it gives me an error : parse: regex mode exceeded max allowed matching groups. actual = 17, limit = 16 When i start new line with | parse kind=regex function, previous variable (#16) has all information that comes after variable #16 Variable #17 has the right information. end of first query with | parse kind=regex and start of a second information in variable #16 and #17 Can someone please give an advice, what should i add to my query to see results in right way? I also tried to use |extend function, but i think, that is not right way in my case because variables can be in different positions in log. Will be thankful to any advice.


Solution

  • you can use the parse-kv operator.

    for example:

    print input = ```2020-05-12 17:01:16 log_id=0001010018 type=event subtype=system pri=information desc="User login/logout successful" user="admin" userfrom="JSON(10.100.55.254)" msg="user 'admin' with profile 'Super_User' logout from JSON(10.100.55.254)" session_id=5108 adminprof="Super_User"```
    | parse-kv input as (log_id:string, type:string, subtype:string, pri:string, ['desc']:string, user:string, userfrom:string, msg:string, session_id:long, adminprof:string) with (pair_delimiter=' ', kv_delimiter='=', quote='"')
    | project-away input
    
    log_id type subtype pri desc user userfrom msg session_id adminprof
    0001010018 event system information User login/logout successful admin JSON(10.100.55.254) user 'admin' with profile 'Super_User' logout from JSON(10.100.55.254) 5108 Super_User