splunksplunk-dashboard

Splunk dynamic conditional formatting

Is it possible in splunk to apply dynamic conditional thresholds

For example I have a service helloworld and it contain multiple endpoints , for this example sake helloworld/greeting and helloworld/process

the thresholds are like this

endpoint,90thPercentile(millisec),95thPercentile(millisec)
helloworld/greeting,20,50
helloworld/process,50,100

Output

endpoint,datetime,execution(millisec),p90(breached/notbreached),p95(breached/notbreached)
helloworld/greeting,04/23/23,8:00:000,25,breached,notbreached 
helloworld/greeting,04/23/23,8:05:000,12,notbreached,notbreached 
helloworld/process,04/23/23,8:00:000,125,breached,breached 
helloworld/process,04/23/23,8:00:000,25,notbreached,notbreached 

Query

...
...
|eval "90"=case(execution>90thPercentile(OF A SPECIFIC ENDPOINT FOR A TOKEN),"breach",1=1,"notbreached"),"95"=case(execution>95thPercentile(OF A SPECIFIC ENDPOINT FOR A TOKEN),"breach",1=1,"notbreached"),

I can easily create hardcoded universal 90thPercentile,95thPercentile tokens, but those will be same for all endpoints, instead i want thresholds applicable per endpoint by looking at that threshold config or token variable

Hope i am making sense, i am exhaused on my research and looks like i need to hard code a panel for each endpoint and hardcode 90th/95thPercentile in eval case statement

Solution

  • The best way to handle dynamically this kind of queries is to use Splunk Lookup functionality

    Lookups can help you to save your thresholds on CSV file in Splunk servers, and refer it in your SPL queries. It's usually used to enrich your Splunk events with static datas, and use them for apply calculations

    1/ Save your treshold file in Lookup

    --> See Lookup Documentation

    2/ Use lookup command to enrich your indexed events

    --> See lookup command Documentation

    3/ Use new created fields to handle all type of endpoint

    Your SPL query will look like this :

    <your base search>
| lookup thresholds.csv endpoint
|eval p90=if(perc90(execution)>90thPercentile(millisec),"breach","notbreached"),
p95=if(perc95(execution)>95thPercentile(millisec),"breach","notbreached")

    Hope it help !