elasticsearch

Elasticsearch Devtools Queries (Regexp/Wildcard) - Not Working


I'm tring to create a regexp query in elasticsearch devtools and it appears not to be working. I was following the docs on the website:

Elasticsearch Regexp

Elasticsearch Wildcard

I've tried all of the above but for some reason it appears like either i'm doing something wrong (obviously because the data exists but not returned) or the data is not formatted right.

This is the Doc i'm running on:

{
  "took": 2036,
  "timed_out": false,
  "_shards": {
    "total": 287,
    "successful": 287,
    "skipped": 256,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 12.63081,
    "hits": [
      {
        "_index": "winlogbeat-7.15.1-2024.02.29-000029",
        "_id": "GD98kI4BzHuLoaiyJTrr",
        "_score": 12.63081,
        "_source": {
          "message": """Timestamp: 30/03/2024 20:50:37

Message:  ProposalId-7389995 ,MyDomain-New , ABC-3128661328
Time to prepare objects before deleting: 0 ms
Before <FillMany>: 0 ms
After <FillMany>: 5555 ms
After deleting(records): 5555 ms

Category: Warning
Priority: 1
EventId: 1
Severity: Warning
Title:
Machine: MyServer
App Domain: /MyDomain/ServiceName/Machine-133562274298846072
ProcessId: 9999
Process Name: c:\windows\system32\cmd.exe
Thread Name:
Win32 ThreadId:1234
Extended Properties: """
        }
      }
    ]
  }
}

And this is The queries i'm trying to run: Regexp

GET /_search
{
  "_source": ["message"],
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2024-01-01",
              "lte": "2024-08-20"
            }
          }
        },
        {
          "regexp": {
            "message": "After\\ \\<FillMany\\>:\\ [2-9][0-9]{3,}\\ ms"
          }
        }
      ]
    }
  }
}

for the Regexp I've tried also the following:

"message": "After \\<FillMany\\>: [2-9][0-9]{3,}.*"

like on the reference page: Elasticsearch Regexp Flags

Event tried to add the Flags (like it showed on this link): Elasticsearch Regexp Flags Example - Email

Does Anyone knows what I'm doing wrong? I tried to search over the internet explaination regarding how everything works when building up a query, sadly i came empty handed.

hope someone can give me a clue here.


Solution

  • Ok, it took me a while but i finally solved my issue! After reading the specific doc i needed in my index by doing the following:

    Get <index name>/_doc/<_id>
    

    for example:

    {
       "_index": "winlogbeat-8.12.1-2024.01.01-00001",
       "_id": "12345678",
       etc...
    }
    

    i noticed that my "message" field is acctually:

    {
        "event_date": {
            "param1": <all my message details>
        }
    }
    

    so all i had to do is very simple:

    GET /_search
    {
      "_source": ["message"],
      "query": {
        "bool": {
          "filter": [
            {
              "range": {
                "@timestamp": {
                  "gte": "2024-01-01",
                  "lte": "2024-08-20"
                }
              }
            },
            {
              "regexp": {
                "winlog.event_data.param1": "After \\<FillMany\\>: [2-9][0-9]{3,}.*"
              }
            }
          ]
        }
      }
    }
    

    and that solved my problem, i was trying to pull data from a feild that wasn't exsit!

    my advice is to always get the doc you want to pull data from with the id and check how it's built.

    Thanks for everyone who took the time to read my issue and tried to solve my problem, much appriciate to everyone!

    I hope this post will be able to help others in the future.