logstashelastic-stackkibana-6

In Kibana, I have fields that contains a question mark `?` not showing in metric field

In Kibana, I have fields that contains a question mark ?. The goal is to create a filter that excludes all entries containing a question mark in the field. So, when i'm trying to create a metric under Aggregation with Term those fields which are in ? mark are not visible there, Please help to understand to a newbie ..

Below is the logstash.conf with the filters i'm using along with screen shot i have attached, please suggest what mistake i'm doing and what can be done..

I have ELK version : 6.2.x

# cat logstash-syslog.conf
input {
  file {
    path => [ "/scratch/rsyslog/*/messages.log" ]
    type => "syslog"
  }
  file {
    path => [ "/scratch/rsyslog/Aug/messages.log" ]
    type => "apic_logs"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "message", "_type", "_index", "_score", "path"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
}
  if [type] == "apic_logs" {
    grok {
      match => { "message" => "%{CISCOTIMESTAMP:syslog_timestamp} %{CISCOTIMESTAMP} %{SYSLOGHOST:syslog_hostname} (?<prog>[\w._/%-]+) %{SYSLOG5424SD:f1}%{SYSLOG5424SD:f2}%{SYSLOG5424SD:f3}%{SYSLOG5424SD:f4}%{SYSLOG5424SD:f5} %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "message", "_type", "_index", "_score", "path"]
   }
 }
}
output {
        if [type] == "syslog" {
        elasticsearch {
                hosts => "noida-elk:9200"
                manage_template => false
                index => "syslog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

output {
        if [type] == "apic_logs" {
        elasticsearch {
                hosts => "noida-elk:9200"
                manage_template => false
                index => "apic_logs-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

enter image description here

Availbale Fields

Solution

  • I fixed my issue!

    Why do I see the symbol ? by fields in the Kibana Discover page When you open the Discover page in Kibana, you might see a question mark ? by fields that are listed in the available fields section instead of the character t. When you reload the list of fields, the type of fields is analyzed, and the question mark ? is replaced by the character t.

    Be sure to check Mark the box include system indices at the extreme right in the below screen shot.

    enter image description here

    Rearranging field columns in the table You can rearrange the field columns in the table. Mouse over the header of the column you want to move, and click the Move column to the left button or the Move column to the right button.

    Reloading the list of fields Complete the following steps to reload the list of fields that are displayed in Kibana:

    Select the Management page, then select Index Patterns to list the indexes that are available.

    Select the index pattern for your space to see every field and the field's associated core type as recorded by Elasticsearch.

    Click the Reload field list button Reload field list to reload the index pattern fields.

    The list of fields is refreshed.