Logstash Elasticsearch output plugin creates a single document index in Elasticsearch

i am newer in ELK, My LOGSTASH 7.8.x loop indefinitely and insert only one row in Elasticsearch ! Here is my sample example :

sample log

2021-07-18 09:15:30,000 INFO Sample log message 01
2022-07-18 10:20:45,111 ERROR Sample log message 02
2023-07-18 11:20:45,222 DEBUG Sample log message 03
2024-07-18 12:20:45,333 WARN Sample log message 04

conf file

input {
  file {
    path => "/home/sample.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    close_older=> "1 second"
 }
}

filter {
    grok {
          match => { "resource" => "%{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:log_message}" }
         }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "sample_logs"
    document_id => "%{my_doc_for_logs}"
    document_type => "_doc"
    codec => "json"
 }
  stdout { codec => rubydebug }
}

By running logstash as follow : bin/logstash -f /etc/logstash/conf.d/sshd.conf --log.level debug

1- Only one line is inserted in Elasticsearch index : sample_logs 2- I have to click on CTRL + C to stop indefinitely loop of logstash, image below :

debug image

cheking Elasticsearch data : only 1 row instead of 4 is inserted !!! curl -XGET "http://localhost:9200/sample_logs/_search?size=1000" | jq .

Elastic Data

1- Only one line is inserted in elasticsearch index : sample_logs 2- I have to click on CTRL + C to stop indefinitly loop of logstash, image below :

The problem is the document_id.

document_id => "%{my_doc_for_logs}"

In your case Elasticsearch indexes all the documents with one and the same id.

Try deleting it, elasticsearch will be indexing with auto generated ids.

If you want a value to assign it, you can use this pattern.

Example:

mutate { add_field => { "[@metadata][id]" => "%{[host][name]}_%{some_field}" } }

document_id => "%{[@metadata][id]}"

Log of indefinitely loop debug below :

[DEBUG] 2024-07-22 09:21:19.583 [pool-3-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2024-07-22 09:21:19.701 [pool-3-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2024-07-22 09:21:19.701 [pool-3-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2024-07-22 09:21:22.781 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
[DEBUG] 2024-07-22 09:21:24.592 [pool-3-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2024-07-22 09:21:24.715 [pool-4-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2024-07-22 09:21:24.716 [pool-4-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2024-07-22 09:21:27.781 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
[DEBUG] 2024-07-22 09:21:29.596 [pool-4-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2024-07-22 09:21:29.728 [pool-3-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2024-07-22 09:21:29.728 [pool-3-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2024-07-22 09:21:32.781 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
[DEBUG] 2024-07-22 09:21:34.599 [pool-3-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2024-07-22 09:21:34.739 [pool-4-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2024-07-22 09:21:34.740 [pool-4-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2024-07-22 09:21:37.781 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
[DEBUG] 2024-07-22 09:21:39.604 [pool-4-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2024-07-22 09:21:39.752 [pool-3-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2024-07-22 09:21:39.753 [pool-3-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2024-07-22 09:21:42.782 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
[DEBUG] 2024-07-22 09:21:44.607 [pool-3-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2024-07-22 09:21:44.762 [pool-4-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2024-07-22 09:21:44.762 [pool-4-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2024-07-22 09:21:47.782 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.