Metricbeat WARN Cannot index event

I've seen a few posts like this, but none of them solved my problem, so:

I created a elastic cluster following this tutorial: https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls-docker.html

I've set the built in passwords, and I can reach elasticsearch

$ curl -k --user elastic:$ELASTIC_PWD https://localhost:9200/
{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "hqrGkTUGR0W2Clsaxp75pQ",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I copied the certificate authority to the host machine (where I will be running metricbeat) as follows:

sudo -s
mkdir -p /etc/pki/root
docker exec kibana cat /usr/share/elasticsearch/config/certificates/ca/ca.crt > /etc/pki/root/ca.pem

I can check the certificate is there:

$ cat /etc/pki/root/ca.pem
-----BEGIN CERTIFICATE-----
<certificate>
-----END CERTIFICATE-----

So far so good, then, I installed metricbeat following this tutorial: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html

And set the following configs at /etc/metricbeat/metricbeat.yml:

setup.kibana:
  host: "https://localhost:5601"
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "metricbeat"
  password: "<password>"
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

The metricbeat user I've created in Kibana following this: https://www.elastic.co/guide/en/beats/metricbeat/current/feature-roles.html

Basically, I created a role called metricbeat_setup with cluster privileges monitor and manage_ilm and index privilege manage on metricbeat-*, then, I created the metricbeat user with roles metricbeat_setup, kibana_admin, ingest_admin, beats_admin, beats_system

I think my problem might be here, I didn't quite understand the necessary permissions so I just went overkill, anyway, continuing...

I ran the setup sudo metricbeat setup -e and everything went fine, no errors in setup, then I started it with sudo service metricbeat start, but nothing gets to elasticsearch, and when I look at sudo service metricbeat status, I get multiple lines of WARN Cannot index event

I tried to look at the logs in /var/log/metricbeat/ but found nothing useful there, also in elasticsearch logs in docker, no useful information as well.

The metricbeat-* index is created and the shard is allocated:

$ curl -k --user elastic:$ELASTIC_PWD https://localhost:9200/_cat/indices/metricbeat-*?pretty
green open metricbeat-7.7.0-2020.05.25-000001 p_0nuiX1S8SARk9QZK01EA 1 1 0 0 566b 283b

$ curl -k --user elastic:$ELASTIC_PWD https://localhost:9200/_cluster/allocation/explain?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  },
  "status" : 400
}

So, please, what am I missing here? I can't find any relevant information in logs or in posts, I've seen posts here saying this was related to lack of disk space, I have 14G free, so I guess that's not the problem. Also, I know metricbeat can connect to both elasticsearch and kibana, because it was able to create the index and the dashboard, I can open the dashboard in kibana but there's no data.

What am I doing wrong?

Thank you

To be able to write in the index, the user needs to have the privilege create_doc in the desired index.

In this case the user needs to have the index privilege of create_doc for every index named metricbeat-*.

source: grant privileges and roles needed for publishing