ı installed netdata on my server and connected it to the netdata dashboard, but I noticed that I cannot see the Docker containers that are currently running on the server on the dashboard. To fix this, I updated the compose.yml file with the following volumes section:
netdata:
image: netdata/netdata
++pid:"host"
container_name: netdata
hostname: example.com # set to fqdn of host
ports:
- 19999:19999
restart: unless-stopped
cap_add:
- SYS_PTRACE
security_opt:
- apparmor:unconfined
volumes:
- netdataconfig:/etc/netdata
- netdatalib:/var/lib/netdata
- netdatacache:/var/cache/netdata
- /etc/passwd:/host/etc/passwd:ro
- /etc/group:/host/etc/group:ro
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /etc/os-release:/host/etc/os-release:ro
- /sys/fs/cgroup:/host/sys/fs/cgroup:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /user-service:/host/user-service:ro
and then I restarted the container. However, when I checked the logs of the netdata container, I saw the following error:
2023-01-05 10:08:38: cgroup-name.sh: INFO: docker container
'7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e' is named 'notification-service'
2023-01-05 10:08:38: cgroup-name.sh: INFO: cgroup 'system.slice_docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope' is called 'notification-service'
2023-01-05 10:08:39: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : Using host prefix directory '/host'
2023-01-05 10:08:39: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open pid_from_cgroup() file '/host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope/tasks'. (errno 2, No such file or directory)
2023-01-05 10:08:39: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope'
2023-01-05 10:08:39: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope'
When I go to the bash of the netdata container, the file path /host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1bs89cc234ba1sd266278a0s9f61280f8b3bb707dd79f9fb46e.scope
actually exists. But there is no file named tasks in this file path.
How can I solve this error and see every container running on the server on the dashboard?
I solved the above error by adding the pid line, but I still can't see the containers running on the server on the dashboard. netdata container logs as follows:
2023-01-05 12:43:35: cgroup-name.sh: INFO: cgroup 'init.scope' is called 'init.scope'
2023-01-05 12:43:35: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928/json
2023-01-05 12:43:35: cgroup-name.sh: INFO: docker container 'd1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928' is named 'definition-service'
2023-01-05 12:43:35: cgroup-name.sh: INFO: cgroup 'system.slice_docker-d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928.scope' is called 'definition-service'
2023-01-05 12:43:36: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : Using host prefix directory '/host'
2023-01-05 12:43:36: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928.scope'
2023-01-05 12:43:37: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928.scope'
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : child pid 1853221 exited with code 1.
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to network namespace of pid 1342633 (errno 1, Operation not permitted)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to pid namespace of pid 1342633 (errno 1, Operation not permitted)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to mount namespace of pid 1342633 (errno 1, Operation not permitted)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : PROCFILE: Cannot open file '/proc/net/dev' (errno 2, No such file or directory)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open file '/proc/net/dev'
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : cannot read cgroup interface list.
2023-01-05 12:43:37: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad/json
2023-01-05 12:43:37: cgroup-name.sh: INFO: docker container '459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad' is named 'mongo_db'
2023-01-05 12:43:37: cgroup-name.sh: INFO: cgroup 'system.slice_docker-459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad.scope' is called 'mongo_db'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : Using host prefix directory '/host'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad.scope'
2023-01-05 12:43:38: ACLK STA [30c26177-2ef2-4768-b509-6e58e18e2643 (example.com)]: QUEUED REMOVED ALERTS
2023-01-05 12:43:38: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad.scope'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : child pid 1853268 exited with code 1.
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to network namespace of pid 1343129 (errno 1, Operation not permitted)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to pid namespace of pid 1343129 (errno 1, Operation not permitted)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to mount namespace of pid 1343129 (errno 1, Operation not permitted)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : PROCFILE: Cannot open file '/proc/net/dev' (errno 2, No such file or directory)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open file '/proc/net/dev'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : cannot read cgroup interface list.
2023-01-05 12:43:39: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179/json
2023-01-05 12:43:39: cgroup-name.sh: INFO: docker container 'ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179' is named 'mail-service'
2023-01-05 12:43:39: cgroup-name.sh: INFO: cgroup 'system.slice_docker-ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179.scope' is called 'mail-service'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : Using host prefix directory '/host'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179.scope'
2023-01-05 12:43:40: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179.scope'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : child pid 1853317 exited with code 1.
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to network namespace of pid 1499298 (errno 1, Operation not permitted)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to pid namespace of pid 1499298 (errno 1, Operation not permitted)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to mount namespace of pid 1499298 (errno 1, Operation not permitted)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : PROCFILE: Cannot open file '/proc/net/dev' (errno 2, No such file or directory)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open file '/proc/net/dev'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : cannot read cgroup interface list.
2023-01-05 12:43:41: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/bcb761b2cd1e89d530cc8973ed21a894e21bc50d0d7138eb8605705ba4ea7b32/json
2023-01-05 12:43:41: cgroup-name.sh: INFO: docker container 'bcb761b2cd1e89d530cc8973ed21a894e21bc50d0d7138eb8605705ba4ea7b32' is named 'netdata'
2023-01-05 12:43:41: cgroup-name.sh: INFO: cgroup 'system.slice_docker-bcb761b2cd1e89d530cc8973ed21a894e21bc50d0d7138eb8605705ba4ea7b32.scope' is called 'netdata'
I saw this similar issue https://github.com/netdata/netdata/issues/11069 and the fix seemed to be to add pid: "host"
, check this comment
https://github.com/netdata/netdata/issues/11069#issuecomment-952228202
Can you try and see if it works?