I'm having a tough time getting my agents working. For some reason my agents pull an empty catalog from the master.
When running puppet agent --test on one of my nodes I get:
root@grid:/home/jenssels# puppet agent --test
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (192.168.1.12)
Notice: Catalog compiled by nexus.lan
Info: Caching catalog for grid.lan
Info: Applying configuration version '1712841471'
Notice: Applied catalog in 0.01 seconds
When I run puppet catalog find grid.lan on the master I see the correct catalog.
When I run puppet catalog find or puppet catalog find grid.lan on the agent node I get a completely different catalog.
Rebooting both master and agent changes nothing.
My agent is running the default config. The only commands I used on the agent node are:
cd /tmp
wget https://apt.puppet.com/puppet8-release-jammy.deb
sudo dpkg -i puppet8-release-jammy.deb
apt-get update -y && apt-get install puppet-agent -y
export PATH=/opt/puppetlabs/bin:$PATH
puppet agent --test
On the server side I did modify some settings as my manifests are on an nfs share (mounted on /puppet). My /etc/puppetlabs/puppet/puppet.conf looks like:
# This file can be used to override the default puppet settings.
# See the following links for more details on what settings are available:
# - https://puppet.com/docs/puppet/latest/config_important_settings.html
# - https://puppet.com/docs/puppet/latest/config_about_settings.html
# - https://puppet.com/docs/puppet/latest/config_file_main.html
# - https://puppet.com/docs/puppet/latest/configuration.html
[server]
vardir = /opt/puppetlabs/server/data/puppetserver
logdir = /var/log/puppetlabs/puppetserver
rundir = /var/run/puppetlabs/puppetserver
pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid
codedir = /etc/puppetlabs/code
[main]
environment = production
basemodulepath = /puppet/modules
[agent]
runinterval = 1m
Then to point the production environment to my nfs share I changed /etc/puppetlabs/code/environments/production/environment.conf to
modulepath = /puppet/production/modules/
manifest = /puppet/production/manifests/
I verified by running puppet config print manifest --section main --environment production which produces
/puppet/production/manifests
I do see calls coming in to the master by checking the logs in /var/log/puppetlabs/puppetserver/puppetserver.log
2024-04-11T14:02:31.192Z DEBUG [qtp1384966666-50] [p.r.core] Processing :post /puppet/v3/catalog/grid.lan
2024-04-11T14:02:31.195Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Evaluating match for Route /^\/puppet\//
2024-04-11T14:02:31.195Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Evaluating match for Route /v3/
2024-04-11T14:02:31.195Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Evaluating match for Route /^\/environments$/
2024-04-11T14:02:31.195Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Did not match path ("/catalog/grid.lan")
2024-04-11T14:02:31.195Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Evaluating match for Route /.*/
2024-04-11T14:02:31.196Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Evicting cache entry for environment :production
2024-04-11T14:02:31.196Z DEBUG [qtp1384966666-50] [p.s.j.puppet-environments] Removing environment 'production' from registry
2024-04-11T14:02:31.196Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Deleted text domain :production: false
2024-04-11T14:02:31.198Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Caching environment :production (ttl = 0 sec)
2024-04-11T14:02:31.199Z DEBUG [qtp1384966666-50] [p.s.j.puppet-environments] Registering environment 'production'
2024-04-11T14:02:31.205Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Found in cache :production (ttl = 0 sec)
2024-04-11T14:02:31.228Z INFO [qtp1384966666-50] [puppetserver] Puppet Compiled catalog for grid.lan in environment production in 0.02 seconds
2024-04-11T14:02:31.228Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Found in cache :production (ttl = 0 sec)
2024-04-11T14:02:31.230Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Puppet::Network::Format[msgpack]: feature msgpack is missing
2024-04-11T14:02:31.231Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Puppet::Network::Format[pson]: feature pson is missing
2024-04-11T14:02:31.231Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Evicting cache entry for environment :production
2024-04-11T14:02:31.231Z DEBUG [qtp1384966666-50] [p.s.j.puppet-environments] Removing environment 'production' from registry
2024-04-11T14:02:31.231Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Deleted text domain :production: false
2024-04-11T14:02:31.232Z DEBUG [qtp1384966666-50] [puppetserver] Puppet Puppet::Network::Format[rich_data_msgpack]: feature msgpack is missing
2024-04-11T14:02:31.232Z DEBUG [qtp1384966666-50] [puppetserver] Puppet catalog supports formats: rich_data_json json yaml dot
2024-04-11T14:02:31.289Z DEBUG [qtp1384966666-48] [p.r.core] Processing :put /puppet/v3/report/grid.lan
2024-04-11T14:02:31.291Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Evaluating match for Route /^\/puppet\//
2024-04-11T14:02:31.291Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Evaluating match for Route /v3/
2024-04-11T14:02:31.292Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Evaluating match for Route /^\/environments$/
2024-04-11T14:02:31.292Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Did not match path ("/report/grid.lan")
2024-04-11T14:02:31.292Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Evaluating match for Route /.*/
2024-04-11T14:02:31.295Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Caching environment :production (ttl = 0 sec)
2024-04-11T14:02:31.295Z DEBUG [qtp1384966666-48] [p.s.j.puppet-environments] Registering environment 'production'
2024-04-11T14:02:31.296Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Puppet::Network::Format[msgpack]: feature msgpack is missing
2024-04-11T14:02:31.296Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Puppet::Network::Format[pson]: feature pson is missing
2024-04-11T14:02:31.296Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Puppet::Network::Format[rich_data_msgpack]: feature msgpack is missing
2024-04-11T14:02:31.297Z DEBUG [qtp1384966666-48] [puppetserver] Puppet report supports formats: json yaml
2024-04-11T14:02:31.301Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Received report to process from grid.lan
2024-04-11T14:02:31.301Z DEBUG [qtp1384966666-48] [puppetserver] Puppet Processing report from grid.lan with processor Puppet::Reports::Store
Both master and agent are running latest version of Ubuntu 22.04 lts. The puppet server version is 8.5.0
On the agent I haven't been able to get anything working. I tried by passing the hostname of the master using --server and using --debug. All calls succeed but result in an empty catalog.
The interesting part is if I run puppet catalog find nexus.lan and puppet catalog find grid.lan on the agent they both produce:
Notice: Compiled catalog for nexus.lan in environment production in 0.01 seconds
{
"tags": [
"settings"
],
"name": "nexus.lan",
"version": 1712844335,
"code_id": null,
"catalog_uuid": "2a65b3c0-a8f9-4bf3-8f28-5d57d845e9fe",
"catalog_format": 2,
"environment": "production",
"resources": [
{
"type": "Stage",
"title": "main",
"tags": [
"stage"
],
"exported": false,
"kind": "compilable_type",
"parameters": {
"name": "main"
}
},
{
"type": "Class",
"title": "Settings",
"tags": [
"class",
"settings"
],
"exported": false,
"kind": "unknown"
},
{
"type": "Class",
"title": "main",
"tags": [
"class"
],
"exported": false,
"kind": "unknown",
"parameters": {
"name": "main"
}
}
],
"edges": [
{
"source": "Stage[main]",
"target": "Class[Settings]"
},
{
"source": "Stage[main]",
"target": "Class[main]"
}
],
"classes": [
"settings"
]
}
Again here both commands produce another result on the master.
Any idea what I'm doing wrong?
So I'll answer my own question. I incorrectly assumed that puppet server would also run as root like the agents which is not the case.
It runs as puppet (Uid/Gid 999) which didn't have read permissions on the nfs share. Changed the uid and gid of the puppet user to a user that does have permissions and now it works.