Puppet client caching but not updating local facts

Question

I have a server pmaster-dev that is a puppet client (its master is pmaster). The server pmaster-dev itself acts as a puppet master for several clients.

When pmaster-dev checks in with pmaster it is caching its facts to a local sqlite3 database file /var/lib/puppet/state/clientconfigs.sqlite. On every subsequent check-in pmaster-dev never updates this local cache. Thus, its facts are always stale. Other clients of pmaster (including pmaster itself) never cache.

How do we either tell it to update the cache or disable caching of the facts? Why is it caching while other clients of pmaster are not?

We are running 2.7.18 on a Debian squeeze system.

Here is the /etc/puppet/puppet.conf file for pmaster-dev:

[agent]
server = pmaster.example.org
environment = master
configtimeout = 300

logdir = /var/log/puppet
vardir = /var/lib/puppet
ssldir = /etc/puppet/ssl
rundir = /var/run/puppet

ca_server = puppetca.example.org
ca_port = 8141

graph = true
report = true
pluginsync = true

classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
diff_args = '-u'
show_diff = true


[master]
certname = pmaster-dev.example.org
syslogfacility = local2

logdir = /var/log/puppet
vardir = /var/lib/puppet
rundir = /var/run/puppet
ssldir = /srv/puppetmaster/ssl

reports = tagmail,lastcheck,logcache
manifest = /srv/puppet/$environment/manifests/site.pp
graph = true
modulepath = /srv/puppet/$environment/modules:/srv/puppet/$environment/services:/srv/puppet/$environment/clients
cacrl = /srv/puppetmaster/ssl/crl.pem
ca = false

manifestdir = /srv/puppet/$environment/manifests
queue_source = stomp://pupqueue-dev.example.org:61613/
async_storeconfigs = true
storeconfigs = false
dbadapter = mysql
dbuser = puppet
dbpassword = secret
dbserver = pupqueue-dev.example.org
dbname = puppet

ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

Answer 1

After some digging in the Puppet Ruby source files, I tracked the issue to a bug where there is conflation of Puppet configuration file sections. It all boils down to the word "master"

Summary: Puppet masters when acting as a Puppet client and in the "master" environment cause problems with the configuration file parsing.

Details: When the puppet agent starts one of the things it does is parse the configuration file (normally /etc/puppet/puppet.conf).

The configuration file is broken up into sections, e.g., "[main]", "[agent]", "[master]", etc. Each of these sections is stored separately in the hash generated from parsing the configuration file. In my case, pmaster-dev has these sections: "memory", "master", "cli", and "agent".

There can also be per-environment sections. For example, if there is an environment called "stable201301" then there can be a section in the configuration file headed "[stable201301]".

For each of the above sections (called "sources" in the code) you examine all the settings that have associated "hooks". If in that section the setting with a hook is defined, you call that setting's hook. Some of the settings that have hooks include catalog_format, node_name_fact, and storeconfigs.

One of the more interesting settings is "async_storeconfigs" which has a hook that sets up a cache class.

Recall that the sections can also include ones for each environment. THE PROBLEM: What if the Puppet master (when acting as a Puppet client) is in the "master" environment?

The usual role for the "[master]" section is for settings for a Puppet master. But if the Puppet master itself is using the "master" environment we have a conflict. In particular, the "async_storeconfigs" setting from the "[master]" gets loaded from the configuration file. Since the Puppet master (as a Puppet client) is in the "master" environment this setting causes the cache class to be set and so when puppet agent --test is run the puppet client looks for cached facts.

Solution(?): You could make sure that a Puppet master never used the "master" environment. A better solution would be to have the client configuration parser not look at the "[master]" section (since that is only applicable to Puppet masters). An even betterer solution would be to separate client and master configuration files altogether.

Answer 2

It seems like you are confused about what clientconfigs.sqlite actually is. This file is part of the Stored Configuration mechanism of Puppet. It is a file maintained by the Puppet Master, not by clients.

Its creation is controlled by the storeconfigs, async_storeconfigs and db* parameters in the [master] section of puppet.conf. The default (when no db* parameters are set) is to use the SQLite adapter.

From the configuration you pasted, it seems like you have configured the master to store the configuration inside a MySQL database via a Stomp broker. So it seems like not updating the SQLite database is by design?

Why this database file has been created to begin with is a good question. My hypothesis would be that Puppet Master started with an incomplete configuration during the initial set up of the server.