Hard to believe it, but it’s coming up on five years since our company stood up its first Puppet master. No cake here. Instead, all we get is a lousy old CA certificate expiry.
At first, I inhaled sharply and mentally prepared myself for the impending chaos of our client certificates becoming invalid.
Instead, I was pleasantly surprised when I stumbled across puppetlabs/certregen which promised to make the whole sorry affair disappear painlessly. It‘s even Puppet 3 compatible!
The secret-sauce is all around the use of a clever Puppet Face to provide the puppet certgen command on your CA server. This simplifies the regeneration of your master’s CA cert to one command, creating a replacement from the exact same key used to create your expiring cert. The new cert’s effectively a drop-in replacement!
The other piece of magic is a simple, super-portable class that manages the contents of $::localcacert on your client nodes to be the same as whatever’s in your master’s $settings::cacert file. It does this by calling the file()function on the Puppet master on catalog compilation, grabbing what the master believes its CA cert to be. So, we update the CA cert on the master and then the nodes take care of themselves!
Of course, because the replacement CA certificate is generated from the same keys as your expiring CA cert, this tool is not suitable for use when your CA’s keys have been compromised.
So how does it work in practice? The module’s README is pretty hot at explaining everything, but for the sake of completeness, here’s the process I used. It assumes that your CA cert hasn’t expired yet.
Install the module
We use Puppetfile, so I added the following to ours:
mod 'puppetlabs-certregen', '0.2.0'Get the expiring cert’s serial
Running puppet certregen ca won’t do anything off the bat — it’ll error and spit out something like so:
$ sudo puppet certregen ca
Error: The serial number of the CA certificate to rotate must be provided. If you are sure that you want to rotate the CA certificate, rerun this command with --ca_serial 01Actually regenerate the CA certificate
When you’re for-reals ready to regenerate your Puppet master’s CA cert, re-run the above command providing the serial number you gleaned previously:
$ sudo puppet certregen ca --ca_serial 01
Notice: Backing up current CA certificate to /var/lib/puppet/ssl/ca/ca_crt.1508247754.pem
Notice: Signed certificate request for ca
CA expiration is now 2022-10-16 13:42:34 UTC
CRL next update is now 2022-10-16 13:42:34 UTCAs you can see, this backs up the current (expiring) CA cert, just in case. Then it spits out the new expiration date of its replacement. Great!
To be clear — your CA server now has a new certificate, but all of the nodes in your Puppet deployment still have the old, expiring version. Let’s fix that!
Enforce node management of CA cert
Now we’ve gotta get some mechanism in place where your nodes can get the updated CA certificate. This is where the certregen::client class comes into play. It defines a File resource that manages each node’s CA certificate (whose path is determined by consulting the $::localcacert variable).
We use Roles & Profiles, so I added the following to our base profile:
include certregen::clientThe important thing is that all of your nodes get this into their catalogs somehow, so you can use the main manifest or whatever.
This change needs to make its way into all of the Puppet directory environments that you care about. In my case, I made the change to our production environment and retroactively applied it to a handful of static environments that we care about. I consider pretty much every other environment as disposable.
Update compile masters
Got any non-CA, compile-only Puppet masters? Get them to complete a Puppet run against your CA as a priority. This’ll update the CA used by these masters to the new one, along with the associated CRL.
drew@pmaster-1 $ sudo puppet agent -t --server puppetca.example.com
...
Notice: /Stage[main]/Certregen::Client/File[/var/lib/puppet/ssl/certs/ca.pem]/content: content changed '{md5}35d987adf79eaaed26780cd0e69f7d52' to '{md5}ed6cb1e53b7ad78ac41629fb8e3e2c38'
Notice: /Stage[main]/Certregen::Client/File[/var/lib/puppet/ssl/crl.pem]/content: content changed '{md5}910a663110f54bf974a26b750c3a8a90' to '{md5}a774a54175b68251b11d0b2e264eec37'
...
Notice: Finished catalog run in 6.42 secondsIf you don’t do this first, any nodes that compile catalogs using these Puppet masters will get the CA cert that the compile master believes to be current — in this case, the old, expiring one! And that’s no good!
Don’t have any compile masters? Then you’re good.
Update CA cert on client nodes
As the nodes across your deployment check in with your Puppet masters, you’ll notice their CA certificate and associated CRL update to the new one:
drew@host1 $ sudo grep 'content changed' /var/log/syslog | grep 'Certregen::Client'
Oct 17 04:06:59 host1 puppet-agent[6556]: (/Stage[main]/Certregen::Client/File[/var/lib/puppet/ssl/certs/ca.pem]/content) content changed '{md5}35d987adf79eaaed26780cd0e69f7d52' to '{md5}ed6cb1e53b7ad78ac41629fb8e3e2c38'
Oct 17 04:06:59 host1 puppet-agent[6556]: (/Stage[main]/Certregen::Client/File[/var/lib/puppet/ssl/crl.pem]/content) content changed '{md5}910a663110f54bf974a26b750c3a8a90' to '{md5}a774a54175b68251b11d0b2e264eec37'If in doubt, you can check that the CA certificate serials of your Puppet CA and your client nodess match!
root@puppetca # openssl x509 -in $(puppet config print cacert) -noout -text | grep Serial
Serial Number: 7 (0x7)
root@node1 # openssl x509 -in $(puppet config print localcacert) -noout -text | grep Serial
Serial Number: 7 (0x7)And whew, you’ve just avoided what could have been a major problem. Just like that. Thanks, Puppet!
Things to remember (or is it “things I found out”?)
After including the ::certregen::client class, the content of each node’s$::localcacert and $::hostcrl files will be managed by Puppet, and kept up to date with the most recent versions located on your Puppetmasters.
That means that if a certificate is revoked on your CA, an entry will be added to its CRL. And that new version of the CRL will subsequently be pushed out across all of the nodes with the ::certregen::client class included in their manifests.
More importantly, consider whether ::certregen::client is going to affect your Puppet CA’s own catalog. Consider where your Puppet CA gets its catalog from.
Is it itself? Then you’re good.
Is it a different master or a load balancer containing a number of compile masters? If so, things could get messy.
If your Puppet CA hits another master for its catalog, it may receive an older version of the CRL than it has itself. In reality, this just means that $::cacert and $::localcacert will differ on your Puppet CA (as will $::cacrl and $::hostcrl), which may add some confusion if you start inspecting certs.
For me, I had a rouge $::cacert living on one of my compile masters, which really added some spice into the mix. It all worked out in the end, though.
To avoid any of these potential issues, I’d make sure the value of server in /etc/puppet/puppet.conf for your Puppet CA is itself. Probably worth making sure your compile masters point at your Puppet CA for their catalogs, too.