Puppet CA Host Certificate Expiry

So you read the title and thought “Hey, I just read that article!”, right? Nah, here’s another article about Puppet CA Server certificates. Similar, but different. I’m just being cheeky by submitting them a couple of hours apart.

CA certs… Host certs… What?!

So, when a normal node runs Puppet for the first time, it contacts its Puppet CA server for a certificate that proves it’s allowed to be a client of that master. A detailed step through of what happens is:

It generates a new SSL keypair ($::hostprivkey and $::hostpubkey)
Using its newly generated $::hostprivkey, it generates a certificate signing request ($::hostcsr) for a certificate and sends that off to whatever’s configured as its Puppet CA server ($::ca_server). No catalog is compiled at this point, because the node isn’t considered a trusted client of the Puppet master.
The CSR from the node ends up in the $::csrdir of the Puppet CA server. If the Puppet CA server is configured to allow auto-signing of the node that made the request, it signs it with its CA certificate ($::cacert) and pops the signed cert into $::signeddir. The CSR gets deleted.
On the node’s next Puppet run, it gets a copy of the signed certificate from the Puppet CA server and is allowed to continue with its run. It gets a catalog and nice things happen to it for the rest of its little life. Or for five years. Which is how long the signed certs are valid for, by default.

So we’ve described a situation where a node requests what I’ll call a host certificate from the CA, which gives one back signed by its CA cert.

Okay, okay. But what about the CA?

You might already know this, but the Puppet CA server is usually a client of itself.

So when it was first spun up, the Puppet CA generated itself an SSL keypair ($::cakey and $::capub) and a CA certificate ($::cacert). This certificate is self-signed, and is marked as a CA cert, meaning it’s allowed to sign other certificate requests.

That’s on the master side. But Puppet masters also run the Puppet agent.

When the Puppet CA ran puppet agent -t for the first time, it went through the steps we went discussed above. Generated itself a new keypair ($::hostprivkey and $::hostpubkey)… Looked to see who its CA server was… (itself!) and submitted a CSR to them ($::hostcsr) and waited until it got a response back (from itself!).

So yeah, the upshot is that the Puppet CA server has two sets of certs. Its CA certs, and its host certs. We can illustrate that clearly here:

# puppet config print hostcert hostprivkey hostpubkey cacert cakey capub | awk '{print $NF}' | xargs md5sum
63c251f890893690df65f79d36eb5d62  /var/lib/puppet/ssl/certs/puppetca.example.com.pem
e2c7e9bb4b5a68c812989c2da702b971  /var/lib/puppet/ssl/private_keys/puppetca.example.com.pem
3b8874ae570cecb3470384b1fc307503  /var/lib/puppet/ssl/public_keys/puppetca.example.com.pem
f4288c68efa8a4f9bd07d0bac62710a6  /var/lib/puppet/ssl/ca/ca_crt.pem
af2f94391cefa3e8d296d0648ebeb5d5  /var/lib/puppet/ssl/ca/ca_key.pem
7c17a8d5cc920b1d923f4b18deefea51  /var/lib/puppet/ssl/ca/ca_pub.pem

In the vast majority of cases, these certificates will have been generated at roughly the same time (usually the same day, almost certainly the same week). That means that they also expire at roughly the same time. But be warned, the content of the two certs is completely different, as the checksums above show!

So you might have went through the steps from my previous post, renewed your CA certs, managed to get everything working just nicely, perform your first Puppet run after the big scary change and be faced with this redness:

Warning: Certificate 'puppetca.example.com' will expire on 2017-11-22T22:08:37UTC

Believe it or not, this is referring to the host cert of your Puppet CA cert. Check out the expiry of it if you don’t believe me:

# openssl x509 -noout -text -in $(puppet config print hostcert) | grep -A2 Valid
        Validity
            Not Before: Nov 22 22:08:37 2017 GMT
            Not After : Nov 22 22:08:37 2022 GMT

Fine, sounds legit. What’s the impact?

Honestly? Not much. It means that once the “Not After” date has expired, the client cert used by your Puppet agent won’t be accepted by the CA any more. That means no catalogues. Boo.

On nodes that aren’t important, fixing this is a cinch. But not only is it a little bit embarrassing when your most important Puppet server can’t be an agent of itself, it comes with some additional configurations. You can’t just nuke $::ssldir and walk away like you usually would.

What’s the big deal, then?

Usually when you want to regenerate a Puppet client’s certs you’ll do this:

root@client# rm -rf $(puppet config print ssldir)

root@puppetca# puppet cert clean client

root@client# puppet agent -t
<keys regen>
<csr created>
<puppet ca signs>
<new cert recieved>
<puppet run continues>

This is a bad idea to run on your Puppet CA. Like, a really bad idea.

It’ll delete the entirety of your Puppet CA’s SSL directory. That means your CA certs, CA keys and all of your estate’s signed certificates gone, up in smoke. (Have I mentioned that it’s a good idea to back this up?)
It’s not so bad if you’ve already deleted all of your SSL data, but if you were to run puppet cert clean puppetca when you still had all of your certs, you’d revoke both your host certs and your CA certs. They’d get appended to your CA’s CRL and none of the nodes in your estate would be able to check in with your Puppet masters any more. This is because your Puppet CA would not believe its own certs are valid any more.

So yeah, don’t do any of that.

Instead, we can perform a bit of surgery on your Puppet CA’s SSL directory and only remove the keys and certs that we mean to. We can even make it so that we can revert things if we mess it up, somehow! Let’s go.

Alternative DNS Names

First up, let’s see what your CA thinks about its own certs:

root@puppetca# puppet cert list --all | grep $(uname -n)
+ "puppetca.example.com"                                         (SHA256) A8:7A:68:3E:E9:61:CB:7A:FF:E9:42:48:FC:57:04:F8:BF:2E:05:8C:4D:EA:AF:F7:E5:3C:D6:5B:EF:8C:E1:6C (alt names: "DNS:puppet", "DNS:puppet-1.example.com", "DNS:puppet.example.com")

The alt names section tells me that the cert that’s already issued has X.509 Subject Alternative Name extensions baked in. This is a list of alternative DNS names that are also valid for the cert that’s been signed. It’s used so that your clients could hit, for example, a friendly CNAME DNS record that references your Puppet CA instead of the CA’s FQDN itself.

We’re gonna make sure that these are configured on your Puppet agent, too. Just so that we’re replacing like-with-like.

root@puppetca# puppet config print dns_alt_names

root@puppetca#

If, like above, you don’t have any output, you’re missing a bit of config. dns_alt_names is be configured in /etc/puppet/puppet.conf under the [main] section. So add something appropriate like this:

dns_alt_names=puppet,puppet-1.example.com,puppet.example.com

Re-running puppet config print dns_alt_names should have more success now.

Back it up, back it up!

Make sure and CYA by backing up everything you touch. And maybe even some stuff you don’t. I went a bit overboard:

# create backup directories
mkdir -p $(puppet config print ssldir)/backup/{host,ca,signed}

# back up our ca certs and crls (local and ca) because we're paranoid
cp $(puppet config print cacert) $(puppet config print ssldir)/backup/ca/cacert.pem
cp $(puppet config print cacrl) $(puppet config print ssldir)/backup/ca/cacrl.pem
cp $(puppet config print localcacert) $(puppet config print ssldir)/backup/ca/localcacert.pem
cp $(puppet config print hostcrl) $(puppet config print ssldir)/backup/ca/hostcrl.pem

# back up our agent-side certs
cp $(puppet config print hostcert) $(puppet config print ssldir)/backup/host/hostcert.pem
cp $(puppet config print hostprivkey) $(puppet config print ssldir)/backup/host/hostprivkey.pem
cp $(puppet config print hostpubkey) $(puppet config print ssldir)/backup/host/hostpubkey.pem

# remove up our ca-side signed cert, backing it up ofc
cp $(puppet config print signeddir)/$(hostname -f).pem $(puppet config print ssldir)/backup/signed/signed.pem

# make sure they're there
find $(puppet config print ssldir)/backup | xargs ls -ld

Now you should have a pretty list of files backed up in `$(puppet config print ssldir)/backup, should the worst happen.

Precision cut

Now for the action. For the avoidance of doubt, all of this is performed on your Puppet CA server.

# 1. stop puppet agent
service puppet stop 

# 2. move the host certs that Puppet Agent knows about
mv $(puppet config print hostcert) /tmp/hostcert.pem
mv $(puppet config print hostprivkey) /tmp/hostprivkey.pem
mv $(puppet config print hostpubkey) /tmp/hostpubkey.pem

# 3. move the signed version of the ca's host cert
mv $(puppet config print signeddir)/$(hostname -f).pem /tmp

# 4. regenerate our host ssl keypair, create csr and submit to ca
puppet agent -t

# 5. get the ca to sign our csr
puppet cert sign $(hostname -f)

# 6. get the agent to retrieve the signed cert and perform a run
puppet agent -t

# 7. restart puppet agent
service puppet start

All going well, you should have new host certs on your Puppet CA server, with the actual CA certs themselves being left well alone!

Now, you can check the validity of your new host cert.

# openssl x509 -noout -text -in $(puppet config print hostcert) | grep -A2 Valid
        Validity
            Not Before: Oct 18 11:55:40 2017 GMT
            Not After : Oct 18 11:55:40 2022 GMT

Compare it to the validity dates of your CA cert if you want to. They’ll differ:

# openssl x509 -noout -text -in $(puppet config print cacert) | grep -A2 Valid
        Validity
            Not Before: Oct 18 10:22:30 2017 GMT
            Not After : Oct 18 10:22:30 2022 GMT

Now we can hopefully put this sorry affair behind us. And get on with some real work wrangling some clouds, eh?

See you in five years!