PKI secrets engine - rotation primitives

OpenBao's PKI Secrets Engine supports multiple issuers in a single mount point. By using the certificate types below, rotation can be accomplished in various situations involving both root and intermediate CAs managed by OpenBao.

X.509 certificate fields

X.509 is a complex specification; modern implementations tend to refer to RFC 5280 for specific details. For validation of certificates, both RFC 5280 and the TLS validation RFC 6125 are important for understanding how to achieve rotation.

The following is a simplification of these standards for the purpose of this document.

Every X.509 certificate begins with an asymmetric key pair, using an algorithm like RSA or ECDSA. This key pair is used to create a Certificate Signing Request (CSR), which contains a set of fields the requester would like in the final certificate (but, it is up to the Certificate Authority (CA) to decide what fields to take from the CSR and which to override). The CSR also contains the public key of the pair, which is signed by the private key of the key pair to prove possession. Usually, the requester would ask for attributes in the Subject field of the CSR or in the Subject Alternative Name extension CSR to be respected in the final certificate. It is up to the CA if these values are trusted or not. When approved by the issuing authority (which may be backed by this asymmetric key itself in the case of a root self-signed certificate), the authority attaches the Subject of its certificate to the issued certificate in the Issuer field, assigns a unique serial number to the issued certificate, and signs the set of fields with its private key, thus creating the certificate.

There are some important restrictions here:

One certificate can only have one Issuer, but this issuer is identified by the Subject on the issuing certificate and its public key.
One key pair can be used for multiple certificates, but one certificate can only have one backing key material.

The following fields on the final certificate are relevant to rotation:

The backing public and private key material (Subject Public Key Info).
- Note that the private key is not included in the certificate but is uniquely determined by the public key material.
The Subject of the certificate.
- This identifies the entity to which the certificate was issued. While the SAN values (in the Subject Alternative Name extension) is useful when validating TLS Server certificates against the negotiated hostname and URI, it isn't generally relevant for the purposes of validating intermediate certificate chains or in rotation.
The Validity period of this certificate.
- Notably, RFC 5280 does not place any requirements around the issued certificate's validity period relative to the validity period of the issuing certificate. However, it does state that certificates ought to be revoked if their status cannot be maintained up to their notAfter date. This is why OpenBao's /pki/issuer/:issuer_ref configuration endpoint maintains the leaf_not_after_behavior per-issuer rather than per-role.
- Additionally, some browsers will place ultimate trust in the certificates in their trust stores, even when these certificates are expired.
  - Note that this only applies to certificates in the trust store; validity periods will still be enforced for certificates not in the store (such as intermediates).
The Issuer and signatureValue of this certificate.
- In the issued certificate's Issuer field, the issuing certificate places its own Subject value. This allows the issuer to be identified later (without having to try signature validation against every known local certificate), when validating the presented certificate and chain.
- The signature over the entire certificate (by the issuer's private key) is then placed in the signatureValue field.
The optional Authority Key Identifier field.
- This field can contain either (or both) of two values:
  - The hash of the issuer's public key. This extension is set and this value is filled in by OpenBao.
  - The Issuer's Subject and Serial Number. This value is not set by OpenBao.
- The latter is a dangerous restriction for the purposes of rotation: it prevents cross-signing and reissuance as the new issuing certificates (while having the same backing key material) will have different serial numbers. See the Limitations of Primitives section below for more information on this restriction.
The Serial Number of this certificate.
- This field is unique to a specific issuer; when a certificate is reissued by its parent authority, it will always have a different serial number field.
The CRL distribution point field.
- This is a field detailing where a CRL is expected to exist for this certificate and under which CRL issuers (defaulting to the issuing certificate itself) the CRL is expected to be signed by. This is mostly informational and for server software like nginx, OpenBao's Cert Auth method, and Apache, CRLs are provided to the server, rather than having the server fetch CRLs for certificates automatically.
- Note that root certificates (in browsers trust stores) are generally not considered revocable. However, if an intermediate is revoked by serial, it will appear on its parent's CRL, and may prevent rotation from happening.

X.509 rotation primitives

Rotation (from an organizational standpoint) can only safely happen with certain intermediate X.509 certificates being issued. To distinguish the two types of certificates used to achieve rotation, this document notates them as primitives.

Rotation of an end-entity certificate is trivial from an X.509 trust chain perspective; this process happens every day and should only depend on what is in the trust store and not the end-entity certificate itself. In OpenBao, the requester would hit the various issuance endpoints (/pki/issue/:name or /pki/sign/:name -- or use the unsafe /pki/sign-verbatim) and swap out the old certificate with the new certificate and reload the configuration or restart the service. Other parts of the organizations might use ACME for certificate issuance and rotation, especially if the service is public-facing (and thus needs to be issued by a Public CA). Given it was signed by a trusted root, any devices connecting to the service would not know the difference.

Rotation of intermediate certificates is almost as easy. Assuming a decent operational setup (wherein during end-entity issuance, the full certificate chain is updated in the service's configuration), this should be as easy as creating a new intermediate CA, signing it against the root CA, and then beginning issuance against the new intermediate certificate. In OpenBao, if the intermediate is generated in an existing mount path (or is moved into such), the requesting entity shouldn't care much. Under ACME, Let's Encrypt has successfully rotated intermediates to present a cross-signed chain (for older Android devices). Assuming the old intermediate's parent(s) are still valid and trusted, certificates issued under old intermediates should continue to validate.

The hard part of rotation--calling for the use of these primitives--is rotating root certificates. These live in every device's trust store and are hard to update from an organization-wide operational perspective. Unless the organization can swap out roots almost instantaneously and simultaneously (e.g., via an agent) with no missed devices, this process will likely span months.

To make this process lower risk, there are various primitive certificate types that use the above certificate fields. Key to their success is the following note:

warning

Note: While certificates are added to the trust store, it is ultimately the associated key material that determines trust: two issuer certificates with the same subject but different public keys cannot validate the same leaf certificate; only if the keys are the same can this occur.

Cross-Signed primitive

This is the most common type of rotation primitive. A common CSR is signed by two CAs, resulting in two certificates. These certificates must have the same Subject (but may have different Issuers and will have different Serial Numbers) and the same backing key material, to allow certificates they sign to be trusted by either variant.

Note that, due to restrictions in how end-entity certificates are used and validated (services and validation libraries expect only one), cross-signing most typically only applies to intermediate.

A note on Cross-Signed roots

Technically, cross-signing can occur between two roots, allowing trust bundles with either root to validate certs issued through the other. However, this process creates a certificate that is effectively an intermediate (as it is no longer self-signed) and usually must be served alongside the trust chain. Given this restriction, it's preferable to instead cross-sign the top-level intermediates under the root unless strictly necessary when the old root certificate has been used to directly issue leaf certificates.

So, the rest of this process flow assumes an intermediate is being cross-signed as this is more common.

Process flow

        -------------------
       | generate key pair | -------------> ...
        -------------------                 ...
           |            |                   ...
 --------------        --------------       ...
| generate CSR |      | generate CSR |      ...
 --------------        --------------       ...
         |                   |              ...
    -----------         -----------         ...
   | signed by |       | signed by |        ...
   | root A    |       | root B    |        ...
    -----------         -----------         ...

Here, a key pair was generated at some point in time. Two CSRs are created and sent to two different root authorities (Root A and Root B). These result in two separate certificates (potentially with different validity periods) with the same Subject and same backing key material.

Note that this cross-signing need not happen simultaneously; there could be a gap of several years between the first and second certificate. Additionally, there's no limit on the number of cross-signed "duplicate" (used loosely--with the same subject and key material) certificates: this could be cross-signed by many different root certificates if necessary and desired.

Certificate hierarchy

 --------                                            --------
| root A |                                          | root B |
 --------                                            --------
   |                                                      |
 ----------------                            ----------------
| intermediate C |  <- same key material -> | intermediate D |
 ----------------              |             ----------------
                               |
                      -------------------
                     | leaf certificates |
                      -------------------

The above process results in two trust paths: either of root A or root B (or both) could exist in the client's trust stores and the leaf certificate would validate correctly. Because the same key material is used for both intermediate certificates (C and D), the issued leaf certificate's signature field would be the same regardless of which intermediate was contacted.

Cross-signing is thus a unifying primitive; two separate trust paths now join into a single one, by having leaf certificate's issuer field to point to two separate paths (via duplication of the certificate in the chain) and would be conditionally validated based on which root is present in the trust store.

This construct is documented and used in several places:

Execution in OpenBao

To create a cross-signed certificate in OpenBao, use the /intermediate/cross-sign endpoint. Here, when creating a cross-signature to all cert B to be validated by cert A, provide the values (key_ref, all Subject parts, &c) for cert B during intermediate generation. Then sign this CSR (using the /issuer/:issuer_ref/sign-intermediate endpoint) with cert A's reference and provide necessary values from cert B (e.g., Subject parts). cert A may live outside OpenBao. Finally, import the cross-signed certificate into OpenBao using the /issuers/import/cert endpoint.

If this process succeeded, and both cert A and cert B and their key material lives in OpenBao, the newly imported cross-signed certificate will have a ca_chain response field during read containing cert A, and cert B's ca_chain will contain the cross-signed cert and its ca_chain value.

warning

Note: Regardless of issuer type, is important to provide all relevant parameters as they were originally; OpenBao does not infer e.g., the Subject name parameters from the existing issuer; it merely reuses the same key material.

Notes on `manual_chain`

If an intermediate is cross-signed and imported into the same mount as its pair, OpenBao will not detect the cross-signed pairs during automatic chain building. As a result, leaf issuance will have a chain that only includes one of these pairs of chains. This is because the leaf issuance's ca_chain parameter copies the value from signing issuer directly, rather than computing its own copy of the chain.

To fix this, update the manual_chain field on the issuers to include the chains of both pairs. For instance, given intA signed by rootA and intB signed by rootB as its cross-signed version, one could do the following:

$ bao patch pki/issuer/intA manual_chain=self,rootA,intB,rootB
$ bao patch pki/issuer/intB manual_chain=self,rootB,intA,rootA

This will ensure that issuance with either copy of the intermediate reports the full cross-signed chain when signing leaf certs.

Reissuance primitive

The second most common type of rotation primitive. In this scheme, the existing key material is used to generate a new certificate, usually at a much later point in time from the existing issuance.

While similar to the cross-signed primitive, this one differs in that usually the reissuance happens after the original certificate expires or is close to expiration and is reissued by the original root CA. In the event of a self-signed certificate (e.g., a root certificate), this parent certificate would be itself. In both cases, this changes the contents of the certificate (due to the new serial number) but allows all existing leaf signatures to still validate.

Unlike the cross-signed primitive, this primitive type can be used on all types of certificates (including leaves, intermediates, and roots).

Process flow

          -------------------
         | generate key pair | ---------------> ...
          -------------------                   ...
           |              |                     ...
 --------------           --------------        ...
| generate CSR |   <->   | generate CSR |       ...
 --------------           --------------        ...
         |                    |                 ...
 ------------------      ------------------     ...
| signed by issuer | -> | signed by issuer | -> ...
 ------------------      ------------------     ...

In this process flow, a single key pair is generated at some point in time and stored. The CSR (with same requested fields) is generated from this common key material and signed by the same issuer at multiple points in time, preserving all critical fields (Subject, Issuer, &c). While there is strictly no limit on the number of times a key can be reissued, at some point safety would dictate the key material should be rotated instead of being continually reissued.

Certificate hierarchy

                          ------
              -----------| root |-------------
             /            ------              \
             |                                |
 ---------------                           ---------------
| original cert | <- same key material -> | reissued cert |
 ---------------              |            ---------------
                              |
                      -------------------
                     | leaf certificates |
                      -------------------

Note that while this again results in two trust paths, depending on which intermediate certificate is presented and is still valid, only a root need be trusted. When a reissued certificate is a root certificate, the issuance link is simply self-loop. But, in this case, note that both certificates are (technically) valid issuers of each other. This means it should be possible to provide a reissued root certificate in the TLS certificate chain and have it chain back to an existing root certificate in a trust store.

This primitive type is thus an incrementing primitive; the life cycle of an existing key is extended into the future by issuing a new certificate with the same key material from the existing authority.

Execution in OpenBao

To create a reissued root certificate in OpenBao, use /issuers/generate/root/existing endpoint. This allows the generation of a new root certificate with the existing key material (via the key_ref request parameter). If this process succeeded, when reading the issuer (via GET /issuer/:issuer_ref), both issuers (old and reissued) will appear in each others' ca_chain response field (unless prevented so by a manual_chain value).

To create a reissued intermediate certificate in OpenBao, this is a three step process:

Use the /issuers/generate/intermediate/existing endpoint to generate a new CSR with the existing key material with the key_ref request parameter.
Sign this CSR via the same signing process under the same issuer. This step is specific to the parent CA, which may or may not be OpenBao.
Finally, use the /intermediate/set-signed endpoint to import the signed certificate from step 2.

If the process to reissue an intermediate certificate succeeded, when reading the issuer (via GET /issuer/:issuer_ref), both issuers (old and reissued) will have the same ca_chain response field, except for the first entry (unless prevented so by a manual_chain value).

warning

Temporal primitives

We can use the above primitive types to rotate roots and intermediates to new keys and extend their lifetimes. This time-based rotation is what ultimately allows us to rotate root certificates.

There's two main variants of this: a forward primitive, wherein an old certificate is used to bless new key material, and a backwards primitive, wherein a new certificate is used to bless old key material. Both of these primitives are independently used by Let's Encrypt in the aforementioned chain of trust document:

The link from DST Root CA X3 to ISRG Root X1 is an example of a forward primitive.
The link from ISRG Root X1 to R3 (which was originally signed by DST Root CA X3) is an example of a backwards primitive.

For most organizations with a hierarchical structured CA setup, cross-signing all intermediates with both the new and old root CAs is sufficient for root rotation.

However, for organizations which have directly issued leaf certificates from a root, the old root will need to be reissued under the new root (with shorter duration) to allow these certificates to continue to validate. This combines both of the above primitives (cross-signing and reissuance) into a single backwards primitive step. In the future, these organizations should probably move to a more standard, hierarchical setup.

Limitations of primitives

The certificate's Authority Key Identifier extension field may contain either or both of the issuer's keyIdentifier (a hash of the public key) or both the issuer's Subject and Serial Number fields. Generating certificates with the latter enabled (luckily not possible in OpenBao, especially so since OpenBao uses strictly random serial numbers) prevents building a proper cross-signed chain without re-issuing the same serial number, which will not work with most browsers' trust stores and validation engines, due to caching of certificates used in successful validations. In the strictest sense, when using a cross-signing primitive (from a different CA), the intermediate could be reissued with the same serial number, assuming no previous certificate was issued by that CA with that serial. This does not work when using a reissuance primitive as these are technically the same authority and thus this authority must issue certificates with unique serial numbers.

Suggested root rotation procedure

The following is a suggested process for achieving root rotation easily and without (outage) impact to the broader organization, assuming best practices are being followed. Some adaption will be necessary.

Note that this process takes time. How much time is dependent on the automation level and operational awareness of the organization.

Generate the new root certificate. For clarity, it is suggested to use a new common name to distinguish it from the old root certificate. Key material need not be the same.
Cross-sign all existing intermediates. It is important to update the manual chain on the issuers as discussed in that section, as we assume servers are configured to combine the certificate field with the ca_chain field on renewal and issuance, thus getting the cross-signed intermediates.
Encourage rotation to pickup the new cross-signed intermediates. With short-lived certificates, this should happen automatically. However, for some long-lived certs, it is suggested to rotate them manually and proactively. This step takes time, and depends on the types of certificates issued (e.g., server certs, code signing, or client auth).
Once all chains have been updated, new systems can be brought online with only the new root certificate, and connect to all existing systems.
Existing systems can now be migrated with a one-shot root switch: the new root can be added and the old root can be removed at the same time. Assuming the above step 3 can be achieved in a reasonable amount of time, this decreases the time it takes to move the majority of systems over to fully using the new root and no longer trusting the old root. This step also takes time, depending on how quickly the organization can migrate roots and ensure all such systems are migrated. If some systems are offline and only infrequently online (or, if they have hard-coded certificate stores and need to reach obsolescence first), the organization might not be ready to move on to future steps.
At this point, since all systems now use the new root, it is safe to remove or archive the old root and intermediates, updating the manual chain to point strictly to the new intermediate+root.

At this point, rotation is fully completed.

API

The PKI secrets engine has a full HTTP API. Please see the PKI secrets engine API for more details.

PKI secrets engine - rotation primitives

X.509 certificate fields​

X.509 rotation primitives​

Cross-Signed primitive​

A note on Cross-Signed roots​

Process flow​

Certificate hierarchy​

Execution in OpenBao​

Notes on manual_chain​

Reissuance primitive​

Process flow​

Certificate hierarchy​

Execution in OpenBao​

Temporal primitives​

Limitations of primitives​

Suggested root rotation procedure​

API​

X.509 certificate fields

X.509 rotation primitives

Cross-Signed primitive

A note on Cross-Signed roots

Process flow

Certificate hierarchy

Execution in OpenBao

Notes on `manual_chain`

Reissuance primitive

Process flow

Certificate hierarchy

Execution in OpenBao

Temporal primitives

Limitations of primitives

Suggested root rotation procedure

API