pki health-check

The pki health-check command verifies the health of the given PKI secrets engine mount against an optional configuration.

This runs with the permissions of the given token, reading various APIs from the mount and /sys against the given OpenBao server

Mounts need to be specified with any namespaces prefixed in the path, e.g., ns1/pki.

Examples

Performs a basic health check against the pki-root mount:

$ bao pki health-check pki-root/

Configuration can be specified using the -health-config flag:

$ bao pki health-check -health-config=mycorp-root.json pki-root/

Using the -list flag will show the list of health checks and any known configuration values (with their defaults) that will be run against this mount:

$ bao pki health-check -list pki-root/

Usage

The following flags are unique to this command:

• -default-disabled - When specified, results in all health checks being disabled by default unless enabled by the configuration file explicitly. The default is false, meaning all default-enabled health checks will run.

• -health-config (string: "") - Path to JSON configuration file to modify health check execution and parameters.

• -list - When specified, no health checks are run, but all known health checks are printed. Still requires a positional mount argument. The default is false, meaning no listing is printed and health checks will execute.

• -return-indicator (string: "default") - Behavior of the return value (exit code) of this command:

  • permission, for exiting with a non-zero code when the tool lacks permissions or has a version mismatch with the server;
  • critical, for exiting with a non-zero code when a check returns a critical status in addition to the above;
  • warning, for exiting with a non-zero status when a check returns a warning status in addition to the above;
  • informational, for exiting with a non-zero status when a check returns an informational status in addition to the above;
  • default, for the default behavior based on severity of message and only returning a zero exit status when all checks have passed and no execution errors have occurred.

This command respects the -format parameter to control the presentation of output sent to stdout. Fatal errors that prevent health checks from executing may not follow this formatting.

Return status and output

This command returns the following exit codes:

• 0 - Everything is good.
• 1 - Usage error (check CLI parameters).
• 2 - Informational message from a health check.
• 3 - Warning message from a health check.
• 4 - Critical message from a health check.
• 5 - A version mismatch between health check and OpenBao Server occurred, preventing one or more health checks from being fully run.
• 6 - A permission denied message was returned from OpenBao Server for one or more health checks.

Note that an exit code of 5 (due to a version mismatch) is not necessarily fatal to the health check.

Each health check outputs one or results in a list. This list contains a mapping of keys (status, status_code, endpoint, and message) to values returned by the health check. An endpoint may occur in more than one health check and is not necessarily guaranteed to exist on the server (e.g., using wildcards to indicate all matching paths have the same result). Tabular form elides the status code, as this is meant to be consumed programatically.

These correspond to the following health check status values:

• status not_applicable / status code 0: exit code 0.
• status ok / status code 1: exit code 0
• status informational / status code 2: exit code 2.
• status warning / status code 3: exit code 3.
• status critical / status code 4: exit code 4.
• status invalid_version / status code 5: exit code 5.
• status insufficient_permissions / status code 6: exit code 6.

Health checks

The following health checks are currently implemented. More health checks may be added in future releases and may default to being enabled.

CA validity period

Name: ca_validity_period

Accessed APIs:

• LIST /issuers (unauthenticated)
• READ /issuer/:issuer_ref/json (unauthenticated)

Config Parameters:

• root_expiry_critical (duration: 182d) - for a duration within which the root's lifetime is considered critical
• intermediate_expiry_critical (duration: 30d) - for a duration within which the intermediate's lifetime is considered critical
• root_expiry_warning (duration: 365d) - for a duration within which the root's lifetime is considered warning
• intermediate_expiry_warning (duration: 60d) - for a duration within which the intermediate's lifetime is considered warning
• root_expiry_informational (duration: 730d) - for a duration within which the root's lifetime is considered informational
• intermediate_expiry_informational (duration: 180d) - for a duration within which the intermediate's lifetime is considered informational

This health check will check each issuer in the mount for validity status, returning a list. If a CA expires within the next 30 days, the result will be critical. If a root CA expires within the next 12 months or an intermediate CA within the next 2 months, the result will be a warning. If a root CA expires within 24 months or an intermediate CA within 6 months, the result will be informational.

Remediation steps:

1. Perform a CA rotation operation to check for CAs that are about to expire.
2. Migrate from expiring CAs to new CAs.
3. Delete any expired CAs with one of the following options:

• Run tidy manually with bao write <mount>/tidy tidy_expired_issuers=true.
• Use the OpenBao API to call delete issuer.

CRL validity period

Name: crl_validity_period

Accessed APIs:

• LIST /issuers (unauthenticated)
• READ /config/crl (optional)
• READ /issuer/:issuer_ref/crl (unauthenticated)
• READ /issuer/:issuer_ref/crl/delta (unauthenticated)

Config Parameters:

• crl_expiry_pct_critical (int: 95) - the percentage of validity period after which a CRL should be considered critically close to expiry
• delta_crl_expiry_pct_critical (int: 95) - the percentage of validity period after which a Delta CRL should be considered critically close to expiry

This health check checks each issuer's CRL for validity status, returning a list. Unlike CAs, where a date-based duration makes sense due to effort required to successfully rotate, rotating CRLs are much easier, so a percentage based approach makes sense. If the chosen percentage exceeds that of the grace_period from the CRL configuration, an informational message will be issued rather than OK.

For informational purposes, it reads the CRL config and suggests enabling auto-rebuild CRLs if not enabled.

Remediation steps:

Use bao write to enable CRL auto-rebuild:

$ bao write <mount>/config/crl auto_rebuild=true

Root certificate issued Non-CA leaves

Name: root_issued_leaves

APIs:

• LIST /issuers (unauthenticated)
• READ /issuer/:issuer_ref/pem (unauthenticated)
• LIST /certs
• READ /certs/:serial (unauthenticated)

Config Parameters:

• certs_to_fetch (int: 100) - a quantity of leaf certificates to fetch to see if any leaves have been issued by a root directly.

This health check verifies whether a proper CA hierarchy is in use. We do this by fetching certs_to_fetch leaf certificates (configurable) and seeing if they are a non-issuer leaf and if they were signed by a root issuer in this mount. If one is found, we'll issue a warning about this, and recommend setting up an intermediate CA.

Remediation steps:

1. Restrict the use of sign, sign-verbatim, issue, and ACME APIs against the root issuer.
2. Create an intermediary issuer in a different mount.
3. Have the root issuer sign the new intermediary issuer.
4. Issue new leaf certificates using the intermediary issuer.

Role allows implicit localhost issuance

Name: role_allows_localhost

APIs:

• LIST /roles
• READ /roles/:name

Config Parameters: (none)

Checks whether any roles exist that allow implicit localhost based issuance (allow_localhost=true) with a non-empty allowed_domains value.

Remediation steps:

1. Set allow_localhost to false for all roles.
2. Update the allowed_domains field with an explicit list of allowed localhost-like domains.

Role allows Glob-Based wildcard issuance

Name: role_allows_glob_wildcards

APIs:

• LIST /roles
• READ /roles/:name

Config Parameters:

• allowed_roles (list: nil) - an allow-list of roles to ignore.

Check each role to see whether or not it allows wildcard issuance and glob domains. Wildcards and globs can interact and result in nested wildcards among other (potentially dangerous) quirks.

Remediation steps:

1. Split any role that need both of allow_glob_domains and allow_wildcard_certificates to be true into two roles.
2. Continue splitting roles until both of the following are true for all roles:
   • The role has allow_glob_domains or allow_wildcard_certificates, but not both.
   • Roles with allow_glob_domains and allow_wildcard_certificates are the only roles required for all SANs on the certificate.
3. Add the roles that allow glob domains and wildcards to allowed_roles so OpenBao ignores them in future checks.

Role sets no_store=false and performance

Name: role_no_store_false

APIs:

• LIST /roles
• READ /roles/:name
• LIST /certs
• READ /config/crl

Config Parameters:

• allowed_roles (list: nil) - an allow-list of roles to ignore.

Checks each role to see whether no_store is set to false.

warning

OpenBao will provide warnings and performance will suffer if you have a large number of certificates without temporal CRL auto-rebuilding and set no_store to true.

Remediation steps:

1. Update none-ACME roles with no_store=false. NOTE: Roles used for ACME issuance must have no_store set to true.
2. Set your certificate lifetimes as short as possible.
3. Use BYOC revocations to revoke certificates as needed.

Accessibility of audit information

Name: audit_visibility

APIs:

• READ /sys/mounts/:mount/tune

Config Parameters:

• ignored_parameters (list: nil) - a list of parameters to ignore their HMAC status.

This health check checks whether audit information is accessible to log consumers, validating whether our list of safe and unsafe audit parameters are generally followed. These are informational responses, if any are present.

Remediation steps:

Use bao secrets tune to set the desired audit parameters:

$ bao secrets tune \
  -audit-non-hmac-response-keys=certificate \
  -audit-non-hmac-response-keys=issuing_ca \
  -audit-non-hmac-response-keys=serial_number \
  -audit-non-hmac-response-keys=error \
  -audit-non-hmac-response-keys=ca_chain \
  -audit-non-hmac-request-keys=certificate \
  -audit-non-hmac-request-keys=issuer_ref \
  -audit-non-hmac-request-keys=common_name \
  -audit-non-hmac-request-keys=alt_names \
  -audit-non-hmac-request-keys=other_sans \
  -audit-non-hmac-request-keys=ip_sans \
  -audit-non-hmac-request-keys=uri_sans \
  -audit-non-hmac-request-keys=ttl \
  -audit-non-hmac-request-keys=not_after \
  -audit-non-hmac-request-keys=serial_number \
  -audit-non-hmac-request-keys=key_type \
  -audit-non-hmac-request-keys=private_key_format \
  -audit-non-hmac-request-keys=ou \
  -audit-non-hmac-request-keys=organization \
  -audit-non-hmac-request-keys=country \
  -audit-non-hmac-request-keys=locality \
  -audit-non-hmac-request-keys=province \
  -audit-non-hmac-request-keys=street_address \
  -audit-non-hmac-request-keys=postal_code \
  -audit-non-hmac-request-keys=permitted_dns_domains \
  -audit-non-hmac-request-keys=policy_identifiers \
  -audit-non-hmac-request-keys=ext_key_usage_oids \
  -audit-non-hmac-request-keys=csr \
   <mount>

ACL policies allow problematic endpoints

Name: policy_allow_endpoints

APIs:

• LIST /sys/policy
• READ /sys/policy/:name

Config Parameters:

• allowed_policies (list: nil) - a list of policies to allow-list for access to insecure APIs.

This health check checks whether unsafe access to APIs (such as sign-intermediate, sign-verbatim, and sign-self-issued) are allowed. Any findings are a critical result and should be rectified by the administrator or explicitly allowed.

Allow If-Modified-Since requests

Name: allow_if_modified_since

APIs:

• READ /sys/internal/ui/mounts

Config Parameters: (none)

This health check verifies if the If-Modified-Since header has been added to passthrough_request_headers and if Last-Modified header has been added to allowed_response_headers. This is an informational message if both haven't been configured, or a warning if only one has been configured.

Remediation steps:

1. Update allowed_response_headers and passthrough_request_headers for all policies with bao secrets tune:

$ bao secrets tune \
    -passthrough-request-headers="If-Modified-Since" \
    -allowed-response-headers="Last-Modified" \
    <mount>

1. Update ACME-specific headers with bao secrets tune (if you are using ACME):

$ bao secrets tune \
    -passthrough-request-headers="If-Modified-Since" \
    -allowed-response-headers="Last-Modified" \
    -allowed-response-headers="Replay-Nonce" \
    -allowed-response-headers="Link" \
    -allowed-response-headers="Location" \
    <mount>

Auto-Tidy disabled

Name: enable_auto_tidy

APIs:

• READ /config/auto-tidy

Config Parameters:

• interval_duration_critical (duration: 7d) - the maximum allowed interval_duration to hit critical threshold.
• interval_duration_warning (duration: 2d) - the maximum allowed interval_duration to hit a warning threshold.
• pause_duration_critical (duration: 1s) - the maximum allowed pause_duration to hit a critical threshold.
• pause_duration_warning (duration: 200ms) - the maximum allowed pause_duration to hit a warning threshold.

This health check verifies that auto-tidy is enabled, with sane defaults for interval_duration and pause_duration. Any disabled findings will be informational, as this is a best-practice but not strictly required, but other findings w.r.t. interval_duration or pause_duration will be critical/warnings.

Remediation steps

Use bao write to enable auto-tidy with the recommended defaults:

$ bao write <mount>/config/auto-tidy \
    enabled=true \
    tidy_cert_store=true \
    tidy_revoked_certs=true \
    tidy_acme=true \
    tidy_revocation_queue=true \
    tidy_cross_cluster_revoked_certs=true \
    tidy_revoked_cert_issuer_associations=true

Tidy hasn't run

Name: tidy_last_run

APIs:

• READ /tidy-status

Config Parameters:

• last_run_critical (duration: 7d) - the critical delay threshold between when tidy should have last run.
• last_run_warning (duration: 2d) - the warning delay threshold between when tidy should have last run.

This health check verifies that tidy has run within the last run window. This can be critical/warning alerts as this can start to seriously impact OpenBao's performance.

Remediation steps:

1. Schedule a manual run of tidy with bao write:

$ bao write <mount>/tidy \
    tidy_cert_store=true \
    tidy_revoked_certs=true \
    tidy_acme=true \
    tidy_revocation_queue=true \
    tidy_cross_cluster_revoked_certs=true \
    tidy_revoked_cert_issuer_associations=true

1. Review the tidy status endpoint, bao read <mount>/tidy-status for additional information.
2. Re-configure auto-tidy based on the log information and results of your manual run.

Too many certificates

Name: too_many_certs

APIs:

• READ /tidy-status
• LIST /certs

Config Parameters:

• count_critical (int: 250000) - the critical threshold at which there are too many certs.
• count_warning (int: 50000) - the warning threshold at which there are too many certs.

This health check verifies that this cluster has a reasonable number of certificates. Ideally this would be fetched from tidy's status or a new metric reporting format, but as a fallback when tidy hasn't run, a list operation will be performed instead.

Remediation steps:

1. Verify that tidy ran recently with bao read:

   $ bao read <mount>/tidy-status
2. Schedule a manual run of tidy with bao write:

   $ bao write <mount>/tidy \
     tidy_cert_store=true \
     tidy_revoked_certs=true \
     tidy_acme=true \
     tidy_revocation_queue=true \
     tidy_cross_cluster_revoked_certs=true \
     tidy_revoked_cert_issuer_associations=true
3. Enable auto-tidy.
4. Make sure that you are not renewing certificates too soon. Certificate lifetimes should reflect the expected usage of the certificate. If the TTL is set appropriately, most certificates renew at approximately 2/3 of their lifespan.
5. Consider setting the no_store field for all roles to true and use BYOC revocations to avoid storage.

Enable ACME issuance

Name: enable_acme_issuance

APIs:

• READ /config/acme
• READ /config/cluster
• LIST /issuers (unauthenticated)
• READ /issuer/:issuer_ref/json (unauthenticated)

Config Parameters: (none)

This health check verifies that ACME is enabled within a mount that contains an intermediary issuer, as this is considered a best-practice to support a self-rotating PKI infrastructure.

Review the ACME Certificate Issuance API documentation to learn about enabling ACME support in OpenBao.

ACME response headers

Name: allow_acme_headers

APIs:

• READ /sys/internal/ui/mounts

Config Parameters: (none)

This health check verifies if the "Replay-Nonce, Link, and Location headers have been added to allowed_response_headers, when the ACME feature is enabled. The ACME protocol will not work if these headers are not added to the mount.

Remediation steps:

Use bao secrets tune to add the missing headers to allowed_response_headers:

$ bao secrets tune \
  -allowed-response-headers="Last-Modified" \
  -allowed-response-headers="Replay-Nonce" \
  -allowed-response-headers="Link" \
  -allowed-response-headers="Location" \
  <mount>