OpenBao Features - Recursive Lists (SCAN) & Filtering

This is the fifth part of a multi-part series on OpenBao's features.

Last time we talked about declarative plugin configuration and how it made deploying and adopting plugins much easier. With OCI-based distribution operators can deploy plugins with just a few configuration snippets, mirroring OpenTofu's approach.

We hinted at addressing two of the most-requested features in HashiCorp Vault: recursive list support and filtering of list responses.

As mentioned there by Vault community members, we've supported recursive lists since OpenBao v2.2.0 and filtered lists since OpenBao v2.4.0. And, for any plugin developers out there, we support it in our external plugin SDK including storage helpers which should work on Vault as well.

Question

What other places need recursive list support?

Reach out to us if we've missed one!

Overview

OpenBao has always supported a custom HTTP verb, LIST, for listing entries under a path. This is also supported via the ?list=true query parameter on a regular GET-verb request, for use when clients cannot support custom verbs.

In designing recursive listing, we realized many endpoints (like KVv2's list entries) made sense as both LIST and SCAN operations. Rather than forcing authors to implement a new endpoint design--for example, moving from a layout like LIST /secrets/metadata/:path to LIST /secrets/metadata-recursive/:path--we opted to introduce a new verb, SCAN for this. This allowed us to extend ACL policies to allow policy authors control over recursive lists without having to investigate a plugin's layout.

For instance, the policy:

path "secrets/*" {
    capabilities = ["read", "create", "update", "list", "patch", "delete"]
}

gives users the ability to perform mostly cheap operations, while restricting their ability to do recursive lists (scan operations). However, if they also add the scan operation:

path "secrets/*" {
    capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
}

this would be more expensive and maybe should only be allowed on specific subdirectories within a KVv2 layout:

path "secrets/metadata/my-app/*" {
    capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
}

Usage

As we discussed in past blogs, OpenBao has support for both pagination and transactional storage. When coupled with SCAN support, this gives users a powerful consistency tool to inspect a large number of secrets at once. For instance, the API call:

SCAN secrets/detailed-metadata/my-app

would return a list response with metadata about each secret as well. While this would usually be a 1+n operation in Vault (list all secret and then fetch its metadata), OpenBao will return them in a single operation, with a transaction to ensure internal consistency of the results.

The same holds true for namespaces: bao namespace list will show all top-level children of the current namespace, but bao namespace scan will recurse and show children-of-children and the full hierarchy.

OpenBao also supports the scan operation via the bao scan command or in the Go API via client.Logical().Scan(...) and related operations.

Security

This ties into another commonly requested feature in HashiCorp Vault: restricting list (and now scans!) to entries which the user can view.

In OpenBao, we implemented a policy keyword, list_scan_response_keys_filter_path, which takes a text/template expression for limiting visible results. Visible results are entries which (when templating is applied according to the filter path) have list access for entries ending in a / or read access otherwise. This means the policy author must know the corresponding type of the plugin and where to map list entries to. For example in KVv2, one could either map entries in a list or scan to the data (<mount>/data/<entry>) or metadata (<mount>/metadata/<entry>) paths.

Consider an ACL policy like:

# Allow listing secrets broadly but limit to visible results (read or list):
path "secrets/metadata/*" {
    capabilities = ["list", "scan"]

    # See also: https://openbao.org/docs/concepts/policies/#filtering-list-or-scan-results
    list_scan_response_keys_filter_path = "{{ .path }}{{ .key }}"
}

# Allow reading secrets and metadata in shared:
path "secrets/data/shared/*" {
    capabilities = ["read"]
}

path "secrets/metadata/shared/*" {
    capabilities = ["read"]
}

# But allow full access to a personal space.
path "secrets/data/personal/*" {
    capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
}

path "secrets/metadata/personal/*" {
    capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
}

In this example, a call to the scan endpoint would show entries under shared/ and personal/ but hide entries under private/ due to the filtering on the list result.

Use of text/template allows for advanced functionality like changing the path as well; for instance, if metadata wasn't widely used but direct secret access should be the control, list_scan_response_keys_filter_path could be written as:

{{ .path | replace "secrets/metadata/" "secrets/data/" }}{{ .key }}

because the list and scan endpoints are under /metadata/ and not /data/.

Looking ahead

Going forward, we'll likely consider supporting Scan(...) and ScanWithData(...) as part of our formal storage interface, allowing easier implementation for plugins.

Tune in next time as we explore ACME-enabled TLS listeners!