OpenBao Features - Paginated Lists

This is the start of a multi-part series on OpenBao's features.

Nearly every single networked interface returning a list of results supports subsets. SQL supports the LIMIT and OFFSET keywords, along with a rich language for filtering returned results. Google Cloud KMS APIs supports pageSize, yielding a nextPageToken, for iterating over multiple pages of results.

Many resources in Vault and OpenBao return lists: KVv2 secrets, PKI's certificate lists, SSH's roles, and more.

Paginated lists were shipped in OpenBao v2.0.0 as our very first feature in our very first release!

So, why doesn't Vault support paginated lists?

The answer lies in its historical great-common-denominator storage interface:

// Storage is the way that logical backends are able read/write data.
type Storage interface {
    List(context.Context, prefix string) (entries []string, err error)
    Get(context.Context, path string) (entry *StorageEntry, err error)
    Put(context.Context, entry *StorageEntry) error
    Delete(context.Context, path string) error
}

(from fork-point:sdk/logical/storage.go).

In supporting over twenty different storage backends, Vault had to support a common core of simple APIs over all potential storage backends. This meant keeping Storage.List(...) to a simple get-all-results list operation. If LIST APIs were to limit results, this would mean they'd still have to fetch them all from storage and only limit them in memory before sending to the client.

Early in OpenBao's history, the decision was made to remove all but the Raft storage backend. This made it possible to iterate on our storage model. Concretely, paginated lists and transactional storage (to be discussed another time!) came out of that. As we re-introduced storage backends like PostgreSQL, we made sure to include the improvements we've made to the storage API from the start.

OpenBao's storage interface now looks like:

// Storage is the way that logical backends are able read/write data.
type Storage interface {
	List(context.Context, prefix string) (entries []string, err error)
	ListPage(context.Context, prefix string, after string, limit int) (entries []string, err error)
	Get(context.Context, path string) (entry *StorageEntry, err error)
	Put(context.Context, entry *StorageEntry) error
	Delete(context.Context, path string) error
}

(from v2.5.5:sdk/logical/storage.go)

meaning that API handlers can now fetch only a subset of results.

This follows a SQL-like interface:

func (b *RaftBackend) ListPage(ctx context.Context, prefix string, after string, limit int) ([]string, error) {
	// ... implementation elided ...
}

(from v2.5.5:physical/raft/raft.go)

taking the new parameters after and limit. These are:

• after: an optional entry to begin listing after for pagination; not required to exist in the list results.
• limit: an optional number of entries to return; defaults to all entries when set to a non-positive number.

We suggest plugin authors pass these through on all API endpoints to the underlying storage calls.

For plugin authors who build against OpenBao's SDK but wish to have compatibility with HashiCorp Vault, we've stubbed the implementation, meaning you can safely expose and use a paginated list API and support it on both server implementations.

In addition, use of paginated lists has lead to improvements like #678 by Fatima Patel, to use pagination during PKI's tidy operations. In the past, tidy operations could consume a lot of memory if a large number of PKI mounts contained a large number of stored leaf certificates. We exposed a new page_size option (defaulting to 1000) to limit the number of certificate serial numbers in memory during a single PKI tidy operation.

By enforcing this with pagination_limit in an ACL policy, operators can now ensure that clients use pagination and set maximum result set sizes going forward.

In short, OpenBao aligns with long-standing industry expectations for expensive list calls and operators have more control with OpenBao than with Vault for managing the performance impact of these types of API requests.

Tune in next time for a discussion on transactional storage!