OpenBao Features - Transactional Storage

This is second part of a multi-part series on OpenBao's features.

Today we focus on transactional storage. While the earlier blog posts focused on the what and how of transactions in Raft, this post will focus on the measurable impact of transactions in OpenBao and their lack in Vault. We will demo some possible ways of creating snapshots which cannot restore and are not consistent on Vault and show how we used transactions to achieve consistency on OpenBao.

Historical context

Recall from last week's post that our starting storage interface was rather limited:

// Storage is the way that logical backends are able read/write data.
type Storage interface {
    List(context.Context, prefix string) (entries []string, err error)
    Get(context.Context, path string) (entry *StorageEntry, err error)
    Put(context.Context, entry *StorageEntry) error
    Delete(context.Context, path string) error
}

(from fork-point:sdk/logical/storage.go).

Also included in the storage model of the fork, though implemented in only a few backends (Raft, CockroachDB, Consul, FoundationDB, and Spanner) was a basic batch application mechanism:

// TxnEntry is an operation that takes atomically as part of
// a transactional update. Only supported by Transactional backends.
type TxnEntry struct {
	Operation Operation
	Entry     *Entry
}

...

// Transactional is an optional interface for backends that
// support doing transactional updates of multiple keys. This is
// required for some features such as replication.
type Transactional interface {
	// The function to run a transaction
	Transaction(context.Context, []*TxnEntry) error
}

(from fork-point:sdk/physical/transactions.go).

Notably, from the implementation of physical.GenericTransactionHandler, we see that this was not an implementation of check-and-set semantics: LIST operations are entirely ignored, any GET operations are dispatched ahead of any writes, and while a rollback log is created and entries read before issuing any writes, they are not compared against any sent writes. This makes batch application rather unsafe if used incorrectly: if a (distributed) lock mechanism or other exclusive ownership semantic does not exist, multiple in-flight transactions can write to the same storage entries. This may produce unexpected results.

Luckily, this mechanism was not exposed at the logical level, hiding its use within Core and all auth and secret plugins, preventing its misuse.

From commit messages we can guess this mechanism was an internal implementation detail of the proprietary Vault Enterprise Performance Replication mode and thus not relevant to improving snapshot consistency.

OpenBao instead moved to a much more powerful interactive transaction model:

// Transactional is an optional interface for backends that support
// interactive (mixed code & statement) transactions in a similar
// style as Go's Database paradigm. This is equivalent to
// physical.Transactional, not the earlier, one-shot version of the
// interface.
type Transactional interface {
	// This function allows the creation of a new interactive transaction
	// handle, only supporting read operations. Attempts to perform write
	// operations (Put(...) or Delete(...)) will err.
	BeginReadOnlyTx(ctx context.Context) (txn Transaction, err error)

	// This function allows the creation of a new interactive transaction
	// handle, supporting read/write transactions. In some cases, the
	// underlying physical storage backend cannot handle parallel read/write
	// transactions.
	BeginTx(ctx context.Context) (txn Transaction, err error)
}

// Transaction is an interactive transactional interface: backend storage
// operations can be performed, and when finished, Commit or Rollback can
// be called. When a read-only transaction is created, write calls (Put(...)
// and Delete(...)) will err out.
type Transaction interface {
	Storage

	// Commit a transaction; this is equivalent to Rollback on a read-only
	// transaction. Either Commit or Rollback must be called to release
	// resources.
	Commit(ctx context.Context) error

	// Rollback a transaction, preventing any changes from being persisted.
	// Either Commit or Rollback must be called to release resources.
	Rollback(ctx context.Context) error
}

(from main:sdk/logical/storage_transactions.go).

Callers of this API can perform arbitrary storage operations interleaved with non-storage calls and have consistency amongst all of them. This is implemented in both supported storage backends, Raft and PostgreSQL.

Reproducers

Looking at plugin code in our fork-point tag, any series of multi-write flow could potentially be affected by a snapshot consistency issue. However, to be affected by snapshot consistency issues, the server needs to assume that either both writes succeeded or neither did.

PKI

One such is in the PKI engine.

When creating a new issuer via <mount>/root/generate/internal, the following storage operations are performed:

1. config/key/<id>, to store the new root CA's key
2. config/keys, to store the new default issuer config
3. config/issuer/<id>, to import the new root CA's certificate
4. config/issuers, to store the new default issuer config
5. crls/<id>, to store the initial empty CRL

Notably consistency between storage operations 3 and 4 are the crucial: failure to store the initial issuer's identifier as default will mean that API compatibility with older Vault versions (and many third-party applications) which are not aware of multi-issuer features is broken. This will cause the API to return an error like:

err=Error making API request.

URL: GET http://localhost:8200/v1/058a0c0f-2dc3-a4a3-2e22-b00811700bac/issuer/default
Code: 500. Errors:

* 1 error occurred:
        * no default issuer currently configured

resp=<nil>

which will persist on all operations until an operator manually creates an association between the generated issuer and the default. Similar issues could occur whenever an issuer is rotated; the key could be persisted but the signing certificate could be dropped from the backup.

Notably, PKI's usage here would be a poor fit for check-and-set semantics: composability of importKey and importIssuer into writeCaBundle would be difficult to achieve while retaining transactional properties. This is why the stronger interactive transaction model is better for OpenBao's use case, even though it limits the theoretical storage backends one could implement as not every potential storage engine (like S3) implements interactive transactions.