HashiDays One conference. Three cities. Find a city near you
Vault
Vault Home

Overview

This page contains the list of deprecations and important or breaking changes for Vault 1.12.x compared to 1.11. Please read it carefully.

Changes

Supported storage backends

Vault Enterprise will now perform a supported storage check at startup. There is no impact on Vault Community Edition users.

For enterprise customers, HashiCorp provides official support for Vault's Integrated Storage and Consul as storage backends. Vault Enterprise customers are strongly recommended to use these supported storage backends for best outcomes. Version 1.12.0 of Vault Enterprise will not start if configured to use a storage backend other than Integrated Storage or Consul. This was meant to protect against issues caused by using unsupported backends that do not support transactional storage. Version 1.12.2 modified this behavior to instead log a warning when unsupported storage backends are used, while ensuring that Vault will start.

Consul dataplane compatibility

If you are using Consul on Kubernetes, please be aware that upgrading to Consul 1.14.0 will impact Consul secrets, storage, and service registration. As of Consul 1.14.0, Consul on Kubernetes uses Consul Dataplane by default instead of client agents. Vault does not currently support Consul Dataplane. Please follow the Consul 1.14.0 upgrade guide to ensure that your Consul on Kubernetes deployment continues to use client agents.

External plugin loading

Vault 1.12.0 introduced a change to how external plugins are loaded. Prior to Vault 1.12.0 plugins were lazy loaded on startup. This means that plugin processes were killed after a successful mount and then respawned when a request is routed to them. Vault 1.12.0 introduced auto mutual TLS for secrets/auth plugins so we do not lazy load them on startup anymore.

Activity Log Changes

Disable client counting activity

License utilization cannot be reported if client counting is disabled. As of Vault Enterprise 1.12.8 and later, client counting cannot be disabled using /sys/internal/counters/config endpoint when automated license utilization reporting is enabled.

Known issues

Pinning to builtin plugin versions may cause failure on upgrade

1.12.0 introduced plugin versions, and with it, the ability to explicitly specify the builtin version of a plugin when mounting an auth, database or secrets plugin. For example, vault auth enable -plugin-version=v1.12.0+builtin.vault approle. If there are any mounts where the builtin version was explicitly specified in this way, Vault may fail to start on upgrading to 1.12.1 due to the specified version no longer being available.

To check whether a mount path is affected, read the tune information, or the database config. The affected plugins are snowflake-database-plugin@v0.6.0+builtin and any plugins with +builtin.vault metadata in their version.

In this example, the first two mounts are affected because plugin_version is explicitly set and is one of the affected versions. The third mount is not affected because it only has +builtin metadata, and is not the Snowflake database plugin. All mounts where the version is omitted, or the plugin is external (regardless of whether the version is specified) are unaffected.

NOTE: Make sure you use Vault CLI 1.12.0 or later to check mounts.

$ vault read sys/auth/approle/tune
Key                  Value
---                  -----
...
plugin_version       v1.12.0+builtin.vault

$ vault read database/config/snowflake
Key                                   Value
---                                   -----
...
plugin_name                           snowflake-database-plugin
plugin_version                        v0.6.0+builtin

$ vault read sys/auth/kubernetes/tune
Key                  Value
---                  -----
...
plugin_version       v0.14.0+builtin

As it is not currently possible to unset the plugin version, there are 3 possible remediations if you have any affected mounts:

  • Upgrade Vault directly to 1.12.2 once released
  • Upgrade to an external version of the plugin before upgrading to 1.12.1;
  • Unmount and remount the path without a version specified before upgrading to 1.12.1. Note: This will delete all data and leases associated with the mount.

The bug was introduced by commit https://github.com/hashicorp/vault/commit/c36330f4c713b886a8a23c08cbbd862a7c530fc8.

Impacted versions

Affects upgrading from 1.12.0 to 1.12.1. All other upgrade paths are unaffected. 1.12.2 will introduce a fix that enables upgrades from affected deployments of 1.12.0.

Mounts associated with deprecated builtin plugins will result in core shutdown on upgrade

As of 1.12.0 Standalone (logical) DB Engines and the AppId Auth Method have been marked with the Pending Removal status. Any attempt to unseal Vault with mounts backed by one of these builtin plugins will result in an immediate shutdown of the Vault core.

NOTE In the event that an external plugin with the same name and type as a deprecated builtin is deregistered, any subsequent unseal of Vault will also result in a core shutdown.

$ vault plugin register -sha256=c805cf3b69f704dfcd5176ef1c7599f88adbfd7374e9c76da7f24a32a97abfe1 auth app-id
Success! Registered plugin: app-id
$ vault auth enable -plugin-name=app-id plugin
Success! Enabled app-id auth method at: app-id/
$ vault auth list -detailed
app-id/    app-id    auth_app-id_3a8f2e24    system         system     default-service    replicated     false        false                      map[]      n/a                        0018263c-0d64-7a70-fd5c-50e05c5f5dc3    n/a        n/a                      c805cf3b69f704dfcd5176ef1c7599f88adbfd7374e9c76da7f24a32a97abfe1    n/a
$ vault plugin deregister auth app-id
Success! Deregistered plugin (if it was registered): app-id
$ vault plugin list -detailed | grep "app-id"
app-id                               auth        v1.12.0+builtin.vault                                 pending removal

The remediation for affected mounts is to set the VAULT_ALLOW_PENDING_REMOVAL_MOUNTS environment variable and replace any Pending Removal feature with the preferred alternative feature.

For more information on the phases of deprecation, see the Deprecation Notices FAQ.

Impacted versions

Affects upgrading from any version of Vault to 1.12.x. All other upgrade paths are unaffected.

vault plugin list fails when audit logging is enabled

If audit logging is enabled, Vault will fail to audit the response from any calls to the GET /v1/sys/plugins/catalog endpoint, which causes the whole request to fail and return a 500 internal server error. From the CLI, this looks like the following:

$ vault plugin list
Error listing available plugins: data from server response is empty

It will produce errors in Vault Server's logs such as:

2022-11-30T20:04:22.397Z [ERROR] audit: panic during logging: request_path=sys/plugins/catalog error="reflect: reflect.Value.Set using value obtained using unexported field"
2022-11-30T20:04:22.398Z [ERROR] core: failed to audit response: request_path=sys/plugins/catalog
  error=
  | 1 error occurred:
  |     * panic generating audit log
  |

As a workaround, listing plugins by type will succeed:

The bug was introduced by commit https://github.com/hashicorp/vault/commit/76165052e54f884ed0aa2caa496083dc84ad1c19.

Impacted versions

Affects versions 1.12.0, 1.12.1, and 1.12.2. A fix will be released in 1.12.3.

PKI OCSP GET requests return malformed request responses

If an OCSP GET request contains a '+' character, a malformed request response will be returned instead of properly processing the request due to a double decoding issue within the handler.

As a workaround, OCSP POST requests can be used which are unaffected.

Impacted versions

Affects version 1.12.3. A fix will be released in 1.12.4.

Rotation configuration persistence issue could lose transform tokenization key versions

A rotation performed manually or via automatic time based rotation after restarting or leader change of Vault, where configuration of rotation was changed since the initial configuration of the tokenization transform can result in the loss of intermediate key versions. Tokenized values from these versions would not be decodeable. It is recommended that customers who have enabled automatic rotation disable it, and other customers avoid key rotation until the upcoming fix.

Affected versions

This issue affects Vault Enterprise with ADP versions 1.10.x and higher. A fix will be released in Vault 1.11.9, 1.12.5, and 1.13.1.

PKI OCSP GET requests can return HTTP redirect responses

If a base64 encoded OCSP request contains consecutive '/' characters, the GET request will return a 301 permanent redirect response. If the redirection is followed, the request will not decode as it will not be a properly base64 encoded request.

As a workaround, OCSP POST requests can be used which are unaffected.

Impacted versions

Affects all current versions of 1.12.x, 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x.

LDAP pagination issue

There was a regression introduced in 1.12.6 relating to LDAP maximum page sizes, resulting in an error no LDAP groups found in groupDN [...] only policies from locally-defined groups available. The issue occurs when upgrading Vault with an instance that has an existing LDAP Auth configuration.

As a workaround, disable paged searching using the following:

vault write auth/ldap/config max_page_size=-1

Impacted versions

Affects Vault 1.12.6.

Slow startup time when storing PKI certificates

There was a regression introduced in 1.12.0 where Vault is slow to start because the PKI secret engine performs a list operation on the stored certificates. If a large number of certificates are stored this can cause long start times on active and standby nodes.

There is currently no workaround for this other than limiting the number of certificates stored in Vault via the PKI tidy or using no_store flag for PKI roles.

Impacted versions

Affects Vault 1.12.0+

PKI storage migration revives deleted issuers

Vault 1.11 introduced Storage v1, a new storage layout that supported multiple issuers within a single mount. Bug fixes in Vault 1.11.6, 1.12.2, and 1.13.0 corrected a write-ordering issue that lead to invalid CA chains. Specifically, incorrectly ordered writes could fail due to load, resulting in the mount being re-migrated next time it was loaded or silently truncating CA chains. This collection of bug fixes introduced Storage v2.

Affected versions

Vault may incorrectly re-migrated legacy issuers created before Vault 1.11 that were migrated to Storage v1 and deleted before upgrading to a Vault version with Storage v2.

The migration fails when Vault finds managed keys associated with the legacy issuers that were removed from the managed key repository prior to the upgrade.

The migration error appears in Vault logs as:

Error during migration of PKI mount: failed to lookup public key from managed key: no managed key found with uuid

Note

Issuers created in Vault 1.11+ and direct upgrades to a Storage v2 layout are not affected.

The Storage v1 upgrade bug was fixed in Vault 1.14.1, 1.13.5, and 1.12.9.

Edit this page on GitHub