Every production system has secrets — database credentials, API keys, TLS certificates, encryption keys, OAuth tokens, webhook signing keys. The question is not whether you have them. The question is whether you can tell an auditor exactly where every secret is stored, who has accessed it in the last 90 days, when it was last rotated, and what would happen if it were compromised right now.
Most teams cannot answer those questions. Secrets accumulate organically: a database password set during initial deployment three years ago, still in use, stored in an environment variable on four servers and a CI/CD pipeline. An API key for a payment processor, committed to a private Git repository in 2024, rotated once when someone remembered, currently shared by two microservices that have no idea the other exists. A TLS private key for your production domain, generated by certbot, stored on disk with standard file permissions, never backed up to a secure location.
For regulated workloads — fintech platforms under FINMA supervision, SaaS companies processing EU personal data under GDPR, crypto exchanges handling customer assets — this ad hoc approach creates compliance exposure that compounds over time. GDPR Article 32 requires "appropriate technical measures" for data security, and regulators increasingly interpret this to include cryptographic key management. FINMA's operational risk circular expects financial institutions to maintain controls over access credentials and encryption keys. The Swiss FADP's implementing ordinance requires that technical measures be proportionate to the sensitivity of the data — and for financial data, "proportionate" means structured, auditable, and automated.
This guide covers how to build secrets management infrastructure that satisfies these requirements — deployed on managed Swiss infrastructure where jurisdictional control over your key material is unambiguous and hardware-level security is available without cloud provider abstraction layers.
Why Secrets Management Is a Compliance Requirement, Not Just an Engineering Preference
Before diving into architecture, it is worth understanding why regulators care about secrets management specifically. The reasoning follows a chain:
Data protection requires encryption. GDPR Article 32(1)(a) explicitly mentions encryption as an appropriate technical measure. The FADP's Article 8 requires security measures proportionate to risk. For financial data, encryption is not optional — it is the baseline expectation.
Encryption is only as strong as key management. AES-256 is unbreakable. AES-256 with the key stored in a plaintext config file next to the encrypted data is theatre. Regulators have learned this distinction. Modern audit frameworks — SOC 2 Type II, ISO 27001 Annex A.10, PCI DSS Requirement 3.5 — all require documented key management procedures covering generation, distribution, storage, rotation, and destruction.
Key management requires infrastructure. You cannot manage cryptographic keys in spreadsheets. You cannot rotate database credentials manually across a fleet of microservices without downtime. You cannot prove to an auditor that a compromised key was revoked within your incident response SLA without automated tooling. Secrets management infrastructure is the mechanism that makes key management policies technically enforceable.
Infrastructure requires jurisdictional control. If your key management system runs on infrastructure in a jurisdiction where foreign authorities can compel key disclosure, your encryption provides weaker guarantees than your compliance documentation claims. This is not theoretical — the US CLOUD Act allows compelled disclosure of data (including encryption keys) held by US companies regardless of where the data is physically stored. Swiss jurisdiction under the FADP eliminates this specific risk.
Architecture: Centralised Secrets Management with HashiCorp Vault
HashiCorp Vault is the de facto standard for secrets management in regulated environments. It is open source, battle-tested in financial services, and designed from the ground up for the access control, audit logging, and key lifecycle requirements that compliance demands. Here is how to deploy it on dedicated Swiss servers for maximum security assurance.
Vault server deployment
The foundation is a hardened Vault deployment on infrastructure you physically control. Cloud-managed Vault services (HCP Vault) are convenient but introduce a third-party trust dependency that complicates regulatory discussions. On a dedicated Swiss server, you control the full stack — from disk encryption to memory isolation to network access.
# /etc/vault.d/vault.hcl
# Production Vault configuration for regulated workloads
# Storage backend — Integrated Raft for high availability
storage "raft" {
path = "/opt/vault/data"
node_id = "vault-ch-01"
# Raft performance tuning for dedicated hardware
performance_multiplier = 1
retry_join {
leader_api_addr = "https://vault-ch-02.internal:8200"
leader_ca_cert_file = "/opt/vault/tls/ca.crt"
leader_client_cert_file = "/opt/vault/tls/vault.crt"
leader_client_key_file = "/opt/vault/tls/vault.key"
}
retry_join {
leader_api_addr = "https://vault-ch-03.internal:8200"
leader_ca_cert_file = "/opt/vault/tls/ca.crt"
leader_client_cert_file = "/opt/vault/tls/vault.crt"
leader_client_key_file = "/opt/vault/tls/vault.key"
}
}
# Listener configuration — TLS only, no plaintext
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/vault/tls/vault.crt"
tls_key_file = "/opt/vault/tls/vault.key"
tls_min_version = "tls13"
# Client certificate authentication for inter-node communication
tls_client_ca_file = "/opt/vault/tls/ca.crt"
# Disable non-TLS listeners entirely
tls_disable = false
}
# API and cluster addresses
api_addr = "https://vault-ch-01.internal:8200"
cluster_addr = "https://vault-ch-01.internal:8201"
# Audit logging — MANDATORY for compliance
# Write to multiple backends for redundancy
audit {
type = "file"
path = "file"
options = {
file_path = "/var/log/vault/audit.log"
log_raw = false # Do not log raw secret values
mode = "0600"
}
}
# Telemetry for monitoring
telemetry {
prometheus_retention_time = "24h"
disable_hostname = true
}
# Security hardening
disable_mlock = false # Keep secrets in RAM, prevent swapping to disk
ui = false # Disable web UI in production — API only
# Maximum lease TTL — force credential rotation
max_lease_ttl = "768h" # 32 days maximum
default_lease_ttl = "24h" # 1 day default — forces frequent renewalKey design decisions here: disable_mlock = false ensures Vault's memory pages are locked in RAM and never written to swap — this prevents secrets from leaking to disk even during memory pressure. The Raft storage backend eliminates external database dependencies (no Consul or PostgreSQL cluster to secure separately). TLS 1.3 minimum with mutual TLS between nodes prevents network-level interception. And the short default lease TTL (24 hours) forces applications to regularly renew their credentials, reducing the window of exposure if a credential is compromised.
Initialisation and unseal with Shamir's Secret Sharing
Vault's initialisation process generates a master encryption key that protects all stored secrets. This key is split using Shamir's Secret Sharing — a cryptographic technique that divides the key into multiple shares, requiring a threshold number of shares to reconstruct it. For regulated environments, this provides separation of duties: no single person can unseal the vault.
# Initialise Vault with 5 key shares, requiring 3 to unseal
# This should be done ONCE, with key holders physically present
vault operator init \
-key-shares=5 \
-key-threshold=3 \
-format=json > /tmp/vault-init.json
# CRITICAL: Distribute key shares to separate individuals
# Each key holder should receive exactly ONE share
# Store shares in separate physical locations (safe deposit boxes, etc.)
# The init output also contains the root token — store it separately
# Unseal process (requires 3 of 5 key holders)
vault operator unseal # Key holder 1 enters their share
vault operator unseal # Key holder 2 enters their share
vault operator unseal # Key holder 3 enters their share
# After unsealing, revoke the root token for daily operations
# Use identity-based authentication instead
vault token revoke $VAULT_ROOT_TOKEN
# For production: consider auto-unseal with a hardware security module
# This eliminates the manual unseal ceremony after restartsThe 5/3 threshold means your organisation can tolerate two key holders being unavailable (vacation, departure, emergency) and still unseal the vault. For larger organisations, a 7/4 or even 7/5 split provides additional resilience. The important thing is that the threshold requires collaboration — a single compromised key holder cannot access the vault's contents.
Dynamic Secrets: Credentials That Expire Before They Can Be Stolen
Static credentials — passwords set once and used forever — are the single biggest secrets management anti-pattern in regulated environments. If a database password has been the same for six months, it has been exposed to every CI/CD pipeline run, every developer who checked the environment variables, every monitoring tool that connects to the database, and every backup that includes the application configuration. The blast radius of compromise is six months of activity.
Dynamic secrets flip this model. Instead of storing a long-lived password, applications request short-lived credentials from Vault at runtime. Vault generates unique credentials for each request, with an expiration time measured in hours rather than months. When the lease expires, the credentials are automatically revoked.
PostgreSQL dynamic credentials
# Configure Vault's database secrets engine for PostgreSQL
# This creates a connection to your database that Vault uses to
# generate and revoke temporary credentials
vault secrets enable database
vault write database/config/production-db \
plugin_name="postgresql-database-plugin" \
allowed_roles="app-readonly,app-readwrite,migration-admin" \
connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/production?sslmode=verify-full" \
username="vault_admin" \
password="initial_strong_password" \
password_authentication="scram-sha-256"
# Rotate the initial static password immediately
# After this, only Vault knows the connection password
vault write -force database/rotate-root/production-db
# Define roles with specific permissions and TTLs
# Read-only role for API services (4-hour credentials)
vault write database/roles/app-readonly \
db_name="production-db" \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\"; \
GRANT SELECT ON ALL TABLES IN SCHEMA customer_data TO \"{{name}}\";" \
revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; \
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA customer_data FROM \"{{name}}\"; \
DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="4h" \
max_ttl="8h"
# Read-write role for transaction processing (2-hour credentials)
vault write database/roles/app-readwrite \
db_name="production-db" \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\"; \
GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";" \
revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; \
REVOKE USAGE ON ALL SEQUENCES IN SCHEMA public FROM \"{{name}}\"; \
DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="2h" \
max_ttl="4h"
# Migration role — short-lived, high-privilege (30-minute credentials)
vault write database/roles/migration-admin \
db_name="production-db" \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' SUPERUSER;" \
revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="30m" \
max_ttl="1h"Notice the different TTLs per role. The migration-admin role gets 30-minute credentials because schema migrations are discrete operations — there is no reason for a superuser credential to exist for longer than the migration takes. The read-write role for transaction processing gets 2-hour credentials, forcing services to renew frequently. Every credential is unique (Vault generates a random username and password each time), so if one credential leaks, it expires within hours and cannot be used to access anything the leaked service was not already connected to.
Application integration pattern
# Python: Application-level Vault integration for dynamic database credentials
import hvac
import psycopg2
import threading
import time
import logging
logger = logging.getLogger(__name__)
class VaultManagedDatabase:
"""
Manages database connections with Vault-issued dynamic credentials.
Handles credential rotation, lease renewal, and graceful failover.
"""
def __init__(self, vault_addr: str, vault_role: str, db_host: str,
db_name: str, auth_method: str = "approle"):
self.vault = hvac.Client(url=vault_addr)
self.vault_role = vault_role
self.db_host = db_host
self.db_name = db_name
self._connection = None
self._lease_id = None
self._lease_expiry = 0
self._lock = threading.Lock()
self._renewal_thread = None
# Authenticate to Vault using AppRole
# (role_id from deployment config, secret_id from secure bootstrap)
if auth_method == "approle":
self._authenticate_approle()
elif auth_method == "kubernetes":
self._authenticate_k8s()
def _authenticate_approle(self):
"""Authenticate using AppRole with response wrapping."""
import os
role_id = os.environ.get("VAULT_ROLE_ID")
# Secret ID delivered via response wrapping token (single-use)
wrapped_token = os.environ.get("VAULT_SECRET_ID_WRAPPED")
if wrapped_token:
# Unwrap the secret ID (consumes the wrapping token)
unwrap_response = self.vault.sys.unwrap(wrapped_token)
secret_id = unwrap_response["data"]["secret_id"]
else:
secret_id = os.environ.get("VAULT_SECRET_ID")
self.vault.auth.approle.login(
role_id=role_id,
secret_id=secret_id,
)
logger.info("Authenticated to Vault via AppRole")
def get_connection(self):
"""Get a database connection with valid credentials."""
with self._lock:
# Check if current connection and credentials are still valid
if self._connection and not self._connection.closed:
if time.time() < self._lease_expiry - 300: # 5-min buffer
return self._connection
else:
logger.info("Credentials expiring soon, rotating")
# Request new credentials from Vault
creds = self.vault.secrets.database.generate_credentials(
name=self.vault_role,
)
username = creds["data"]["username"]
password = creds["data"]["password"]
self._lease_id = creds["lease_id"]
lease_duration = creds["lease_duration"]
self._lease_expiry = time.time() + lease_duration
logger.info(
f"New database credentials issued: user={username}, "
f"lease_ttl={lease_duration}s, lease_id={self._lease_id[:20]}..."
)
# Close old connection gracefully
if self._connection and not self._connection.closed:
try:
self._connection.close()
except Exception:
pass
# Establish new connection
self._connection = psycopg2.connect(
host=self.db_host,
database=self.db_name,
user=username,
password=password,
sslmode="verify-full",
sslrootcert="/etc/ssl/certs/db-ca.crt",
connect_timeout=10,
)
# Start lease renewal in background
self._start_renewal()
return self._connection
def _start_renewal(self):
"""Background thread to renew lease before expiry."""
if self._renewal_thread and self._renewal_thread.is_alive():
return
def renew_loop():
while True:
time_to_expiry = self._lease_expiry - time.time()
if time_to_expiry <= 0:
break
# Renew at 2/3 of remaining lease time
sleep_time = max(time_to_expiry * 2 / 3, 30)
time.sleep(sleep_time)
try:
self.vault.sys.renew_lease(
lease_id=self._lease_id,
increment=None, # Use default TTL
)
logger.info(f"Lease renewed: {self._lease_id[:20]}...")
except Exception as e:
logger.warning(f"Lease renewal failed: {e}")
break
self._renewal_thread = threading.Thread(
target=renew_loop, daemon=True
)
self._renewal_thread.start()
def close(self):
"""Revoke credentials and close connection."""
if self._lease_id:
try:
self.vault.sys.revoke_lease(self._lease_id)
logger.info(f"Lease revoked: {self._lease_id[:20]}...")
except Exception as e:
logger.warning(f"Lease revocation failed: {e}")
if self._connection and not self._connection.closed:
self._connection.close()
# Usage in a fintech application
db = VaultManagedDatabase(
vault_addr="https://vault.internal:8200",
vault_role="app-readonly",
db_host="db.internal",
db_name="production",
)
# The application never sees or stores credentials
# Vault handles generation, rotation, and revocation
conn = db.get_connection()
cursor = conn.cursor()
cursor.execute("SELECT balance FROM accounts WHERE id = %s", (account_id,))
# On shutdown, credentials are explicitly revoked
db.close()The critical design principle: your application code never contains, stores, or logs credentials. It asks Vault for a connection, uses it, and releases it. The credentials exist only in memory for the duration of the lease. When the lease expires or the application shuts down, the credentials are revoked in the database — they literally cease to exist. An attacker who dumps the application's memory gets credentials that will expire in hours, not credentials that have been valid for years.
Encryption as a Service: Vault's Transit Secrets Engine
Beyond credential management, regulated workloads need application-level encryption — encrypting sensitive fields in your database, signing tokens, generating HMACs for webhook verification. The naive approach is to embed encryption logic in your application with keys stored alongside the code. The compliant approach is to delegate all cryptographic operations to a dedicated service that manages key lifecycle independently.
Vault's Transit secrets engine provides encryption as a service. Your application sends plaintext to Vault and receives ciphertext. The encryption key never leaves Vault — your application cannot extract it, which means a compromised application server cannot leak your encryption keys.
# Configure Transit secrets engine for application-level encryption
vault secrets enable transit
# Create encryption keys with specific purposes and rotation policies
# Customer PII encryption key — AES-256-GCM, auto-rotates every 30 days
vault write transit/keys/customer-pii \
type="aes256-gcm96" \
auto_rotate_period="720h" \
deletion_allowed=false \
exportable=false \
min_decryption_version=1 \
min_encryption_version=0
# Payment data encryption key — stricter rotation for PCI compliance
vault write transit/keys/payment-data \
type="aes256-gcm96" \
auto_rotate_period="168h" \
deletion_allowed=false \
exportable=false
# Token signing key — RSA for JWT signing
vault write transit/keys/token-signing \
type="rsa-4096" \
auto_rotate_period="2160h" \
deletion_allowed=false \
exportable=false
# Webhook HMAC key
vault write transit/keys/webhook-hmac \
type="aes256-gcm96" \
auto_rotate_period="720h"# Application: encrypt sensitive data before database storage
import hvac
import base64
class FieldEncryptor:
"""Encrypts sensitive database fields via Vault Transit."""
def __init__(self, vault_client: hvac.Client):
self.vault = vault_client
def encrypt_pii(self, plaintext: str) -> str:
"""Encrypt a PII field (name, email, phone, etc.)."""
b64_input = base64.b64encode(plaintext.encode()).decode()
result = self.vault.secrets.transit.encrypt_data(
name="customer-pii",
plaintext=b64_input,
context=None, # Add derived key context for per-record keys
)
return result["data"]["ciphertext"] # "vault:v2:base64..."
def decrypt_pii(self, ciphertext: str) -> str:
"""Decrypt a PII field."""
result = self.vault.secrets.transit.decrypt_data(
name="customer-pii",
ciphertext=ciphertext,
)
return base64.b64decode(result["data"]["plaintext"]).decode()
def encrypt_payment(self, card_data: dict) -> str:
"""Encrypt payment data with the stricter rotation key."""
import json
b64_input = base64.b64encode(json.dumps(card_data).encode()).decode()
result = self.vault.secrets.transit.encrypt_data(
name="payment-data",
plaintext=b64_input,
)
return result["data"]["ciphertext"]
def rewrap_field(self, ciphertext: str, key_name: str) -> str:
"""
Re-encrypt data with the latest key version WITHOUT decrypting.
Use during key rotation — the plaintext never leaves Vault.
"""
result = self.vault.secrets.transit.rewrap_data(
name=key_name,
ciphertext=ciphertext,
)
return result["data"]["ciphertext"]
# Usage: store encrypted PII in PostgreSQL
enc = FieldEncryptor(vault_client)
# Encrypt before INSERT
encrypted_email = enc.encrypt_pii("customer@example.com")
encrypted_phone = enc.encrypt_pii("+41 44 123 4567")
cursor.execute(
"INSERT INTO customers (id, email_encrypted, phone_encrypted, country) "
"VALUES (%s, %s, %s, %s)",
(customer_id, encrypted_email, encrypted_phone, "CH")
)
# Decrypt on SELECT (only when needed, by authorised service)
cursor.execute("SELECT email_encrypted FROM customers WHERE id = %s", (customer_id,))
row = cursor.fetchone()
email = enc.decrypt_pii(row[0]) # Plaintext never stored in DBThe rewrap operation is particularly important for compliance. When Vault rotates the encryption key (automatically, per your rotation schedule), existing ciphertext is still encrypted with the old key version. The rewrap call re-encrypts the data with the new key version — crucially, without the plaintext ever leaving Vault. You can run a background job that iterates through your database and rewraps every encrypted field after a key rotation, and at no point does the raw data leave the boundary of the Vault server. This satisfies PCI DSS Requirement 3.6.4 (cryptographic key changes for keys that have reached the end of their cryptoperiod) without the complexity and risk of a decrypt-re-encrypt migration.
HSM Integration: Hardware-Backed Key Protection
For the highest assurance — required by some financial regulators and recommended by most — encryption keys should be protected by a Hardware Security Module (HSM). An HSM is a dedicated cryptographic processor that generates and stores keys in tamper-resistant hardware. Keys in an HSM cannot be extracted, even by someone with physical access to the device. If someone opens the HSM's enclosure, the device detects the intrusion and destroys its stored keys.
On dedicated Swiss servers, you can deploy HSMs as PCIe cards in the server chassis or as network-attached appliances. Vault integrates with HSMs through its auto-unseal mechanism and through PKCS#11 for Transit operations.
# Vault configuration with HSM auto-unseal
# This replaces Shamir key shares with HSM-protected unseal
# vault.hcl HSM seal configuration
seal "pkcs11" {
lib = "/usr/lib/softhsm/libsofthsm2.so" # HSM PKCS#11 library
slot = "0"
pin = "env:VAULT_HSM_PIN" # PIN from environment, not config file
key_label = "vault-master-key"
mechanism = "0x1085" # CKM_AES_KEY_WRAP_PAD
hmac_mechanism = "0x0251" # CKM_SHA256_HMAC
generate_key = "true" # HSM generates the key (never leaves hardware)
}
# For production with a network HSM (e.g., Thales Luna, Securosys Primus):
# seal "pkcs11" {
# lib = "/usr/lib/libCryptoki2_64.so"
# slot = "1"
# pin = "env:VAULT_HSM_PIN"
# key_label = "vault-master-2026"
# mechanism = "0x1085"
# generate_key = "true"
# }With HSM auto-unseal, the Vault startup process changes fundamentally. Instead of requiring three key holders to manually enter their Shamir shares after every restart, Vault automatically unseals by requesting its master key from the HSM. The HSM authenticates the request (PIN plus optional mutual TLS), releases the wrapped key, and Vault resumes operation. This enables unattended restarts — critical for high-availability deployments — without sacrificing key protection.
The compliance benefit is substantial. When an auditor asks "where is your master encryption key stored?" the answer is "in a FIPS 140-2 Level 3 certified hardware security module, physically located in a Swiss data centre, and the key has never existed outside of that hardware." That is a categorically different answer from "in five pieces, distributed to five people who we hope have stored them securely."
Access Control: Who Gets Which Secrets
Secrets management without granular access control is a centralised password file with extra steps. Vault's policy engine lets you define exactly which secrets each application, team, and individual can access — and the audit log records every access decision.
# Vault policies for a fintech platform
# Policy: payment-service
# Can read payment processing credentials and encrypt payment data
# Cannot access customer PII keys or admin credentials
path "database/creds/app-readwrite" {
capabilities = ["read"]
}
path "transit/encrypt/payment-data" {
capabilities = ["update"]
}
path "transit/decrypt/payment-data" {
capabilities = ["update"]
}
# Explicitly deny access to PII encryption keys
path "transit/encrypt/customer-pii" {
capabilities = ["deny"]
}
path "transit/decrypt/customer-pii" {
capabilities = ["deny"]
}
# Policy: customer-api
# Can read customer data (including PII decryption) but not payment data
path "database/creds/app-readonly" {
capabilities = ["read"]
}
path "transit/encrypt/customer-pii" {
capabilities = ["update"]
}
path "transit/decrypt/customer-pii" {
capabilities = ["update"]
}
path "transit/encrypt/payment-data" {
capabilities = ["deny"]
}
path "transit/decrypt/payment-data" {
capabilities = ["deny"]
}
# Policy: migration-runner
# Highly privileged but extremely short-lived
# Only accessible via specific CI/CD pipeline with approval gates
path "database/creds/migration-admin" {
capabilities = ["read"]
}
# Can rotate database root credentials
path "database/rotate-root/production-db" {
capabilities = ["update"]
}
# Policy: security-team
# Can view audit logs and key metadata but NOT access secrets
path "sys/audit" {
capabilities = ["read", "list"]
}
path "transit/keys/*" {
capabilities = ["read", "list"] # Metadata only — keys are non-exportable
}
path "sys/leases/lookup" {
capabilities = ["update"]
}
# Cannot read any actual secrets
path "database/creds/*" {
capabilities = ["deny"]
}
path "secret/*" {
capabilities = ["deny"]
}The principle of least privilege is enforced at the policy level, not by convention. The payment-service literally cannot access customer PII encryption keys — Vault will deny the request and log the attempt. The security team can audit key usage and review access patterns but cannot read the actual secrets. This separation of duties is exactly what auditors look for, and it is enforced cryptographically rather than procedurally.
Key Rotation and Lifecycle Management
Key rotation is one of those compliance requirements that everyone documents and few implement correctly. The challenge is not generating a new key — it is ensuring that all systems transition to the new key without downtime, that data encrypted with the old key remains accessible, and that old key versions are eventually retired according to your cryptoperiod policy.
#!/bin/bash
# /usr/local/bin/key-rotation-manager.sh
# Automated key rotation with compliance tracking
# Runs via cron; integrates with monitoring and alerting
set -euo pipefail
VAULT_ADDR="https://vault.internal:8200"
LOG_FILE="/var/log/vault/key-rotation.log"
ALERT_WEBHOOK="${ALERT_WEBHOOK_URL}"
log_event() {
echo "{\"timestamp\": \"$(date -Iseconds)\", \"event\": \"$1\", \
\"key\": \"$2\", \"detail\": \"$3\"}" >> "${LOG_FILE}"
}
rotate_transit_key() {
local KEY_NAME="$1"
local MIN_VERSION="$2"
# Get current key info
KEY_INFO=$(vault read -format=json "transit/keys/${KEY_NAME}")
CURRENT_VERSION=$(echo "${KEY_INFO}" | jq -r '.data.latest_version')
CURRENT_MIN_DECRYPT=$(echo "${KEY_INFO}" | jq -r '.data.min_decryption_version')
log_event "rotation_start" "${KEY_NAME}" "current_version=${CURRENT_VERSION}"
# Rotate the key (creates new version, old versions remain for decryption)
vault write -f "transit/keys/${KEY_NAME}/rotate"
NEW_VERSION=$((CURRENT_VERSION + 1))
log_event "key_rotated" "${KEY_NAME}" "new_version=${NEW_VERSION}"
# Update minimum encryption version (new data uses new key only)
vault write "transit/keys/${KEY_NAME}/config" \
min_encryption_version="${NEW_VERSION}"
log_event "min_encrypt_updated" "${KEY_NAME}" \
"min_encryption_version=${NEW_VERSION}"
# After data rewrap completes (separate job), update min decryption version
# This retires old key versions — they can no longer decrypt anything
if [ "${MIN_VERSION}" -gt 0 ] && \
[ "${CURRENT_MIN_DECRYPT}" -lt "${MIN_VERSION}" ]; then
vault write "transit/keys/${KEY_NAME}/config" \
min_decryption_version="${MIN_VERSION}"
log_event "old_versions_retired" "${KEY_NAME}" \
"min_decryption_version=${MIN_VERSION}"
fi
echo "Rotated ${KEY_NAME}: v${CURRENT_VERSION} -> v${NEW_VERSION}"
}
# Rotate keys according to compliance schedule
echo "=== Key Rotation Run: $(date -Iseconds) ==="
# Customer PII key — monthly rotation
rotate_transit_key "customer-pii" 0
# Payment data key — weekly rotation (PCI DSS)
rotate_transit_key "payment-data" 0
# Check for keys approaching maximum cryptoperiod
KEYS=$(vault list -format=json transit/keys | jq -r '.[]')
for KEY in ${KEYS}; do
KEY_INFO=$(vault read -format=json "transit/keys/${KEY}")
LATEST=$(echo "${KEY_INFO}" | jq -r '.data.latest_version')
OLDEST_ACTIVE=$(echo "${KEY_INFO}" | jq -r '.data.min_decryption_version')
# Alert if more than 12 versions active (means old data not being rewrapped)
ACTIVE_VERSIONS=$((LATEST - OLDEST_ACTIVE + 1))
if [ "${ACTIVE_VERSIONS}" -gt 12 ]; then
log_event "compliance_warning" "${KEY}" \
"active_versions=${ACTIVE_VERSIONS} — data rewrap may be behind schedule"
# Send alert
curl -s -X POST "${ALERT_WEBHOOK}" \
-H "Content-Type: application/json" \
-d "{\"text\": \"Key rotation warning: ${KEY} has ${ACTIVE_VERSIONS} active versions. Data rewrap may be behind schedule.\"}"
fi
done
echo "=== Key Rotation Complete ==="The lifecycle flow: rotate the key (new version created) → update minimum encryption version (new data always uses latest) → run data rewrap job (existing data re-encrypted with new key) → update minimum decryption version (retire old key versions). This four-step process is the only way to fully rotate an encryption key without data loss or downtime.
Data rewrap job for post-rotation cleanup
#!/usr/bin/env python3
"""
Post-rotation data rewrap job.
Re-encrypts all stored ciphertext with the latest key version.
Runs after key rotation; tracks progress for compliance reporting.
"""
import psycopg2
import hvac
import json
import time
import logging
from datetime import datetime, timezone
logger = logging.getLogger(__name__)
class DataRewrapper:
"""Re-encrypts database fields with the latest Vault Transit key version."""
def __init__(self, vault_client: hvac.Client, db_conn):
self.vault = vault_client
self.db = db_conn
def rewrap_table(self, table: str, columns: list, key_name: str,
batch_size: int = 500) -> dict:
"""
Rewrap encrypted columns in a table.
Returns statistics for compliance reporting.
"""
stats = {
"table": table,
"key": key_name,
"started_at": datetime.now(timezone.utc).isoformat(),
"total_rows": 0,
"rewrapped": 0,
"already_current": 0,
"errors": 0,
}
# Get the current key version
key_info = self.vault.secrets.transit.read_key(name=key_name)
current_version = key_info["data"]["latest_version"]
version_prefix = f"vault:v{current_version}:"
cursor = self.db.cursor()
# Process in batches to avoid locking the table
offset = 0
while True:
col_list = ", ".join(["id"] + columns)
cursor.execute(
f"SELECT {col_list} FROM {table} "
f"ORDER BY id LIMIT %s OFFSET %s",
(batch_size, offset)
)
rows = cursor.fetchall()
if not rows:
break
for row in rows:
row_id = row[0]
stats["total_rows"] += 1
updates = {}
needs_update = False
for i, col in enumerate(columns):
ciphertext = row[i + 1]
if ciphertext is None:
continue
# Skip if already encrypted with current version
if ciphertext.startswith(version_prefix):
stats["already_current"] += 1
continue
try:
# Rewrap: Vault re-encrypts with latest key
# Plaintext NEVER leaves Vault during this operation
result = self.vault.secrets.transit.rewrap_data(
name=key_name,
ciphertext=ciphertext,
)
updates[col] = result["data"]["ciphertext"]
needs_update = True
except Exception as e:
logger.error(f"Rewrap failed for {table}.{col} "
f"id={row_id}: {e}")
stats["errors"] += 1
if needs_update:
set_clause = ", ".join(
[f"{col} = %s" for col in updates.keys()]
)
cursor.execute(
f"UPDATE {table} SET {set_clause} WHERE id = %s",
list(updates.values()) + [row_id]
)
stats["rewrapped"] += 1
self.db.commit()
offset += batch_size
# Rate limit to avoid overloading Vault
time.sleep(0.1)
stats["completed_at"] = datetime.now(timezone.utc).isoformat()
return stats
# Run rewrap for all encrypted tables
rewrapper = DataRewrapper(vault_client, db_conn)
results = []
results.append(rewrapper.rewrap_table(
"customers",
["email_encrypted", "phone_encrypted", "address_encrypted"],
"customer-pii"
))
results.append(rewrapper.rewrap_table(
"payment_methods",
["card_data_encrypted", "billing_encrypted"],
"payment-data"
))
# Write compliance report
report = {
"rotation_id": f"rot-{int(time.time())}",
"completed": datetime.now(timezone.utc).isoformat(),
"tables": results,
"all_clear": all(r["errors"] == 0 for r in results),
}
with open("/var/log/vault/rewrap-report.json", "a") as f:
f.write(json.dumps(report) + "\n")
logger.info(f"Rewrap complete: {json.dumps(report, indent=2)}")The rewrap report is your compliance evidence that key rotation actually completed end-to-end. When an auditor asks "you rotate your encryption keys monthly — can you show me proof?" you hand them the rewrap reports showing every table, every column, the number of records re-encrypted, and the completion timestamp. This is the kind of concrete evidence that distinguishes "we have a policy" from "we have an implemented, verified policy."
Audit Logging: Every Secret Access Recorded
Vault's audit log captures every API request — every secret read, every credential generation, every encryption operation, every policy evaluation. For regulated environments, this audit trail is as important as the secrets management itself.
# Enable multiple audit backends for redundancy
# If ALL audit backends fail, Vault stops serving requests
# (fail-closed — compliance over availability)
# Primary: local file (collected by rsyslog to central audit store)
vault audit enable file \
file_path=/var/log/vault/audit.log \
log_raw=false \
hmac_accessor=true \
mode=0600
# Secondary: syslog (independent transport path)
vault audit enable syslog \
tag="vault-audit" \
facility="AUTH" \
log_raw=false
# Example audit log entry (formatted for readability):
# {
# "time": "2026-09-24T10:15:23.847Z",
# "type": "request",
# "auth": {
# "client_token": "hmac-sha256:abc123...",
# "accessor": "hmac-sha256:def456...",
# "display_name": "approle-payment-service",
# "policies": ["default", "payment-service"],
# "token_type": "service",
# "token_ttl": 3600
# },
# "request": {
# "id": "req-7f8a9b2c",
# "operation": "update",
# "path": "transit/encrypt/payment-data",
# "remote_address": "10.30.0.15",
# "wrap_ttl": 0
# },
# "response": {
# "data": {
# "ciphertext": "hmac-sha256:ghi789..." # Value HMAC'd, not logged raw
# }
# }
# }
#
# Note: actual secret values are HMAC'd in audit logs
# You can verify an HMAC matches a suspected value without the log revealing itThe fail-closed behaviour is a deliberate design choice. If Vault cannot write to any audit backend (all disks full, all syslog destinations unreachable), it stops processing requests entirely. This guarantees there are no unaudited secret accesses — ever. For regulated workloads, an unaudited access is worse than a denied access, because you cannot prove to a regulator what happened during the gap.
Disaster Recovery: Because Key Loss Is Permanent Data Loss
If you lose your encryption keys, you lose your data. Not "it becomes hard to access" — it is mathematically irrecoverable. This makes Vault backup and disaster recovery a Tier-0 operational requirement for any organisation using centralised secrets management.
#!/bin/bash
# /usr/local/bin/vault-backup.sh
# Automated Vault backup with integrity verification
# Runs daily; stores encrypted snapshots on separate storage
set -euo pipefail
BACKUP_DIR="/backup/vault"
DATE=$(date +%Y-%m-%d-%H%M)
BACKUP_FILE="${BACKUP_DIR}/vault-raft-${DATE}.snap"
RETENTION_DAYS=90
# Take Raft snapshot (consistent point-in-time backup)
vault operator raft snapshot save "${BACKUP_FILE}"
# Verify the snapshot is valid
SNAPSHOT_SIZE=$(stat -c%s "${BACKUP_FILE}")
if [ "${SNAPSHOT_SIZE}" -lt 1000 ]; then
echo "ERROR: Snapshot suspiciously small (${SNAPSHOT_SIZE} bytes)"
exit 1
fi
# Encrypt the snapshot with a separate key (defense in depth)
# This key is stored in the HSM, NOT in Vault itself
gpg --batch --yes --symmetric \
--cipher-algo AES256 \
--passphrase-file /etc/vault-backup/encryption-key \
--output "${BACKUP_FILE}.gpg" \
"${BACKUP_FILE}"
# Remove unencrypted snapshot
shred -u "${BACKUP_FILE}"
# Generate checksum for integrity verification
sha256sum "${BACKUP_FILE}.gpg" > "${BACKUP_FILE}.gpg.sha256"
# Copy to secondary storage location (different physical server)
rsync -az --checksum \
"${BACKUP_FILE}.gpg" "${BACKUP_FILE}.gpg.sha256" \
backup@vault-backup-ch-02.internal:/backup/vault/offsite/
# Retention: remove backups older than retention period
find "${BACKUP_DIR}" -name "*.snap.gpg" -mtime +${RETENTION_DAYS} -delete
find "${BACKUP_DIR}" -name "*.sha256" -mtime +${RETENTION_DAYS} -delete
# Log backup completion
echo "{\"timestamp\": \"$(date -Iseconds)\", \"action\": \"backup_complete\", \
\"file\": \"${BACKUP_FILE}.gpg\", \"size\": $(stat -c%s "${BACKUP_FILE}.gpg"), \
\"replicated\": true}" >> /var/log/vault/backup.log
echo "Vault backup complete: ${BACKUP_FILE}.gpg"Note that the backup encryption key is stored in the HSM, not in Vault. If Vault is the thing you are backing up, you cannot depend on Vault to protect the backup. This is a common bootstrapping mistake — teams encrypt their Vault backup with a key stored in Vault, creating a circular dependency that makes disaster recovery impossible.
Test your restore procedure quarterly. Not "verify the backup file exists" — actually restore it to a staging environment, unseal it, and verify that you can access secrets. An untested backup is not a backup; it is hope.
Monitoring Vault in Production
A secrets management system that goes down silently is a security incident waiting to happen. Applications unable to fetch credentials will either fail (visible) or fall back to cached credentials (invisible and potentially dangerous). Monitor Vault as critical infrastructure.
# Prometheus alerting rules for Vault
# /etc/prometheus/rules/vault-health.yml
groups:
- name: vault_health
interval: 30s
rules:
# Vault is sealed (cannot serve requests)
- alert: VaultSealed
expr: vault_core_unsealed == 0
for: 1m
labels:
severity: critical
team: security
annotations:
summary: "Vault is sealed — all secret operations blocked"
runbook: "Initiate unseal procedure with key holders"
# Leadership loss in HA cluster
- alert: VaultNoLeader
expr: vault_raft_leader == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Vault HA cluster has no leader"
# High rate of access denied (potential attack or misconfiguration)
- alert: VaultAccessDeniedSpike
expr: |
rate(vault_audit_log_request_failure[5m]) > 10
for: 5m
labels:
severity: warning
team: security
annotations:
summary: "High rate of Vault access denials"
description: "{{ $value }} denied requests/sec — check for credential stuffing or misconfigured policies"
# Lease expiry approaching without renewal
- alert: VaultLeaseExpiryWarning
expr: |
vault_expire_num_leases > 1000
and rate(vault_expire_num_leases[1h]) > 100
labels:
severity: warning
annotations:
summary: "Rapid lease accumulation — applications may not be renewing"
# Audit log write failures (Vault will stop serving if all backends fail)
- alert: VaultAuditBackendFailure
expr: vault_audit_log_response_failure > 0
for: 1m
labels:
severity: critical
compliance: true
annotations:
summary: "Vault audit backend write failure"
description: "If all audit backends fail, Vault stops serving requests"
# Backup age check
- alert: VaultBackupStale
expr: |
(time() - vault_last_backup_timestamp) > 86400 * 2
labels:
severity: warning
team: security
annotations:
summary: "Vault backup older than 48 hours"Swiss Infrastructure Advantages for Secrets Management
Running your secrets management infrastructure on managed Swiss infrastructure provides advantages that are specifically relevant to key management compliance:
Jurisdictional protection of key material. Encryption keys are the highest-value targets in any infrastructure. If a foreign authority can compel disclosure of your encryption keys, your entire data protection architecture collapses — every encrypted field, every secured communication, every protected backup becomes accessible. Swiss jurisdiction under the FADP provides legal protection against unilateral key disclosure orders from non-Swiss authorities. The mutual legal assistance process required for cross-border key requests provides time, transparency, and legal review that does not exist under frameworks like the CLOUD Act.
Physical HSM hosting. Network HSMs located in Swiss data centres provide hardware-backed key protection where the HSM hardware is physically secured in a jurisdiction with strong data protection laws. The physical security of the data centre (biometric access, 24/7 monitoring, visitor logging) becomes part of your key management security posture. On dedicated Swiss servers, you can install PCIe HSM cards directly in your server chassis, eliminating network transport of key material entirely.
No shared tenancy for Vault storage. On dedicated hardware, Vault's Raft storage sits on disks that are physically yours — not shared storage volumes where a noisy neighbour's I/O patterns could theoretically leak information through side channels. For the highest security classifications, physical isolation of the storage layer is a requirement, not a preference.
Regulatory alignment. Swiss financial regulators (FINMA) have clear expectations for key management in supervised entities. Your compliance documentation can reference specific Swiss regulatory frameworks, Swiss data centre certifications, and Swiss HSM providers — creating a coherent compliance narrative that does not require explaining cross-border complexities. For EU-regulated clients, the Swiss adequacy decision means your key management infrastructure is in a jurisdiction that the EU recognises as providing adequate data protection.
Common Secrets Management Failures
Based on real-world incidents and audit findings, these are the patterns that consistently cause compliance problems:
Secrets in environment variables treated as secure. Environment variables are visible to anyone who can run ps auxe, read /proc/[pid]/environ, or access your container orchestration API. They appear in crash dumps, debugging tools, and process listings. They are not encrypted, not access-controlled, and not audited. Environment variables are a delivery mechanism for injecting secrets into applications at startup — they are not a storage mechanism. The secret should come from Vault; the environment variable should contain a Vault token or AppRole credential, not the actual secret.
Shared service accounts instead of per-application credentials. When three microservices share the same database credential, you cannot tell which service performed a given database operation. Your audit trail shows "the shared account did something" — which is useless for incident response and actively harmful for compliance reporting. Dynamic per-application credentials from Vault solve this completely: each service gets its own unique credential, every database operation is attributable to a specific service instance.
Key rotation that rotates the key but not the data. Generating a new key version is step one. If existing ciphertext is never rewrapped with the new key, the old key version must remain active indefinitely, defeating the purpose of rotation. After every key rotation, schedule a data rewrap job and verify its completion before declaring the rotation complete. Without the rewrap, you have key accumulation, not key rotation.
No separation between secret management and secret usage. The team that manages Vault should not be the same team that writes application code using Vault-managed secrets. The people who define policies should not be the people whose access those policies control. Separation of duties is not bureaucracy — it prevents a single compromised individual from both accessing secrets and suppressing the evidence of that access.
Backup key stored in the same system as the backed-up data. If your Vault backup encryption key is stored in Vault, and Vault fails catastrophically, you cannot decrypt your backup to restore Vault. The backup encryption key should be stored in an HSM, in a physical safe, or in a completely separate key management system. One team we worked with discovered this circular dependency during a disaster recovery test — and they were fortunate it was a test.
Implementation Roadmap
Implementing production-grade secrets management is a multi-phase project. Here is a pragmatic sequence that delivers compliance value incrementally without requiring a big-bang migration:
• Week 1-2: Deploy a single-node Vault server on dedicated Swiss infrastructure with Raft storage. Enable audit logging from day one. Configure the Transit secrets engine and migrate your most sensitive encryption operations (payment data, customer PII) to Vault Transit. This immediately removes encryption keys from your application servers.
• Week 3-4: Configure the database secrets engine for your primary PostgreSQL and MySQL instances. Start with read-only credentials for non-critical services. Verify that dynamic credential generation and lease renewal work correctly under load. This is the highest-risk migration — test thoroughly in staging first.
• Month 2: Expand to a three-node Vault HA cluster with Raft consensus. Migrate remaining services to dynamic database credentials. Implement AppRole authentication for all application services with response-wrapped secret ID delivery. Set up automated key rotation and data rewrap jobs.
• Month 3: Integrate HSM for auto-unseal and optionally for Transit key storage. Implement comprehensive monitoring and alerting. Run a disaster recovery test with full restore from backup. Document the end-to-end key management lifecycle for compliance reporting.
• Month 4: Implement PKI secrets engine for automated TLS certificate management. Migrate static API keys and webhook secrets to Vault KV with enforced rotation policies. Conduct a mock audit exercise to verify that all compliance evidence is available and coherent.
Each phase is independently valuable. Even completing just weeks 1-2 — centralised encryption keys in Vault Transit — eliminates the most common audit finding: encryption keys stored alongside the data they protect.
The Honest Assessment
HashiCorp Vault is operationally complex. It requires dedicated infrastructure, careful networking, security-conscious configuration, ongoing monitoring, and a team that understands both the cryptographic concepts and the operational realities. A misconfigured Vault deployment can be worse than no secrets management — it centralises all your secrets in a single target while providing a false sense of security.
The unseal ceremony, even with HSM auto-unseal, requires operational procedures that must be tested. Vault cluster upgrades require careful planning. The Raft storage backend, while simpler than Consul, still requires understanding of distributed consensus behaviour during network partitions. And the application integration work — migrating from static credentials to dynamic secrets — touches every service in your platform.
But the alternative is indefensible in a regulatory context. Static credentials that never rotate, encryption keys stored in config files, no audit trail of key access, no key lifecycle management, no automated rotation — these are not theoretical risks. They are findings that auditors document, regulators cite, and incident response teams discover at the worst possible moment.
The infrastructure foundation matters. Swiss managed infrastructure under the FADP provides the jurisdictional clarity that ensures your key material is protected not just technically but legally. Dedicated servers give you the hardware control for HSM integration, memory locking, and physical storage isolation. And the combination — Vault on Swiss dedicated hardware with HSM-backed key protection — gives you an answer to the auditor's question that is both technically rigorous and legally sound: your secrets are managed by auditable, automated infrastructure, protected by tamper-resistant hardware, governed by Swiss law, and every access is recorded.
That is not just a compliance checkbox. That is infrastructure built to earn trust.