Skip to content

multi: add HashiCorp Vault KV v2 secret support#92

Open
Roasbeef wants to merge 2 commits into
mainfrom
vault-kv2-support
Open

multi: add HashiCorp Vault KV v2 secret support#92
Roasbeef wants to merge 2 commits into
mainfrom
vault-kv2-support

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Jul 1, 2026

Copy link
Copy Markdown
Member

In this PR, we add vault as a third secret source and target alongside the existing file and k8s backends, wired into the init-wallet, store-secret and load-secret commands. This lets an lnd deployment keep its wallet seed and password in HashiCorp Vault directly, so the secrets never need to be materialized as a Kubernetes Secret in etcd.

This supersedes #62, which was a straight port of an older lnd-fork branch written against a KV v1 Vault with a hand-rolled Kubernetes login. Our production Vault targets the KV v2 secrets engine, so that version wouldn't work as-is: a KV v2 read comes back with the values nested under data, which the KV v1 Logical().Read path doesn't unwrap.

What's here

  • KV v2 via the official helper. Reads and writes go through client.KVv2(mount), which handles the nested data/ path shaping KV v2 requires. Writes are read-modify-write merges, so storing one entry (e.g. the seed) never clobbers a sibling entry (e.g. the password) in the same secret. Using Put on a merged map (rather than Patch) keeps the required policy to just create/read/update.
  • Idempotency preserved. The errTargetExists sentinel is kept on the write path, so --error-on-existing keeps working across restarts. This matters: we never want to silently rotate a seed out from under an already-initialized wallet.
  • Kubernetes auth. Authentication uses the Kubernetes auth method via the official vault/api/auth/kubernetes login helper. The pod's projected ServiceAccount token is exchanged for a Vault token bound to a role and policy. The auth mount, role and token path are configurable; the server address comes from VAULT_ADDR or --vault.addr.
  • Docs + example + tests. A README section, an example-init-wallet-vault.sh startup script, and a vault_test.go suite (a mock KV v2 + k8s-auth server covering round-trip, no-clobber merge, the overwrite guard, newline trimming, and the not-found/missing-key/empty-value error paths).

The docs/vault-integration-design.md file captures how our production Vault in lightning-infra is set up (KV v2, Kubernetes auth, VSO), the gap versus the KV v1 approach, and the companion infra wiring (an lnd policy scoped to secret/data/lnd/*, a k8s auth role, and the lnd chart init-script) that will follow in a separate lightning-infra PR.

Roasbeef added 2 commits July 1, 2026 13:05
In this commit, we add `vault` as a third secret source and target
alongside the existing `file` and `k8s` backends, wired into the
`init-wallet`, `store-secret` and `load-secret` commands. This lets an
lnd deployment keep its wallet seed and password in HashiCorp Vault
directly, so the secrets never need to be materialized as a Kubernetes
Secret in etcd.

The implementation targets the KV v2 secrets engine using the official
`vault/api` KV v2 helper: reads and writes go through
`client.KVv2(mount)`, which handles the nested `data/` path shaping that
KV v2 requires. Writes are read-modify-write merges so that storing one
entry (e.g. the seed) never clobbers a sibling entry (e.g. the password)
in the same secret. The `errTargetExists` sentinel is preserved on the
write path so the existing idempotency contract (`--error-on-existing`)
keeps working across restarts, which is important to make sure we never
silently rotate a seed out from under an initialized wallet.

Authentication uses the Kubernetes auth method via the official
`vault/api/auth/kubernetes` login helper. The pod's projected
ServiceAccount token is exchanged for a Vault token bound to a role and
policy. The auth mount, role and token path are all configurable; the
Vault server address is taken from the `VAULT_ADDR` environment variable
or the `--vault.addr` flag.

A companion `example-init-wallet-vault.sh` and a README section round out
the change.
In this commit, we add the design doc that motivates the KV v2 Vault
variant: how our production Vault in lightning-infra is actually set up
(KV v2, Kubernetes auth, VSO), the gap versus the earlier KV v1 approach,
the locked design decisions, and the companion infra wiring needed to
roll it out.

@calvinrzachman calvinrzachman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vault support 🔒 I tested this in a cluster against our lnd chart and the happy path works end to end. The seed and password make it to Vault, wallet unlocks, and the node comes online, with no Kubernetes wallet secret created.

A couple of small hardening items where the Vault path deviates from the k8s backend predecessor. Neither fires in our current deployment (single replica, write-once text secrets) and both are small fixes, so they're more for k8s backend parity rather than anything blocking.

Comment thread vault.go

logger.Infof("Attempting to write entry %s of secret %s in vault",
opts.SecretKeyName, opts.SecretPath)
if _, err := kv.Put(ctx, opts.SecretPath, data); err != nil {

@calvinrzachman calvinrzachman Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k8s fails loud on a concurrent write, since Update carries a resourceVersion. Vault's Put here is unconditional, so two processes writing different entries of the same secret can overwrite one another and lose an entry with no error. Ran a test with two concurrent stores against the same path, and the Vault path silently dropped a key, while the same race on --target=k8s failed noticeably. A check-and-set on the version gives Vault the same loud-on-conflict behavior:

if _, err := kv.Put(ctx, opts.SecretPath, data,
    api.WithCheckAndSet(version)); err != nil {
    return err
}

This needs two initializers writing the same path at once, which a single-replica deployment avoids, so it is not a live bug given the init script we're using. But lndinit is a general tool, and a custom init script that calls store-secret concurrently would hit it, so matching the k8s backend's behavior seems worth having.

Comment thread vault.go
opts.SecretKeyName, opts.SecretPath, errTargetExists)
}

data[opts.SecretKeyName] = content

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KV v2 stores values as JSON strings, so a non-UTF-8 value has its invalid bytes replaced with U+FFFD and cannot be recovered. As an example an admin.macaroon stored via --batch --target=vault comes back corrupted. The k8s path has --k8s.base64 for this, so we may want to match that with a --vault.base64 option (encode on write, decode on read) here. A tls.cert/key round-trips clean, so it is a binary vs text thing, not macaroon-specific.

This is not hit today, since the chart keeps only text seed and password in Vault, but it would be a prerequisite if we later move the RPC secrets into Vault too (e.g. VSO-syncing them out to consumer namespaces).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants