Recently I've been thinking about how I could distribute secrets to my NixOS machines in a... relatively... decent way.
At the same time, I've been wanting to move to using an SSH CA to issue myself credentials, rather than a variety of SSH public keys.
In any case, my Vault setup ends up looking like this:
- Vault instance, running on Google Cloud Run with autounsealing from Cloud HSM, backed by Cloud Storage
- Vault configuration using Terranix (Terraform, but with the config in Nix)
- Vault App ID credentials on each machine, with Vault Agent used to automatically auth
tokendfor issuing service credentials
secretsmgrfor managing SSH CA host certificates and ACME certificates
accessfor issuing SSH CA user certificates
Let's go over these one at a time from a relatively low-level perspective - and I'll describe in more detail the mechanics of SSH CAs in a separate post. If you're interested in some good docs on how SSH CAs work, written by some colleagues of mine - then I refer you to those instead, since I'll primarily be focusing on my specific setup with Vault rather than a more general introduction.
Vault instance on GCP
First off: why on GCP when I have a bunch of physical boxen I'm managing?
The reason is simple: if I have a lot riding on it, I'd rather that it doesn't have the same failure domain as the stuff I'm hosting. It makes it much easier to recover if I don't have to rely on having my own infrastructure up in order to recover it.
On the other hand... it does mean I'm putting my root of trust "out of my hands". I'll take that risk.
Broadly speaking, my setup roughly mirrors Kelsey Hightower's Serverless Vault with Cloud Run - although I build the Docker container using Nix.
It's a relatively neat setup, although... it turns out to be expensive. Maybe I'll move it to Oracle Cloud's free tier running on one of their ARM64 instances.
One important thing to notice is that I install the
vault-acme secret engine for
issuing SSL certificates using ACME.
This allows me to just store the Let's Encrypt stuff within Vault and not have to distribute my DNS server's credentials to each individual server. I could run a separate service to do this, but it's super convenient to just have Vault do it, since everything already authenticates to it.
I use Terranix to manage the Vault configuration - this is because I have all my server's configs in my repo, so I can actually introspect the NixOS configuration to determine how I want to build the Vault config.
I have a helper script called
terraform that acts like the normal Terraform
binary after having compiled my Vault configuration, so I can run
./terraform apply and have it just work the way you'd expect. At present it requires GCP
credentials to be issued separately, using gcloud, since I store my Terraform
state in a GCS bucket, but I'm hoping to instead grant access to this using a
Vault-managed GCP service account instead (still with the ability to use my
"normal" Google account though, if needed, because obviously I need to be able
to use it to fix Vault if Vault is broken...)
In particular, I generate identities per server defined in the config, and provide myself some useful hooks to make "app" policy configuration more easy.
What do I mean by an "app"
My configuration basically defines an app as a separate Unix user; if you are
running as a Linux user named
fooservice on server
barserver (and the Vault
configuration says that
fooservice is intended to exist on that server), then
tokend will issue you a Vault token with the policies
server/barserver/app/fooservice, if those policies exist.
This is super useful: for instance, in the main case apps are only deployed on
one host, and if I'm moving it around then it makes sense for it to have access
to the same secrets. I use Pomerium as an authenticating
proxy, and so there's an
app/pomerium policy which grants access to secrets
However sometimes there are users which are deployed on more than one machine -
gitlab-runner - and that user should only get access to secrets on
one specific host. I use this concept for granting access to
a server called
clouvider-lon01 to be able to deploy to this blog! It has
to get an OAuth token to a specific GCP service account with permission to
deploy to Firebase Hosting via the
policy, but the
gitlab-runner user anywhere else is not permitted to get
access to this secret.
There are some secrets which aren't super secret and should be generally
accessible by users on servers, even if they don't have their own Vault token.
tokend checks to see if the user talking to it (via a Unix domain socket) is
a normal user rather than a service user, and if so will issue a token with the
server-user policy instead.
This token really just has a credential to get access to my Nix binary cache on Google Cloud Storage, so it's not super confidential. There aren't really many instances where this is useful, and in general on "client" devices I expect to authenticate to Vault and get a more fully-fledged token as myself.
It doesn't have access to, for instance, issue SSH user certificates. That power is restricted to "real" authenticated users who have authenticated directly with Vault.
Servers are also permitted to have server-wide secrets. This is mostly just
secretsmgr at the moment - arguably this could be its own app.
By default, servers have
kv/server/$HOSTNAME, and to issue ACME certificates, and the Nix binary
cache credentials. They also have the power to issue subtokens with
lesser-power than themselves.
...how about as a diagram?
The description of the above might be a little confusing in terms of the Vault policy hierarchy, so here's an example:
- Vault issues the Vault Agent on
clouvider-lon01a token. This token includes the Vault policies
app/deployer. The app policies (
app/deployer) are attached because the server configuration in the repository states that those two applications are intended to be deployed on that server.
clouvider-lon01uses the token held by the Vault Agent directly to refresh any TLS or SSH certificates needed by the server.
clouvider-lon01has no token of its own, but uses the one held by the Vault Agent to issue app- or user-specific sub-tokens, with a subset of the policies attached to the initial token.
tokend, which issues it a subtoken with just the
clouvider-lon01also talks to
tokend, but it gets a different subtoken which instead has the
- My own personal user account,
lukegb, can also talk to
tokendto get a subtoken with the
server-userpolicies. This token is very limited compared to a standard
user-policy token, which needs to be issued by using the Vault API directly to authenticate as a user based on some OpenID Connect credentials.
Vault App ID credentials
I use the "App ID" mode in Vault to provision secrets to servers; when setting
a machine up (a process I have not yet automated), I run
which revokes all existing secret IDs for that host and dumps out a Vault
token, which can
be used one time only to get the secret ID for that host.
script installed on every machine which will then install the secret for me.
Future work in this space for me is binding the secret to the TPM (e.g. using mTLS auth) so I don't have to stick the secret ID on disk... but then again I'm not a multinational corporation, and my secrets aren't worth that much.
The Vault Agent is a daemon that serves a number of purposes:
- It can act as a proxy which keeps an internal Vault token and automatically refreshes it, and then attaches it to every request that it proxies to Vault as received on a Unix socket (or a TCP socket... but I don't use it like that)
- It can use templates to write secrets to disk (with drawbacks, hence the
- Probably some other functionality I don't really use
In my setup, only a few things have direct access to the Vault Agent socket,
and in future I might get rid of it from my setup entirely.
secretsmgr have access, and that's pretty much it. This is because the powers
that its Vault token gets are a combination of all the policies granted to the
machine, including all the apps running on it, so any app with access to its
Unix socket effectively gets all the secrets shared to anything on the server.
The secrets I use it to write to disk are strictly the plain KV type, rather than anything more sophisticated, but I do use some relatively complicated Polkit rules to allow it to reload/restart services when those secrets change.
The user-based authentication I mentioned above (with the app policies and the
server-user) policy is powered by
which is a daemon that listens on a Unix socket and proxies requests through
the local Vault Agent, with a token issued that has a subset of the powers of
the original server-wide token.
The ACLs on talking to
tokend are much more permissive than those for talking
directly to the Vault agent, because the token you get depends on your identity.
secretsmgr exists to solve some problems I was having with getting Vault
Agent to write secrets that require more complex Vault requests, like the TLS
certificates using ACME (which have ratelimits imposed by Let's Encrypt!), and
SSH host certificates.
It's a pretty simple binary which runs using a systemd timer unit, starts up, checks the remaining lifetime of the certificates it's responsible for, and then reissues them if required.
Similar to the Vault Agent above, I use some Polkit rules to allow it to restart the ACME certificate consumers (usually nginx or pomerium), and sshd.
access checks to see if there's currently an active Vault token. If not, then
it launches the
vault login flow which in my case asks me to log in with my
Google account. If that succeeds, or if I already had a token, then it
generates a new ED25519 private key and asks Vault to sign it with a lifetime
of about 24 hours, and then inserts it into the SSH agent. This means the key
never has to hit disk, since it can just reside in the SSH agent.
The token that this flow issues is a
user token (i.e. not a
token, nor an
admin token), which has permission to look at some specific
secrets related to that user, things which are generally shared -- like those
Nix binary cache credentials, but doesn't have general access to administrate
Vault. I issue
admin tokens with a lifetime of 1h when I need them, and at
some point will try to scope them down further -- although since I use them to
deploy all the policies there is a limit to what I can feasibly do.