Spike: Investigating new Fido2-backed OpenSSH keys
I would like to secure ssh access to some servers with a second factor.
Criteria
- It shouldn’t be too easy to misconfigure, especially when it comes to accidentally locking myself out or being insecure
- It should be minimal effort during usage
- The key material should be generated on a hardware token without possibility of retrieval
- Rolling out keys to multiple servers and rotating keys should be automatable using Ansible
Secondary and Non-Goals
- It doesn’t have to be free or necessarily the cheapest hardware token available.
- I don’t need a retrieval option, I’m assuming that if I lose the key I can access the server in another way to deploy a new one.
tl;dr
- It’s relatively new but most distros’ LTS versions are supported.
- If you don’t trust NIST curves, you need to have a recent yubikey (Firmware >= 5.2.3, according to this announcement you should be fine if you bought it no earlier than 2020)
Scope
Completely ignoring the hardware second factor aspect, ssh keys fit the bill. I recently noticed that there’s two new types of keys: ecdsa-sk and ed25519-sk. According to this Stackoverflow post, they work like regular keys except they also require a Yubikey to use.
I’ve been using my Yubikey 5c for a while and like it, so this sounds intriguing.
Log
If you’re not interested in the details or my process, skip this section.
I have my Yubikey plugged in and try to create a key using ssh-keygen -t ed25519-sk -f testkey. It fails with this error:
> ssh-keygen -t ed25519-sk -f testkey
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
/usr/lib/ssh/ssh-sk-helper: error while loading shared libraries: libfido2.so.1: cannot open shared object file: No such file or directory
ssh_msg_recv: read header: Connection reset by peer
client_converse: receive: unexpected internal error
reap_helper: helper exited with non-zero exit status
Key enrollment failed: unexpected internal errorArch Linux apparently doesn’t automatically install libfido2. After running sudo pacman -Sy libfido2, it fails in another way. (At this point, I also discovered a recent Arch Linux issue about this, but luckily it seems to have already been fixed.)
> ssh-keygen -t ed25519-sk -f testkey
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Key enrollment failed: invalid format`The reporter of the Arch Linux issue linked above also mentioned community/libu2f-host and community/libu2f-server, so I tried to install them too, but there doesn’t seem to be a libu2f-host package. Installing libu2f-host didn’t change anything.
After reading the lwn post from the Arch Linux issue a bit, I know more about how it should work but still nothing about why it doesn’t.
So I tried it with ssh-keygen -t ecdsa-sk -f testkey, which works. It makes me touch the key and asks for a passphrase. I would prefer ed25519 though, so I’m looking into it a bit more.
I found this github issue, which looks like my error at first. But my verbose stderr is different:
> ssh-keygen -vvvv -t ed25519-sk -f testkey
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
debug3: start_helper: started pid=144578
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/lib/ssh/ssh-sk-helper
debug1: sshsk_enroll: provider "internal", device "(null)", application "ssh:", userid "(null)", flags 0x01, challenge len 0
debug1: sshsk_enroll: using random challenge
debug1: sk_probe: 1 device(s) detected
debug1: sk_probe: selecting sk by touch
debug1: ssh_sk_enroll: using device /dev/hidraw14
debug1: ssh_sk_enroll: fido_dev_make_cred: FIDO_ERR_INVALID_ARGUMENT
debug1: sshsk_enroll: provider "internal" failure -1
debug1: ssh-sk-helper: Enrollment failed: invalid format
debug1: main: reply len 8
debug3: ssh_msg_send: type 5
debug1: client_converse: helper returned error -4
debug3: reap_helper: pid=144578
Key enrollment failed: invalid formatI ran a few commands that were suggested to debug, and some seemed weird, but another thought came to my mind: Does my Yubikey support ed25519-sk? According to this guide, it needs firmware version 5.2.3. I installed ykman (package yubikey-manager on Arch) and ran ykman info:
> ykman info ~
WARNING: PC/SC not available. Smart card protocols will not function.
Device type: YubiKey 5C
Serial number: 10348433
Firmware version: 5.1.2
Form factor: Keychain (USB-C)
Enabled USB interfaces: FIDO, CCID
Applications
FIDO2 Disabled
OTP Disabled
FIDO U2F Enabled
OATH Enabled
YubiHSM Auth Not available
OpenPGP Enabled
PIV EnabledMeh. Let’s shelve this and focus on the spike, I’ll see if I can update the firmware later.
I copied the ecdsa-key I generated earlier to a server of mine and tried to log in, but it’s still asking me for a password. My suspicion is that the host also needs to support the -sk keys. ssh -V on the server reports OpenSSH_7.9p1, while I have OpenSSH_8.9p1 locally. According to the openssh 8.2 release notes, the feature was released in that version.
That server is running debian 10, which isn’t the current one, but this still makes me think whether I can get a recent-enough OpenSSH version on all relevant servers.
I have another server running OpenSSH_8.8p1, so I added the public key there.
For that machine, it works great:
> ssh -o IdentitiesOnly=yes -o IdentityFile=testkey myserver
Enter passphrase for key 'testkey':
Confirm user presence for key ECDSA-SK SHA256:8Z+EQ594E5FfqlZP/EaqhloBZZuEI2d64RjpkPcHYVk
User presence confirmed
________________
< sick motd bruh >
----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||I think these results give me enough information for now.
Question and sidetracks
Is there actually essential key material generated on the Yubikey that never leaves it? How does it work?
Looks good. Quoting https://lwn.net/Articles/812537/
FIDO/U2F OpenSSH keys consist of two parts: a "key handle" part stored in the private key file on disk, and a per-device private key that is unique to each FIDO/U2F token and that cannot be exported from the token hardware. These are combined by the hardware at authentication time to derive the real key that is used to sign authentication challenges.
There seems to be a way to store everything on the Yubikey so that there’s no need to keep a key file around. Keyword is “resident keys”, but I don’t need that at the moment, so I’m not looking into it right now.
My firmware version only supports ecdsa-sk, not ed25519-sk. Can I upgrade it?
Can I install a recent-enough OpenSSH version on all relevant servers?
- Current Debian ships OpenSSH 8.4, Ubuntu 20.04 LTS ships 8.2, both of which should be fine. Centos 7 ships OpenSSH 7.4, which I assume won’t work. Alpine 3.15 ships 8.4.
Reflection
- Generally, it seems to work great, even though I’d prefer ed25519.
- OpenSSH server version requirements are concerning.
- Yubikey 5c is not the cheapest, but it might be preferable for developers because of features besides Fido.
Looking at the criteria defined before the spike:
It shouldn’t be too easy to misconfigure, especially when it comes to accidentally locking myself out or being insecure Using ssh keys for this is cool, I know pretty well how to deal with them.
It should be minimal effort during usage Literally just touching the button. There’s even some potential to make it even more convenient (resident keys, ssh-add -K, no-touch-required)
The key material should be generated on a hardware token without possibility of retrieval Confirmed with information from LWN and the OpenSSH 8.2 release notes.
Rolling out keys to multiple servers and rotating keys should be automatable using Ansible Trivial with ssh keys
It doesn’t have to be free or necessarily the cheapest hardware token available.
Yubikey 5c is reasonably priced in my opinion. There might also be cheaper options, possibly with less features. The cheaper [[https://www.yubico.com/de/product/security-key-nfc-by-yubico/|Yubico Security Key NFC]] also supports Fido2 and comes at 25€.
