Using certificates with SSH can be useful when a developer loses their key and needs to regenerate it. We won’t have to enter all the servers to install the new key. We will simply generate another one, sign it with the CA, add the old one to the revocation list, and distribute the revocation list to the servers.

OpenSSH supports certificate authentication of users and hosts using a new, minimal OpenSSH certificate format (not X.509) since version 5.4 :

Add support for certificate authentication of users and hosts using a new, minimal OpenSSH certificate format (not X.509).
Certificates contain a public key, identity information and some validity constraints and are signed with a standard SSH public key using ssh-keygen(1).
CA keys may be marked as trusted in authorized_keys or via a TrustedUserCAKeys option in sshd_config(5) (for user authentication),
or in known_hosts (for host authentication).

We create a user to manage the CA on the CA server:

We create the CA:

We copy the ca.pub to the ssh servers:

We edit the config on the ssh server to allow authentication using pubkey-certificate:

TrustedUserCAKeys /etc/ssh/ca.pub

Restart the service:

Generate the client’s certificate in the CA based on its pubkey, the syntax is as follows:

An example of a valid certificate for one week for the user kr0m would be:

Signed user key id_rsa-cert.pub: id "kr0m" serial 1 for root valid from 2019-05-20T22:41:00 to 2019-05-27T22:42:20

NOTE: A valid time interval can be from now until X time or a specific range, from X to Y separated by “;”.

We can check the certificate data with:

id_rsa-cert.pub:
 Type: ssh-rsa-cert-v01@openssh.com user certificate
 Public key: RSA-CERT SHA256:oV14spSAEcsEWiXREUYvBgVEyEumRMBQGtwbZXXF6ZE
 Signing CA: RSA SHA256:xhAgl+pmJ4Y5jZgAM1hGCFgIwbidV7xRKphbxtfDvqE
 Key ID: "kr0m"
 Serial: 1
 Valid: from 2019-05-20T22:41:00 to 2019-05-27T22:42:20
 Principals: 
 root
 Critical Options: (none)
 Extensions: 
 permit-X11-forwarding
 permit-agent-forwarding
 permit-port-forwarding
 permit-pty
 permit-user-rc

Pass the certificate to the client:

Try to connect:

In the server logs we can see the access record:

May 20 22:46:04 rsaserver sshd[715]: Accepted publickey for root from x.x.x.x port 50102 ssh2: RSA-CERT ID kr0m (serial 1) CA RSA SHA256:xhAgl+pmJ4Y5jZgAM1hGCFgIwbidV7xRKphbxtfDvqE

If we try to connect after a week, we would see in the logs:

May 20 23:00:33 rsaserver sshd[1619]: error: Certificate invalid: expired

Authentication by certificates allows us to define “principals” which act as access tokens, so we can define access zones. This is useful for restricting access to certain servers.

Configure the ssh servers:

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Define the principals:

We restart the service:

The idea is to have a generic principal that sysadmins (root-everywhere) will use and a more specific one for users.
To access this server, our certificate must have been issued with the zone-webservers or root-everywhere principal.
If we try to connect with the certificate without the principal, the following error will appear:

May 20 23:02:34 rsaserver sshd[935]: error: Certificate does not contain an authorized principal

We regenerate the certificate in the CA indicating the principal:

We update the certificate on the client and connect:

We can make the principal dynamic by running a script/binary on the ssh server to obtain it, for example by consulting an active directory:

AuthorizedPrincipalsCommand /usr/bin/adquery user –P %u
AuthorizedPrincipalsCommandUser root

The variables supported in the principal configuration are:

AuthorizedPrincipalsCommand accepts the tokens %%, %F, %f, %h, %i, %K, %k, %s, %T, %t, %U, %u.
AuthorizedPrincipalsFile accepts the tokens %%, %h, %U, %u.

%% A literal `%'.
%F The fingerprint of the CA key.
%f The fingerprint of the key or certificate.
%h The home directory of the user.
%i The key ID in the certificate.
%K The base64-encoded CA key.
%k The base64-encoded key or certificate for authentication.
%s The serial number of the certificate.
%T The type of the CA key.
%t The key or certificate type.
%U The numeric user ID of the target user.
%u The username.

We can check through tcpdump that there is no connection between the ssh server and the CA in the login process:
In the CA:

We make the connection to the ssh server from the client:

In the tcpdump, there is no traffic, which is good because the CA will not be a PoF if we are attacked by brute force on the ssh service.

But on the other hand, to revoke certificates, we will have to distribute the revoked keys file to each ssh server independently. This file can be distributed via pssh, ansible, puppet, chef, cron…

In my case, the best option is to schedule a wget of the revoked keys file on each server using crontab, and if necessary, apply the revocation immediately by executing the wget manually through pssh.

To revoke a certificate, we must configure the ssh servers to consult the list of revoked keys and revoke it on the CA.
On the ssh server:

Restart the service:

On the CA, we revoke the user’s key:

We can check if the key is on the list of revoked keys:

id_rsa.pub (kr0m@DirtyCow): REVOKED

NOTE: If we revoke a key by accident, we will have no choice but to regenerate the ssh keys and the associated certificate.

Distribute the revoked keys file to all ssh servers:

Check that the client can no longer connect:

In the logs, we will see:

May 20 23:49:57 rsaserver sshd[1306]: error: Authentication key RSA SHA256:oV14spSAEcsEWiXREUYvBgVEyEumRMBQGtwbZXXF6ZE revoked by file /etc/ssh/revoked_keys

NOTE: Revoked keys are independent of whether we use certs or pubkeys. Without enabling certs, we can use key banning.

It is recommended to generate certificates with an expiration date of X and also revoke keys if a developer leaves the project. This way, we only have to maintain the fired developers in the last X days on the revocation list.

If we want to make it possible to log in only through pubkeys-certs:

PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no

We restart the service:

NOTE: There is no way to configure it only with certificates and without pubkey authentication.

The key concepts that we must extract from this article are:

• Distribute the CA pubkey to ssh servers.
• Distribute the list of revoked keys when a developer leaves the project.
• Each department must have a different principal, so we can use the same CA for all certificates of all users.
• Schedule as root a script that checks the content of the authorized_keys of all users on the system. This way, if the server were compromised, we would detect it quickly and easily. This script could also add the keys found to the revoked keys file in case a developer tries to gain access through authorized_keys.

The CA server is a critical security point. An attacker with access to the CA could generate a certificate with which to access all servers in our infrastructure. The only way to detect these accesses would be to review the logs of the final servers.