This is not the latest version of Linkerd!

This documentation is for an older version of Linkerd. You may want the Linkerd 2.12 (current) documentation instead.

Manually Rotating Control Plane TLS Credentials

Linkerd’s automatic mTLS feature uses a set of TLS credentials to generate TLS certificates for proxies: a trust anchor, and an issuer certificate and private key. The trust anchor has a limited period of validity: 365 days if generated by linkerd install, or a customized value if generated manually.

Thus, for clusters that are expected to outlive this lifetime, you must manually rotate the trust anchor. In this document, we describe how to accomplish this without downtime.

Independent of the trust anchor, the issuer certificate and key pair can also expire (though it is possible to use cert-manager to set up automatic rotation. This document also covers how to rotate the issuer certificate and key pair without downtime.

Prerequisites

These instructions use the step and jq CLI tools.

Understanding the current state of your system

Begin by running:

linkerd check --proxy

If your configuration is valid and your credentials are not expiring soon, you should see output similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

However, if you see a message warning you that your trust anchor (“trust root”) or issuer certificates are expiring soon, then you must rotate them.

Note that this document only applies if the trust root and issuer certificate are currently valid. If your trust anchor or issuer certificate have expired, please follow the Replacing Expired Certificates Guide instead.

For example, if your issuer certificate has expired, you will see a message similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
× issuer cert is within its validity period
issuer certificate is not valid anymore. Expired on 2019-12-19T09:02:01Z
see https://linkerd.io/checks/#l5d-identity-issuer-cert-is-time-valid for hints

If your trust anchor has expired, you will see a message similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
× trust roots are within their validity period
Invalid roots:
* 79461543992952791393769540277800684467 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T09:11:30Z
see https://linkerd.io/checks/#l5d-identity-roots-are-time-valid  for hints

Rotating the trust anchor

Rotating the trust anchor without downtime is a multi-step process: you must generate a new trust anchor, bundle it with the old one, rotate the issuer certificate and key pair, and finally remove the old trust anchor from the bundle. If you simply need to rotate the issuer certificate and key pair, you can skip directly to Rotating the identity issuer certificate and ignore the trust anchor rotation steps.

Generate a new trust anchor

First, generate a new trust anchor certificate and private key:

step certificate create root.linkerd.cluster.local ca-new.crt ca-new.key --profile root-ca --no-password --insecure

Note that we use --no-password --insecure to avoid encrypting these files with a passphrase. Store the private key somewhere secure so that it can be used in the future to generate new issuer certificates.

Bundle your original trust anchor with the new one

Next, we need to bundle the trust anchor currently used by Linkerd together with the new anchor. The following command uses kubectl to fetch the Linkerd config, jq/yq to extract the current trust anchor, and step to combine it with the newly generated trust anchor:

kubectl -n linkerd get cm linkerd-config -o=jsonpath='{.data.values}' \
  | yq e .identityTrustAnchorsPEM - > original-trust.crt

step certificate bundle ca-new.crt original-trust.crt bundle.crt
rm original-trust.crt

Deploying the new bundle to Linkerd

At this point you can use the linkerd upgrade command to instruct Linkerd to work with the new trust bundle:

linkerd upgrade --identity-trust-anchors-file=./bundle.crt | kubectl apply -f -

or you can also use the helm upgrade command:

helm upgrade linkerd2 --set-file identityTrustAnchorsPEM=./bundle.crt

This will restart the proxies in the Linkerd control plane, and they will be reconfigured with the new trust anchor.

Finally, you must restart the proxy for all injected workloads in your cluster. For example, doing that for the emojivoto namespace would look like:

kubectl -n emojivoto rollout restart deploy

Now you can run the check command to ensure that everything is ok:

linkerd check --proxy

You might have to wait a few moments until all the pods have been restarted and are configured with the correct trust anchor. Meanwhile you might observe warnings:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2019-12-19T09:51:19Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
‼ data plane proxies certificate match CA
    Some pods do not have the current trust bundle and must be restarted:
        * emojivoto/emoji-d8d7d9c6b-8qwfx
        * emojivoto/vote-bot-588499c9f6-zpwz6
        * emojivoto/voting-8599548fdc-6v64k
        * emojivoto/web-67c7599f6d-xx98n
        * linkerd/linkerd-sp-validator-75f9d96dc-rch4x
        * linkerd/linkerd-tap-68d8bbf64-mpzgb
        * linkerd/linkerd-web-849f74b7c6-qlhwc
    see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

When the rollout completes, your check command should stop warning you that pods need to be restarted. It may still warn you, however, that your issuer certificate is about to expire soon:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2019-12-19T09:51:19Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

Rotating the identity issuer certificate

To rotate the issuer certificate and key pair, first generate a new pair:

step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca-new.crt --ca-key ca-new.key

Provided that the trust anchor has not expired and that, if recently rotated, all proxies have been updated to include a working trust anchor (as outlined in the previous section) it is now safe to rotate the identity issuer certificate by using the upgrade command again:

linkerd upgrade --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key | kubectl apply -f -

exp=$(cat ca-new.crt | openssl x509 -noout -dates | grep "notAfter" | sed -e 's/notAfter=\(.*\)$/"\1"/' | TZ='GMT' xargs -I{} date -d {} +"%Y-%m-%dT%H:%M:%SZ")

helm upgrade linkerd2
  --set-file identity.issuer.tls.crtPEM=./issuer-new.crt
  --set-file identity.issuer.tls.keyPEM=./issuer-new.key
  --set identity.issuer.crtExpiry=$exp

At this point Linkerd’s identity control plane service should detect the change of the secret and automatically update its issuer certificates.

To ensure this has happened, you can check for the specific Kubernetes event:

kubectl get events --field-selector reason=IssuerUpdated -n linkerd

LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
9s          Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity issuer

Restart the proxy for all injected workloads in your cluster to ensure that their proxies pick up certificates issued by the new issuer:

kubectl -n emojivoto rollout restart deploy

Run the check command to make sure that everything is going as expected:

linkerd check --proxy

You should see output without any certificate expiration warnings (unless an expired trust anchor still needs to be removed):

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

Removing the old trust anchor

We can now remove the old trust anchor from the trust bundle we created earlier. The upgrade command can do that for the Linkerd components:

linkerd upgrade  --identity-trust-anchors-file=./ca-new.crt  | kubectl apply -f -

helm upgrade linkerd2 --set-file --set-file identityTrustAnchorsPEM=./ca-new.crt

Note that the ./ca-new.crt file is the same trust anchor you created at the start of this process. Additionally, you can use the rollout restart command to bring the configuration of your other injected resources up to date:

kubectl -n emojivoto rollout restart deploy
linkerd check --proxy

Finally the output of the check command should not produce any warnings or errors:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

Congratulations, you have rotated your trust anchor! 🎉