Replacing expired certificates

If any of your TLS certs are approaching expiry and you are not relying on an external certificate management solution such as cert-manager, you can follow Manually Rotating Control Plane TLS Credentials to update them without incurring downtime. However, if any of your certificates have already expired, your mesh is already in an invalid state and any measures to avoid downtime are not guaranteed to give good results. Instead, you need to replace the expired certificates with valid certificates.

Replacing only the issuer certificate

It might be the case that your issuer certificate is expired. If this it true running linkerd check --proxy will produce output similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
× issuer cert is within its validity period
    issuer certificate is not valid anymore. Expired on 2019-12-19T09:21:08Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-is-time-valid for hints

In this situation, if you have installed Linkerd with a manually supplied trust root and you have its key, you can follow the instructions to rotate your identity issuer certificate to update your expired certificate.

Replacing the root and issuer certificates

If your root certificate is expired or you do not have its key, you need to replace both your root and issuer certificates at the same time. If your root has expired linkerd check will indicate that by outputting an error similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
× trust roots are within their validity period
    Invalid roots:
        * 272080721524060688352608293567629376512 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T10:05:31Z
    see https://linkerd.io/checks/#l5d-identity-roots-are-time-valid for hints

You can follow Generating your own mTLS root certificates to create new root and issuer certificates. Then use the linkerd upgrade command:

linkerd upgrade \
    --identity-issuer-certificate-file=./issuer-new.crt \
    --identity-issuer-key-file=./issuer-new.key \
    --identity-trust-anchors-file=./ca-new.crt \
    --force \
    | kubectl apply -f -

Usually upgrade will prevent you from using an issuer certificate that will not work with the roots your meshed pods are using. At that point we do not need this check as we are updating both the root and issuer certs at the same time. Therefore we use the --force flag to ignore this error.

Once this is done, you’ll need to explicitly restart the control plane so that everything in the control plane is configured to use the new trust anchor:

kubectl rollout restart -n linkerd deploy

If you run linkerd check --proxy before the restart is completed, you will probably see warnings about pods not having the current trust bundle:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
‼ data plane proxies certificate match CA
    Some pods do not have the current trust bundle and must be restarted:
        * linkerd/linkerd-controller-5b69fd4fcc-7skqb
        * linkerd/linkerd-destination-749df5c74-brchg
        * linkerd/linkerd-grafana-6dcf86b74b-vvxjq
        * linkerd/linkerd-prometheus-74cb4f4b69-kqtss
        * linkerd/linkerd-proxy-injector-cbd5545bd-rblq5
        * linkerd/linkerd-sp-validator-6ff949649f-gjgfl
        * linkerd/linkerd-tap-7b5bb954b6-zl9w6
        * linkerd/linkerd-web-84c555f78-v7t44
    see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

These warnings should disappear once the restart is completed. Once they do, you can use kubectl rollout restart to restart your meshed workloads to bring their configuration up to date. After that is done, linkerd check should run with no warnings or errors:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA