Rotating webhooks certificates

Linkerd uses the Kubernetes admission webhooks and extension API server to implement some of its core features like automatic proxy injection and service profiles validation.

Also, the viz extension uses a webhook to make pods tappable, as does the jaeger extension to turn on tracing on pods.

To secure the connections between the Kubernetes API server and the webhooks, all the webhooks are TLS-enabled. The x509 certificates used by these webhooks are issued by the self-signed CA certificates embedded in the webhooks configuration.

By default, these certificates have a validity period of 365 days. They are stored in the following secrets:

In the linkerd namespace: linkerd-policy-validator-k8s-tls, linkerd-proxy-injector-k8s-tls and linkerd-sp-validator-k8s-tls
In the linkerd-viz namespace: tap-injector-k8s-tls
In the linkerd-jaeger namespace: jaeger-injector-k8s-tls

The rest of this documentation provides instructions on how to renew these certificates.

Renewing the webhook certificates

To check the validity of all the TLS secrets (using step):

# assuming you have viz and jaeger installed, otherwise trim down these arrays
# accordingly
SECRETS=("linkerd-policy-validator-k8s-tls" "linkerd-proxy-injector-k8s-tls" "linkerd-sp-validator-k8s-tls" "tap-injector-k8s-tls" "jaeger-injector-k8s-tls")
NS=("linkerd" "linkerd" "linkerd-viz" "linkerd-jaeger")
for idx in "${!SECRETS[@]}"; do \
  kubectl -n "${NS[$idx]}" get secret "${SECRETS[$idx]}" -ojsonpath='{.data.tls\.crt}' | \
    base64 --decode - | \
    step certificate inspect - | \
    grep -iA2 validity; \
done

Manually delete these secrets and use upgrade/install to recreate them:

for idx in "${!SECRETS[@]}"; do \
  kubectl -n "${NS[$idx]}" delete secret "${SECRETS[$idx]}"; \
done

linkerd upgrade | kubectl apply -f -
linkerd viz install | kubectl apply -f -
linkerd jaeger install | kubectl apply -f -

The above command will recreate the secrets without restarting Linkerd.

Note

For Helm users, use the helm upgrade command to recreate the deleted secrets.

If you render the helm charts externally and apply them with kubectl apply (e.g. in a CI/CD pipeline), you do not need to delete the secrets manually, as they wil be overwritten by a new cert and key generated by the helm chart.

Confirm that the secrets are recreated with new certificates:

for idx in "${!SECRETS[@]}"; do \
  kubectl -n "${NS[$idx]}" get secret "${SECRETS[$idx]}" -ojsonpath='{.data.crt\.pem}' | \
    base64 --decode - | \
    step certificate inspect - | \
    grep -iA2 validity; \
done

Ensure that Linkerd remains healthy:

linkerd check

Restarting the pods that implement the webhooks and API services is usually not necessary. But if the cluster is large, or has a high pod churn, it may be advisable to restart the pods manually, to avoid cascading failures.

If you observe certificate expiry errors or mismatched CA certs, restart their pods with:

kubectl -n linkerd rollout restart deploy \
  linkerd-proxy-injector \
  linkerd-sp-validator \

kubectl -n linkerd-viz rollout restart deploy tap tap-injector
kubectl -n linkerd-jaeger rollout restart deploy jaeger-injector