Slack
Discourse
This is not the latest version of Linkerd!
This documentation is for an older version of Linkerd. You may want the Linkerd 2.12 (current) documentation instead.

Automatic mTLS

By default, Linkerd automatically enables mutually-authenticated Transport Layer Security (mTLS) for all TCP traffic between meshed pods. This means that Linkerd adds authenticated, encrypted communication to your application with no extra work on your part. (And because the Linkerd control plane also runs on the data plane, this means that communication between Linkerd’s control plane components are also automatically secured via mTLS.)

See Caveats and future work below for some details.

What is mTLS?

A full definition of mTLS is outside the scope of this doc. For an overview of what mTLS is and how it works in Kuberentes clusters, we suggest reading through A Kubernetes engineer’s guide to mTLS.

Which traffic can Linkerd automatically mTLS?

Linkerd transparently applies mTLS to all TCP communication between meshed pods. However, there are still ways in which you may still have non-mTLS traffic in your system, including:

  • Traffic to or from non-meshed pods (e.g. Kubernetes healthchecks)
  • Traffic on ports that were marked as skip ports, which bypass the proxy entirely.

You can verify which traffic is mTLS’d in a variety of ways. External systems such as Buoyant Cloud can also automatically generate reports of TLS traffic patterns on your cluster.

Operational concerns

Linkerd’s mTLS requires some preparation for production use, especially for long-lived clusters or clusters that expect to have cross-cluster traffic.

The trust anchor generated by the default linkerd install CLI command expires after 365 days. After that, it must be manually rotated—a non-trivial task. Alternatively, you can provide the trust anchor yourself and control the expiration date, e.g. setting it to 10 years rather than one year.

Kubernetes clusters that make use of Linkerd’s multi-cluster communication must share a trust anchor. Thus, the default linkerd install setup will not work for this situation and you must provide an explicit trust anchor.

Similarly, the default cluster issuer certificate and key expire after a year. These must be rotated before they expire. Alternatively, you can set up automatic rotation with cert-manager.

External systems such as Buoyant Cloud can be used to monitor cluster credentials and to send reminders if they are close to expiration.

How does Linkerd’s mTLS implementation work?

The Linkerd control plane contains a certificate authority (CA) called identity. This CA issues TLS certificates to each Linkerd data plane proxy. Each certificate is bound to the Kubernetes ServiceAccount identity of the containing pod. These TLS certificates expire after 24 hours and are automatically rotated. The proxies use these certificates to encrypt and authenticate TCP traffic to other proxies.

On the control plane side, Linkerd maintains a set of credentials in the cluster: a trust anchor, and an issuer certificate and private key. These credentials can be generated by Linkerd during install time, or optionally provided by an external source, e.g. Vault or cert-manager. The issuer certificate and private key are stored in a Kubernetes Secret; this Secret is placed in the linkerd namespace and can only be read by the service account used by the Linkerd control plane’s identity component.

On the data plane side, each proxy is passed the trust anchor in an environment variable. At startup, the proxy generates a private key, stored in a tmpfs emptyDir which stays in memory and never leaves the pod. The proxy connects to the control plane’s identity component, validating the connection to identity with the trust anchor, and issues a certificate signing request (CSR). The CSR contains an initial certificate with identity set to the pod’s Kubernetes ServiceAccount, and the actual service account token, so that identity can validate that the CSR is valid. After validation, the signed trust bundle is returned to the proxy, which can use it as both a client and server certificate. These certificates are scoped to 24 hours and dynamically refreshed using the same mechanism.

Finally, when a proxy receives an outbound connection from the application container within its pod, it looks up that destination with the Linkerd control plane. If it’s in the Kubernetes cluster, the control plane provides the proxy with the destination’s endpoint addresses, along with metadata including an identity name. When the proxy connects to the destination, it initiates a TLS handshake and verifies that that the destination proxy’s certificate is signed by the trust anchor and contains the expected identity.

Caveats and future work

There are a few known gaps in Linkerd’s ability to automatically encrypt and authenticate all communication in the cluster. These gaps will be fixed in future releases:

  • Linkerd does not currently enforce mTLS. Any unencrypted requests inside the mesh will be opportunistically upgraded to mTLS. Any requests originating from inside or outside the mesh will not be automatically mTLS’d by Linkerd. This will be addressed in a future Linkerd release, likely as an opt-in behavior as it may break some existing applications.

  • Ideally, the ServiceAccount token that Linkerd uses would not be shared with other potential uses of that token. In future Kubernetes releases, Kubernetes will support audience/time-bound ServiceAccount tokens, and Linkerd will use those instead.