Slack
Discourse

Control Plane Debug Endpoints

All of the control plane components expose runtime profiling information through the path /debug/pprof, using Go’s pprof package. This endpoint is disabled by default but can be enabled to gather profiling data.

You can consume the provided data with go tool pprof to generate output in many formats (PDF, DOT, PNG, etc).

The following diagnostics are provided (a summary with links is provided at /debug/pprof):

  • allocs: A sampling of all past memory allocations
  • block: Stack traces that led to blocking on synchronization primitives
  • cmdline: The command line invocation of the current program
  • goroutine: Stack traces of all current goroutines
  • heap: A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.
  • mutex: Stack traces of holders of contended mutexes
  • profile: CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.
  • threadcreate: Stack traces that led to the creation of new OS threads
  • trace: A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.

Example Usage

The pprof endpoint can be enabled by setting the --set enablePprof=true flag when installing or upgrading Linkerd or by setting the enablePprof=true Helm value.

This data is served over the admin-http port. To find this port, you can examine the pod’s yaml, or for the identity pod for example, issue a command like so:

kubectl -n linkerd get po \
    $(kubectl -n linkerd get pod -l linkerd.io/control-plane-component=identity \
        -o jsonpath='{.items[0].metadata.name}') \
    -o=jsonpath='{.spec.containers[*].ports[?(@.name=="admin-http")].containerPort}'

Then use the kubectl port-forward command to access that port from outside the cluster (in this example the port is 9990):

kubectl -n linkerd port-forward \
    $(kubectl -n linkerd get pod -l linkerd.io/control-plane-component=identity \
        -o jsonpath='{.items[0].metadata.name}') \
    9990

It is now possible to use go tool to inspect this data. For example to generate a graph in a PDF file describing memory allocations:

go tool pprof -seconds 5 -pdf http://localhost:9990/debug/pprof/allocs