How can I find the Prometheus endpoint


#1

Hey, Linkerd 2.0 question: I need to integrate it’s Prometheus endpoint with datadog, where can I find the Prometheus url/endpoint to connect to?
I saw most examples point to 9990 port but when I look at the Prometheus service in the linkerd namespace I see 9090 as the port with: http://prometheus.linkerd.svc.cluster.local:9090
Is this correct (From inside the cluster)?
Also my integration needs to access: admin/metrics/prometheus
Would the corrrect url to point it to be: “http://prometheus.linkerd.svc.cluster.local:9090/admin/metrics/prometheus”?


#2

Hi @alonisser, for Datadog integration you probably want to use Prometheus’ /federate API. This will provide metrics for all Linkerd 2.0 proxies running in your Kubernetes cluster, in Prometheus metrics exposition format.

Connecting to Prometheus’ /federate endpoint

Inside Kubernetes cluster

curl -G --data-urlencode 'match[]={job="linkerd-proxy"}' http://prometheus.linkerd.svc.cluster.local:9090/federate

Outside Kubernetes cluster

kubectl -n linkerd port-forward $(kubectl -n linkerd get po --selector=linkerd.io/control-plane-component=prometheus -o jsonpath='{.items[*].metadata.name}') 9090:9090

and then:

curl -G --data-urlencode 'match[]={job="linkerd-proxy"}' "http://localhost:9090/federate"

Background

For reference, here’s a quick survey of other metrics-related endpoints in the Linkerd ecosystem.

Linkerd /metrics endpoint

Each Linkerd 2.0 proxy, injected as a sidecar with your application, serves a /metrics endpoint, providing application-level metrics for all requests transiting through your application’s pod.

For example, to view /metrics from a single Linkerd 2.0 proxy, running in the linkerd namespace:

kubectl port-forward -n linkerd $(kubectl -n linkerd get pods -l linkerd.io/control-plane-ns=linkerd -o jsonpath='{.items[0].metadata.name}') 4191:4191

and then…

curl localhost:4191/metrics

Prometheus JSON API

Similar to the /federate API, Prometheus provides a JSON API to retrieve all metrics:

curl http://prometheus.linkerd.svc.cluster.local:9090/api/v1/query?query=request_total

Prometheus /metrics endpoint

Similar to Linkerd’s /metrics endpoint, Prometheus’ /metrics endpoint provides metrics strictly about the Prometheus process itself, such as storage and memory.

curl http://prometheus.linkerd.svc.cluster.local:9090/metrics

Additional docs

More info about integrating with Linkerd’s metrics at https://linkerd.io/2/observability/