Linkerd tracelog config

Hi,

I was actually looking for a way to assign a unique id like trace id to every request that goes past the linkerd proxy server. Linkerd has a telemetry option of kind io.l5d.tracelog. I have configured my linkerd to collect trace logs with the telemetry option. Where exactly can I see those trace logs?

My linkerd telemetry config is as follows

telemetry:
    - kind: io.l5d.tracelog
      sampleRate: 0.2
      level: TRACE

My linkerd is deployed on kubernetes and I see no logs related to trace info in my linkerd logs. So how or where exactly can I find my trace information?

Hey @prasaanth – all good questions. The tracelog telemeter writes to the main linkerd log, which defaults to stderr. You can redirect that to a file by setting the -log.output flag when starting linkerd.

Linkerd’s default log level is INFO, and your tracelog telemetry configuration specifies TRACE. So with the default logging level, you won’t see any of the trace logs in linkerd’s logs. To remedy this, you can either change your telemeter config to level: INFO, or you can set the -log.level=TRACE flag when starting linkerd.

And linkerd is already assigning a unique ID to each request. If you’re using http, linkerd will add a l5d-reqid header to all requests that it forwards, and you can read the unique ID from that header. More info about informational headers can be found here.

Hey @klingerf

Thanks a lot for this. I was able to start my linkerd with TRACE log level and got some info as well. Thanks for your reply.

And regarding the l5d-reqid, is there any way I could read them out of the box without the use of any plugins?

If you’re using HTTP, l5d-reqid is sent as a standard HTTP header, so your applications shouldn’t need any plugins to read the header; they’d just need to know the header name.

Hey @klingerf

l5d-reqid is what I was looking for. Thank you. :slight_smile:

But I have a scenario where I have three microservices deployed on Kubernetes and all proxied through Linkerd. I need to save the complete request so that I can retry them in future in case any problems. For example I call micro-service A which internally calls micro-service B and C. Now both micro-service B and C are proxied through the same Linkerd instance and hence it has the same host as that of A. The problem is every time the request comes out of Linkerd it assigns a different reqid to it as it does not know whether it was a new call or an internal call. Now it seems to be difficult to trace the call graph for A.
How would I make sure the reqid for that of A propagates to B and C so that I can get hold of the entire request graph?

Hey @prasaanth, great question. This is actually what the l5d-reqid header gives you. As long as you propagate request context, all of the intermediary requests will be assigned the same trace id, which is what’s transmitted in the l5d-reqid header (apologies for the confusing naming). So if a single request requires calling services A, B, and C, all three of those calls should have the same reqid.

Preserving the reqid across multiple linkerd calls requires request context propagation. Basically, your applications have to forward all of the l5d-ctx* headers so that Linkerd can successfully establish that two separate requests should be assigned the same reqid. There are a lot of other benefits to context propagation too, so it’s good practice to propagate context wherever possible within your application. There’s a longer note about that in this blog post:

https://buoyant.io/2016/05/17/distributed-tracing-for-polyglot-microservices/#request-context

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.