Zipkin not able to collect tracing info

Hi,

I want to collect tracing info for all my microservices using zipkin and I can read linkerd provides out of the box support for zipkin. I have enabled zipkin telemetry in my linkerd configuration and deployed zipkin as well in my k8s cluster.

Zipkin.yml (945 Bytes)

And I have enabled telemetry in linkerd as follows

telemetry:
- kind: io.l5d.prometheus 
- kind: io.l5d.recentRequests
  sampleRate: 0.25
- kind: io.l5d.zipkin
  host: zipkin-collector.linkerd.svc.cluster.local
  port: 51010
  sampleRate: 1.0

But I still dont see Zipkin collecting trace info of linkerd. Is there something I am missing out?

Hey @prasaanth – the io.l5d.zipkin telemeter uses Zipkin’s scribe collector. So you need to make sure you start the Zipkin server with SCRIBE_ENABLED=true for it to collect traces, and the scribe collector runs on a different port than the web UI.

If you instead wanted to use Zipkin’s http collector, that runs on the same port as the web UI, but requires a separate telemetry plugin that’s not shipped with linkerd by default. Checkout linkerd-zipkin for more details about that.

Hi @klingerf
Yeah. My scribe collector is running on port 9410 which is a different port to that of the web UI. I have also changed my telemetry config to listen to the same scribe collector port.

---
apiVersion: v1
kind: ReplicationController
metadata:
  name: zipkin
spec:
  replicas: 1
  selector:
    app: zipkin
  template:
    metadata:
      name: zipkin
      labels:
        app: zipkin
    spec:
      containers:
      - name: zipkin
        image: openzipkin/zipkin:1.20
        env:
        - name: SCRIBE_ENABLED
          value: "true"
        ports:
        - name: scribe
          containerPort: 9410
        - name: http
          containerPort: 9411
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: zipkin-collector
  name: zipkin-collector
spec:
  type: ClusterIP
  selector:
    app: zipkin
  ports:
  - name: scribe
    port: 9410
    targetPort: 9410
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: zipkin
  name: zipkin
spec:
  type: LoadBalancer
  selector:
    app: zipkin
  ports:
  - name: http
    port: 80
    targetPort: 9411

This is how my linkerd telemetry looks like

telemetry:
    - kind: io.l5d.prometheus 
    - kind: io.l5d.recentRequests
      sampleRate: 0.25
    - kind: io.l5d.zipkin
      host: zipkin-collector.linkerd.svc.cluster.local
      port: 9410
      sampleRate: 1.0

I tried logging into the agent and doing a curl on this port and it seems to be not responding as well.

Hey @prasaanth – interesting. All of your configs look correct to me. I’m surprised that nothing is showing up in the Zipkin UI. I’d try looking at at the linkerd stats to see if there’s any additional information there. What stats do you see when you fetch metrics from one of your linkerd instances with:

curl :9990/admin/metrics.json?pretty=1 | grep zipkin

Hey @klingerf

I could see the grep actually returned nothing related to zipkin. Its wierd that zipkin still couldnt reach out to the linkerd instances.

Hmm, this makes it seem like the zipkin telemeter isn’t properly loaded at all. Is there anything printed in linkerd’s logs to suggest that it had issues loading the telemeter?

Not exactly. I can see that linkerd logs shows that zipkin tracer was enabled. But there are no error logs which means that zipkin was started properly.
When I tried running zipkin as a docker image and exporting my trace info from linkerd to the docker host I was able to see the trace info.
Zipkin is not able to read the trace info on kubernetes which is wierd.

Ah, interesting. This sounds like the communication between linkerd and zipkin pods is blocked. Are you running in a cluster with any networking plugins enabled that would block pod-to-pod communication by default? Maybe you need to update network policy to allow the data to be sent.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.