Issues Connecting Linkerd to Namerd

I am trying to setup Namerd with Linkerd in OpenShift and Linkerd is setup as a daemon set. After starting up my cluster, I can access the Namerd admin ui and see my dtabs (currently stored in memory). I can launch the Linkerd admin ui but if I try to view the dtabs or the Namerd tabs, I get the following error:

E 0814 13:30:23.081 UTC THREAD24 TraceId:bb6c363ce98eea97: dtab default lookup failed
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /#/io.l5d.k8s/myproject/thrift/namerd, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available
E 0814 13:30:23.082 UTC THREAD24 TraceId:bb6c363ce98eea97: dtab default lookup failed
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /#/io.l5d.k8s/myproject/thrift/namerd, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available
D 0814 13:30:23.099 UTC THREAD24 TraceId:bb6c363ce98eea97: dtab default released
D 0814 13:30:23.099 UTC THREAD24 TraceId:bb6c363ce98eea97: dtab default released
E 0814 13:30:23.101 UTC THREAD24: adminhttp
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /#/io.l5d.k8s/myproject/thrift/namerd, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available
W 0814 13:30:23.114 UTC THREAD24: Exception propagated to the default monitor (upstream address: /172.17.0.5:40134, downstream address: n/a, label: adminhttp).
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /#/io.l5d.k8s/myproject/thrift/namerd, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available

I’m assuming I don’t have Linkerd configured correctly to talk to my Namerd instance.

Namerd Config

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: namerd-config
data:
  config.yml: |-
    admin:
      port: 9990

    namers:
    - kind: io.l5d.k8s
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.http
      experimental: true
      host: localhost
      port: 8001
      transformers:
        - kind: io.l5d.k8s.daemonset
          namespace: myproject
          port: incoming
          service: l5d-incoming
          hostNetwork: true
          
    storage:
      kind: io.l5d.inMemory
      namespaces:
        default: |
          /cluster   => /#/io.l5d.fs;
          /srv        => /#/io.l5d.k8s/myproject/http;
          /svc/user-svc => /srv/user-svc;
          /svc/company-svc => /srv/company-svc;

    interfaces:
    # used by linkerds to receive updates
    - kind: io.l5d.thriftNameInterpreter
      ip: 0.0.0.0
      port: 4100
    # used by namerctl to manage configuration
    - kind: io.l5d.httpController
      ip: 0.0.0.0
      port: 4180
---
kind: ReplicationController
apiVersion: v1
metadata:
  name: namerd
spec:
  replicas: 1
  selector:
    app: namerd
  template:
    metadata:
      labels:
        app: namerd
    spec:
      dnsPolicy: ClusterFirst
      volumes:
      - name: namerd-config
        configMap:
          name: namerd-config
      containers:
      - name: namerd
        image: buoyantio/namerd:1.1.2
        args:
        - /io.buoyant/namerd/config/config.yml
        ports:
        - name: thrift
          containerPort: 4100
        - name: http
          containerPort: 4180
        - name: admin
          containerPort: 9990
        volumeMounts:
        - name: "namerd-config"
          mountPath: "/io.buoyant/namerd/config"
          readOnly: true
      - name: kubectl
        image: buoyantio/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
---
apiVersion: v1
kind: Service
metadata:
  name: namerd-admin
spec:
  selector:
    app: namerd
  type: LoadBalancer
  ports:
  - name: admin
    port: 80
    targetPort: 9990
---
apiVersion: v1
kind: Service
metadata:
  name: namerd
spec:
  selector:
    app: namerd
  type: LoadBalancer
  ports:
  - name: thrift
    port: 4100
  - name: http
    port: 4180
  - name: admin
    port: 9990

Linkerd Config:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: l5d-config
data:
  config.yaml: |-
    admin:
      port: 9990

    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25

    usage:
      orgId: linkerd-examples-daemonset-namerd

    routers:
    - protocol: http
      label: outgoing
      interpreter:
        kind: io.l5d.namerd
        dst: /#/io.l5d.k8s/myproject/thrift/namerd
        namespace: default 
      servers:
      - port: 4140
        ip: 0.0.0.0
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX

    - protocol: http
      label: incoming
      interpreter:
        kind: io.l5d.namerd
        dst: /#/io.l5d.k8s/myproject/thrift/namerd
        namespace: default 
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      servers:
      - port: 4141
        ip: 0.0.0.0
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: l5d
  name: l5d
spec:
  template:
    metadata:
      labels:
        app: l5d
    spec:
      hostNetwork: true
      volumes:
      - name: l5d-config
        configMap:
          name: "l5d-config"
      containers:
      - name: l5d
        image: buoyantio/linkerd:1.1.2
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - /io.buoyant/linkerd/config/config.yaml
        ports:
        - name: outgoing
          containerPort: 4140
          hostPort: 4140
        - name: incoming
          containerPort: 4141
        - name: admin
          containerPort: 9990
        volumeMounts:
        - name: "l5d-config"
          mountPath: "/io.buoyant/linkerd/config"
          readOnly: true

      - name: kubectl
        image: buoyantio/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
---
apiVersion: v1
kind: Service
metadata:
  name: l5d-outgoing
spec:
  selector:
    app: l5d
  type: LoadBalancer
  ports:
  - name: outgoing
    port: 80
    targetPort: 4140
---
apiVersion: v1
kind: Service
metadata:
  name: l5d-incoming
spec:
  selector:
    app: l5d
  type: LoadBalancer
  ports:
  - name: incoming
    port: 80
    targetPort: 4141
---
apiVersion: v1
kind: Service
metadata:
  name: l5d-admin
spec:
  selector:
    app: l5d
  type: LoadBalancer
  ports:
  - name: admin
    port: 80
    targetPort: 9990

Any help would be greatly appreciated!!

Hi @lmarie. A couple things to check:

  1. Are you able to use Kubernetes DNS to specify the namerd address, for example:
    dst: /$/inet/namerd.default.svc.cluster.local/4100 (from https://github.com/linkerd/linkerd-examples/blob/master/k8s-daemonset/k8s/linkerd-namerd.yml#L44) ? If not you’ll need to configure a namers section in the linkerd config to be able to find /#/io.l5d.k8s/myproject/thrift/namerd.
  2. Is namerd deployed in the myproject namespace?

@siggy I don’t think I’m able to specify the namerd address using Kubernetes DNS with OpenShift (need to research this more). I added the following namer to my Linkerd config and I am now able to see data in the Dtab and Namerd tabs of the Admin UI. I’m getting other errors now but they are related to missing dtab entries.

namers:
- kind: io.l5d.k8s
  experimental: true
  host: localhost
  port: 8001

My instance of Namerd and Linkerd are both in the myproject namespace. Thanks for all the help!