Linkerd on Mac fails to connect to remote namerd

I’m trying to run a local linkerd process which uses a remote namerd service. The namerd is behind a VPC but I am connected to a VPN and can access the namerd service directly. When I start linkerd it doesn’t report any errors.

If I try to access a service I get

E 1118 19:11:40.319 UTC THREAD30 TraceId:78f4be8cf22d3f71: service failure: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/test, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available

When I go to the local linkerd admin and try to view the dtab, linkerd prints the following error

E 1118 19:11:48.355 UTC THREAD27 TraceId:89d18f7b71727209: dtab incoming lookup failed
Failure(Released, flags=0x02) with RemoteInfo -> Upstream Address: /127.0.0.1:61267, Upstream id: Not Available, Downstream Address: /10.300.0.46:31872, Downstream label: interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig, Trace Id: 0489fc08262ff40e.0489fc08262ff40e<:0489fc08262ff40e with Service -> interpreter/io.buoyant.namerd.iface.NamerdInterpreterConfig

10.300.0.46 is the internal IP of the namerd service.

My linkerd config

admin:
    ip: 0.0.0.0
    port: 9990
telemetry:
- kind: io.l5d.prometheus
-
    kind: io.l5d.recentRequests
    sampleRate: 1
usage:
    enabled: false
routers:
-
    label: outgoing
    protocol: http
    servers:
    -
        port: 4140
        ip: 0.0.0.0
    interpreter:
        kind: io.l5d.namerd
        dst: /$/inet/10.300.0.46/31872
        namespace: outgoing
-
    label: incoming
    protocol: http
    servers:
    -
        port: 4141
        ip: 0.0.0.0
    interpreter:
        kind: io.l5d.namerd
        dst: /$/inet/10.300.0.46/31872
        namespace: incoming
        transformers:
        - kind: io.l5d.localhost

This is the same linkerd config I’m using in my k8s cluster except the namerd dest is the local name. I am also using this same config in a Linux VM running linkerd and it works in both of those.

Any thoughts what I’m missing?

Hey @Travis thanks for including your config. We can try and reproduce the issue you are running into as closely as we can based on the info you provided. We appreciate any other additional information that may helping in figuring out the issue you are running into.

Hey @Travis I wanted to see if you were still running into issues. We may need more info from you. When you open up the linkerd admin dtab UI are you getting the error message you posted above or are you getting that message from the linkerd logs?

Yes, still getting the issues.

All the errors messages I gave previously are from logs. The UI just shows empty dtab as if I never set them even though they are set.

I’m running linkerd and namerd 1.3.2 by the way. I’m on Mac OS Sierra. I’ve tried running locally on my mac and inside of a docker container. Both have same behavior.

Hmm interesting, can you confirm if you are seeing an empty dtab in the namerd UI as well? Maybe we can find something peculiar if we take a look at screenshots i.e. Namerd admin UI, Linkerd admin UI. Also what does your Namerd config look like?

Hi @MrTravisB. Can you confirm that linkerd can connect to 10.300.0.46:31872. Can you curl -v from your mac? As it’s a thrift interface I would not expect a successful reply, but at least an indication the connection is working. Also if you can share your namerd config that is also helpful.