Linkerd top in browser not working

Hi,

I am running linkerd in a Digital Ocean Kubernetes cluster. I can run, for example:

linkerd top namespace/linkerd

in the terminal and it works just fine. Running it via the web interface gives me an error:

 Error during WebSocket handshake: Unexpected response code: 403

and the server logs say:

time="2020-12-27T02:07:58Z" level=error msg="websocket: request origin not allowed by Upgrader.CheckOrigin"

which, I guess, is straight forward enough, but I am not sure how to fix this. I am running linkerd dashboard behind ambassador, the mapping is:

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: linkerd
spec:
  prefix: /
  host: linkerd.wetsnow.com
  add_linkerd_headers: true
  resolver: endpoint
  host_rewrite: linkerd-web.linkerd.svc.cluster.local:8084
  service: linkerd-web.linkerd.svc.cluster.local:8084
  allow_upgrade:
  - websocket

Is the problem due to the host_rewrite? I access linkerd externally as “linkerd.mydomain.com”. Is there a way to configure this so it works properly?

Thank you
Dan

Hi @MrWetsnow, the ambassador configuration looks correct. Would you mind checking the web component logs when this request fails?

kubectl logs -n linkerd linkerd-web

Is any other functionality of the dashboard not working?

Sorry for the delay:

2021/01/15 22:46:52 http: superfluous response.WriteHeader call from go.opencensus.io/plugin/ochttp.(*trackingResponseWriter).WriteHeader (server.go:221)
time="2021-01-15T22:47:07Z" level=error msg="websocket: request origin not allowed by Upgrader.CheckOrigin"

@MrWetsnow I’m a bit baffled by this one.

That log output appears to be an error with sending opencensus data. Do you have the distributing tracing add-on installed?

Do you get the same error and behavior when you run linkerd dashboard to access the dashboard directly, instead of going through the ingress?

Those are two separate errors, but they both seem to show up at the same time. I did have the tracing add-on installed at the time.

I have since then remove the distributed tracing add-on in lieu of installing jaeger manually. So right now the distributing tracing add-on is off, but the error still persists.

If I run linkerd dashboard then both tap and top work just fine. It only breaks going through ambassador. It seems that the websocket request is making it through ambassador to the backend (otherwise we wouldn’t see the log entry), but perhaps ambassador is stripping some information that the backend is expecting? Is there a way to enable additional debugging?

I wonder if this has to do with:

  host_rewrite: linkerd-web.linkerd.svc.cluster.local:8084

Just reading about this, e.g., here: websocket - GoDoc

I should also mention that I have other services behind this ambassador instance (e.g. grafana) which also use websockets, and they work fine. But none of them have a host_rewrite setting, only llinkerd does.

I have to admit that I’m a bit stumped by this one @MrWetsnow

Would you mind opening a GitHub issue with steps to repro so we can track it there?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.