K3s + Linkerd 2.x expose service on NodeIP

Help
#1

Hi!

I run a k3s-based cluster. On my nodes, there is a native Apache running on ports 80 and 443, so I deploy k3s (v1.18.4-k3s1) with the --no-traefik option since Traefik otherwise hogs the ports from the native Apache which must not happen. I have a client that lives outside the cluster and wants to access a service on each node’s host IP at port 8080. When I set it up as a NodePort service and a DaemonSet (to make one copy run at each node that matches the selectors) and deploy it via kubectl without Linkerd, everything works nicely.

I then want to mesh it with Linkerd (I use 2.8.1), but when I do, accessing the service will cause a 502 bad request. Wiresharking into things, I can see that traffic comes into the cluster but then there are routing issues that I do not quite understand. I was thinking perhaps I am barking up the wrong tree and should use an ingress server for this? I did some experiments with nginx-ingress, but just like Traefik I cannot make it not take over port 80 and 443 from the native Apache webserver on the node.

I do not want a load balancer here that would treat all nodes like they were interchangeable, since they are not and that is also why the client will contact each node individually (with the external IP which works fine with the NodePort setup as long as I do not mesh it).

Do you guys have any input on how I should proceed here? I feel it should probably be a fairly simple setup only that I have failed to find it yet.

BR,

/J

#2

hi @joakimr-axis, I found an issue that is similar to the behavior you’re describing.

The first thing I’d do is use the debug container inside the pod to see what traffic is getting into the container, if at all.

Are there any errors in the proxy logs for the NodePort service? It sounds like you did a bit of wireshark sleuthing and I’d be curious to know where the 502 response comes from. If it’s in the proxy, then we should see it in the proxy logs.

Using an ingress controller is a simple solution, and you can configure nginx to use ports other than 80 and 443, by setting the http-port and https-port arguments, respectively.

Out of curiosity, what kind of traffic is being sent to port 8080? One quick fix would be to use the --skip-inbound-ports to include port 8080. This will prevent the proxy from attempting to handle any traffic that comes in on port 8080. Because the traffic is coming from an unmeshed client, this will act as a pass through and there is no real downside.

Charles

#3

Thanks for your input, @cpretzer! I conducted more experiments now with the skip ports part and found out a strange behavior:

In order to narrow things down I used another container, https://hub.docker.com/repository/docker/d97jro/echo-server, which is a minimal server that echoes whatever requests it gets. Very useful as a test container. It listens to HTTP traffic on port 8080 and I deployed it with a NodePort 30081. Unmeshed, I can then do curl http://<nodes_host_ip>:30081/teststring from a computer outside the cluster and get “teststring” echoed back. Then I mesh the very same YAML file with Linkerd (NB! No ports skipped or anything, just plug & play linkerd inject!) and the behavior is still intact. But now I can monitor everything in the Linkerd dashboard and everything is fine and dandy just like I wanted them to be. All this then without any skipped ports.

Then I revert back to my other container, where I use a URL for retrieving that microservice’s version. In the unmeshed state, with the NodePort 30080, I call curl http://<nodes_host_ip>:30080/getversioncmd from a computer outside the cluster and get the expected string. Meshing it in the same way as the echo server, I am back in the 502 bad gateway trench trying that same curl command. But! If I exec into a Linkerd debug sidecar container, and try curl http://<nodes_cluster_internal_ip>:8080/getversioncmd, the version string is properly returned (as one would expect it to be).

For the calls from outside the cluster, linkerd-proxy will report:

[ 248.309742278s] WARN inbound:accept{peer.addr=10.42.2.1:38922}:source{target.addr=10.42.2.40:8080}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)

I sniff the traffic with tcpdump for the curl call from outside the cluster and see this

1	0.000000	10.42.3.1	10.42.3.23	HTTP	268	GET /getprotocolversion HTTP/1.1
2	0.002643	10.42.3.23	10.42.3.1	HTTP	150	HTTP/1.1 502 Bad Gateway

(I will now do some more experiments with the ingress server and see if I can make it not steal the nodes’ native Apaches’ thunder.)

BR,

/J

#4

Thanks for the update @joakimr-axis, It’s interesting that everything works as expected with the echo server.

I most often see the error below when there is a firewall rule or address binding issue.

Can you share the YAML files for both the echo service and your service? If there is sensitive information, you can email them to me at charles@buoyant.io, or DM me at https://slack.linkerd.io

Charles

#5

The echo server YAML looks like this:

---
apiVersion: v1
kind: Namespace
metadata:
  name: echo-app
---
apiVersion: v1
kind: Service
metadata:
  name: echo-server
  namespace: echo-app
  labels:
    app: echo-app
spec:
  type: NodePort
  ports:
    - name: http
      port: 8080
      nodePort: 30081
    - name: tcp
      port: 8090
      nodePort: 30090
  selector:
    app: echo-app
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: echo-server
  namespace: echo-app
spec:
  selector:
    matchLabels:
      app: echo-app
  template:
    metadata:
      labels:
        app: echo-app
    spec:
      containers:
        - name: echo-server
          image: docker.io/d97jro/echo-server:latest

And the one for the app that does not work looks like this (I have removed a handful of volume mounts to make it more readable):

---
apiVersion: v1
kind: Namespace
metadata:
  name: foo
---
apiVersion: v1
kind: Service
metadata:
  name: foo
  namespace: foo
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30080
      name: foo
  selector:
    app: foo
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: foo
  namespace: foo
  labels:
    app: foo
    version: v1
spec:
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
    spec:
      nodeSelector:
        kubernetes.io/os: linux
        beta.kubernetes.io/arch: arm
      containers:
        - name: foo
          image: internal.registry/experiment/arm32v7/foo:1234
          ports:
            - containerPort: 8080
          env:
            - name: DBUS_SYSTEM_BUS_ADDRESS
              value: unix:path=/var/run/dbus/system_bus_socket
          securityContext:
            capabilities:
              add:
                - SYS_RESOURCE
          volumeMounts:
            - name: run-dbus
              mountPath: /run/dbus
              readOnly: false
        - name: bar
          image: internal.registry/experiment/arm32v7/bar:latest
          ports:
            - containerPort: 42420
          env:
            - name: DBUS_SYSTEM_BUS_ADDRESS
              value: unix:path=/var/run/dbus/system_bus_socket
          securityContext:
            capabilities:
              add:
                - SYS_RESOURCE
          volumeMounts:
            - name: var-run-dbus
              mountPath: /var/run/dbus
              readOnly: false
      volumes:
        - name: run-dbus
          hostPath:
            path: /run/dbus
        - name: var-run-dbus
          hostPath:
            path: /var/run/dbus
#6

Addition: The cluster runs behind a network proxy, so I use linkerd inject --manual since the webhooks don’t play ball when there’s a proxy.

#7

@joakimr-axis I wonder if the iptables rules are being defined properly.

What is the output from the kubectl logs <pod-name> linkerd-init?

#8

@cpretzer Hi, I’m a colleague of @joakimr-axis working on the same issue. He is currently on vacation but he advised me to follow up with you on this thread. I tried out the command you suggested and this is the output
error: container linkerd-init is not valid for the pod

#9

@cpretzer Sorry for the previous post. I had a mistake while injecting the pod which is why the error was occurring

This is the actual output of the linkerd-init logs

2020/07/09 14:08:48 Tracing this script execution as [1594303728]
2020/07/09 14:08:48 State of iptables rules before run:
2020/07/09 14:08:48 > iptables -t nat -vnL
2020/07/09 14:08:48 < Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

2020/07/09 14:08:48 > iptables -t nat -F PROXY_INIT_REDIRECT
2020/07/09 14:08:48 < iptables: No chain/target/match by that name.

2020/07/09 14:08:48 > iptables -t nat -X PROXY_INIT_REDIRECT
2020/07/09 14:08:48 < iptables: No chain/target/match by that name.

2020/07/09 14:08:48 Will ignore port(s) [4190 4191] on chain PROXY_INIT_REDIRECT
2020/07/09 14:08:48 Will redirect all INPUT ports to proxy
2020/07/09 14:08:48 > iptables -t nat -F PROXY_INIT_OUTPUT
2020/07/09 14:08:49 < iptables: No chain/target/match by that name.

2020/07/09 14:08:49 > iptables -t nat -X PROXY_INIT_OUTPUT
2020/07/09 14:08:49 < iptables: No chain/target/match by that name.

2020/07/09 14:08:49 Ignoring uid 2102
2020/07/09 14:08:49 Redirecting all OUTPUT to 4140
2020/07/09 14:08:49 Executing commands:
2020/07/09 14:08:49 > iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -N PROXY_INIT_OUTPUT -m comment --comment proxy-init/redirect-common-chain/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -o lo ! -d 127.0.0.1/32 -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-non-loopback-local-traffic/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1594303728
2020/07/09 14:08:50 <
2020/07/09 14:08:50 > iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1594303728
2020/07/09 14:08:50 <
2020/07/09 14:08:50 > iptables -t nat -vnL
2020/07/09 14:08:50 < Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 PROXY_INIT_REDIRECT all – * * 0.0.0.0/0 0.0.0.0/0 /* proxy-init/install-proxy-init-prerouting/1594303728 */

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 PROXY_INIT_OUTPUT all – * * 0.0.0.0/0 0.0.0.0/0 /* proxy-init/install-proxy-init-output/1594303728 */

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain PROXY_INIT_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 PROXY_INIT_REDIRECT all – * lo 0.0.0.0/0 !127.0.0.1 owner UID match 2102 /* proxy-init/redirect-non-loopback-local-traffic/1594303728 /
0 0 RETURN all – * * 0.0.0.0/0 0.0.0.0/0 owner UID match 2102 / proxy-init/ignore-proxy-user-id/1594303728 /
0 0 RETURN all – * lo 0.0.0.0/0 0.0.0.0/0 / proxy-init/ignore-loopback/1594303728 /
0 0 REDIRECT tcp – * * 0.0.0.0/0 0.0.0.0/0 / proxy-init/redirect-all-outgoing-to-proxy-port/1594303728 */ redir ports 4140

Chain PROXY_INIT_REDIRECT (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp – * * 0.0.0.0/0 0.0.0.0/0 multiport dports 4190,4191 /* proxy-init/ignore-port-4190,4191/1594303728 /
0 0 REDIRECT tcp – * * 0.0.0.0/0 0.0.0.0/0 / proxy-init/redirect-all-incoming-to-proxy-port/1594303728 */ redir ports 4143

#10

hi @ssen-axis, I have to admit, I’m a bit baffled by this one. I see that the DBUS_SYSTEM_BUS_ADDRESS environment variable is set on the containers that are failing, but I don’t know enough about that environment variable to know whether it will affect Linkerd functionality. I did a bit of research and I don’t think that variable will do anything.

I think the next thing to do is use tshark in the Linkerd debug container in verbose mode to look at the packets going between the proxy and the service.

#11

Hi,

Here’s a snapshot of the network traffic from the Linkerd debug container in verbose mode

1448582 448784.621284138 10.42.1.1 → 10.42.1.3 HTTP 184 GET /ready HTTP/1.1
1448583 448784.621332587 10.42.1.3 → 10.42.1.1 TCP 68 4191 → 44624 [ACK] Seq=1 Ack=117 Win=64256 Len=0 TSval=2886559065 TSecr=808006533
1448584 448784.622296700 10.42.1.3 → 10.42.1.1 HTTP 149 HTTP/1.1 200 OK
1448585 448784.622391815 10.42.1.1 → 10.42.1.3 TCP 68 44624 → 4191 [ACK] Seq=117 Ack=82 Win=64896 Len=0 TSval=808006534 TSecr=2886559066
1448586 448784.622500184 10.42.1.3 → 10.42.1.1 TCP 68 4191 → 44624 [FIN, ACK] Seq=82 Ack=117 Win=64256 Len=0 TSval=2886559066 TSecr=808006534
1448587 448784.623031817 10.42.1.1 → 10.42.1.3 TCP 68 44624 → 4191 [FIN, ACK] Seq=117 Ack=83 Win=64896 Len=0 TSval=808006534 TSecr=2886559066
1448588 448784.623111088 10.42.1.3 → 10.42.1.1 TCP 68 4191 → 44624 [ACK] Seq=83 Ack=118 Win=64256 Len=0 TSval=2886559066 TSecr=808006534
1448589 448786.832996254 10.42.1.1 → 10.42.1.3 HTTP 119 GET /getprotocolversion HTTP/1.1
1448590 448786.835302713 127.0.0.1 → 127.0.0.1 TCP 76 [TCP Port numbers reused] 32830 → 8080 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=3209274411 TSecr=0 WS=128
1448591 448786.835365524 127.0.0.1 → 127.0.0.1 TCP 56 8080 → 32830 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
1448592 448786.836223700 10.42.1.3 → 10.42.1.1 HTTP 140 HTTP/1.1 502 Bad Gateway

I’ve highlighted the HTTP parts

#12

This is helpful @ssen-axis, the /ready/ request is the kubelet running the readiness check on the proxy container.

The output below appears as though the 502 bad gateway is coming from the service listening on 8080. Does that service set or manipulate headers in the response?

To get the packet level request for the request, can you run this tshark command using linkerd-debug against the failing container?

k exec -it <pod-name> -c linkerd-debug -- tshark -i any -f "tcp" -V \ -Y "tcp.port == 8080"

#13

Thank you so much for the help @cpretzer

The output below appears as though the 502 bad gateway is coming from the service listening on 8080. Does that service set or manipulate headers in the response?

I’ll need to check with the application developer regarding this question. In the mean time, I ran the tshark command specified to get the packet level request. I’m providing the relevant parts below

Hypertext Transfer Protocol
GET /getprotocolversion HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET /getprotocolversion HTTP/1.1\r\n]
[GET /getprotocolversion HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: GET
Request URI: /getprotocolversion
Request Version: HTTP/1.1
Host: host ip:30080\r\n
\r\n
[Full request URI: http: //host ip:30080/getprotocolversion]
[HTTP request 1/1]

Hypertext Transfer Protocol
HTTP/1.1 502 Bad Gateway\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 502 Bad Gateway\r\n]
[HTTP/1.1 502 Bad Gateway\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.1
Status Code: 502
[Status Code Description: Bad Gateway]
Response Phrase: Bad Gateway
content-length: 0\r\n
[Content length: 0]
date: Wed, 22 Jul 2020 08:36:22 GMT\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.003058133 seconds]
[Request in frame: 1]
[Request URI: http: //host ip:30080/getprotocolversion]