K3s + Linkerd 2.x expose service on NodeIP

cpretzer · July 1, 2020, 4:29am

hi @joakimr-axis, I found an issue that is similar to the behavior you’re describing.

The first thing I’d do is use the debug container inside the pod to see what traffic is getting into the container, if at all.

Are there any errors in the proxy logs for the NodePort service? It sounds like you did a bit of wireshark sleuthing and I’d be curious to know where the 502 response comes from. If it’s in the proxy, then we should see it in the proxy logs.

Using an ingress controller is a simple solution, and you can configure nginx to use ports other than 80 and 443, by setting the http-port and https-port arguments, respectively.

Out of curiosity, what kind of traffic is being sent to port 8080? One quick fix would be to use the --skip-inbound-ports to include port 8080. This will prevent the proxy from attempting to handle any traffic that comes in on port 8080. Because the traffic is coming from an unmeshed client, this will act as a pass through and there is no real downside.

Charles

joakimr-axis · July 1, 2020, 11:55am

Thanks for your input, @cpretzer! I conducted more experiments now with the skip ports part and found out a strange behavior:

In order to narrow things down I used another container, https://hub.docker.com/repository/docker/d97jro/echo-server, which is a minimal server that echoes whatever requests it gets. Very useful as a test container. It listens to HTTP traffic on port 8080 and I deployed it with a NodePort 30081. Unmeshed, I can then do curl http://<nodes_host_ip>:30081/teststring from a computer outside the cluster and get “teststring” echoed back. Then I mesh the very same YAML file with Linkerd (NB! No ports skipped or anything, just plug & play linkerd inject!) and the behavior is still intact. But now I can monitor everything in the Linkerd dashboard and everything is fine and dandy just like I wanted them to be. All this then without any skipped ports.

Then I revert back to my other container, where I use a URL for retrieving that microservice’s version. In the unmeshed state, with the NodePort 30080, I call curl http://<nodes_host_ip>:30080/getversioncmd from a computer outside the cluster and get the expected string. Meshing it in the same way as the echo server, I am back in the 502 bad gateway trench trying that same curl command. But! If I exec into a Linkerd debug sidecar container, and try curl http://<nodes_cluster_internal_ip>:8080/getversioncmd, the version string is properly returned (as one would expect it to be).

For the calls from outside the cluster, linkerd-proxy will report:

[ 248.309742278s] WARN inbound:accept{peer.addr=10.42.2.1:38922}:source{target.addr=10.42.2.40:8080}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)

I sniff the traffic with tcpdump for the curl call from outside the cluster and see this

1	0.000000	10.42.3.1	10.42.3.23	HTTP	268	GET /getprotocolversion HTTP/1.1
2	0.002643	10.42.3.23	10.42.3.1	HTTP	150	HTTP/1.1 502 Bad Gateway

(I will now do some more experiments with the ingress server and see if I can make it not steal the nodes’ native Apaches’ thunder.)

BR,

/J

cpretzer · July 1, 2020, 8:53pm

Thanks for the update @joakimr-axis, It’s interesting that everything works as expected with the echo server.

I most often see the error below when there is a firewall rule or address binding issue.

Can you share the YAML files for both the echo service and your service? If there is sensitive information, you can email them to me at charles@buoyant.io, or DM me at https://slack.linkerd.io

Charles

joakimr-axis · July 2, 2020, 6:45am

The echo server YAML looks like this:

---
apiVersion: v1
kind: Namespace
metadata:
  name: echo-app
---
apiVersion: v1
kind: Service
metadata:
  name: echo-server
  namespace: echo-app
  labels:
    app: echo-app
spec:
  type: NodePort
  ports:
    - name: http
      port: 8080
      nodePort: 30081
    - name: tcp
      port: 8090
      nodePort: 30090
  selector:
    app: echo-app
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: echo-server
  namespace: echo-app
spec:
  selector:
    matchLabels:
      app: echo-app
  template:
    metadata:
      labels:
        app: echo-app
    spec:
      containers:
        - name: echo-server
          image: docker.io/d97jro/echo-server:latest

And the one for the app that does not work looks like this (I have removed a handful of volume mounts to make it more readable):

---
apiVersion: v1
kind: Namespace
metadata:
  name: foo
---
apiVersion: v1
kind: Service
metadata:
  name: foo
  namespace: foo
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30080
      name: foo
  selector:
    app: foo
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: foo
  namespace: foo
  labels:
    app: foo
    version: v1
spec:
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
    spec:
      nodeSelector:
        kubernetes.io/os: linux
        beta.kubernetes.io/arch: arm
      containers:
        - name: foo
          image: internal.registry/experiment/arm32v7/foo:1234
          ports:
            - containerPort: 8080
          env:
            - name: DBUS_SYSTEM_BUS_ADDRESS
              value: unix:path=/var/run/dbus/system_bus_socket
          securityContext:
            capabilities:
              add:
                - SYS_RESOURCE
          volumeMounts:
            - name: run-dbus
              mountPath: /run/dbus
              readOnly: false
        - name: bar
          image: internal.registry/experiment/arm32v7/bar:latest
          ports:
            - containerPort: 42420
          env:
            - name: DBUS_SYSTEM_BUS_ADDRESS
              value: unix:path=/var/run/dbus/system_bus_socket
          securityContext:
            capabilities:
              add:
                - SYS_RESOURCE
          volumeMounts:
            - name: var-run-dbus
              mountPath: /var/run/dbus
              readOnly: false
      volumes:
        - name: run-dbus
          hostPath:
            path: /run/dbus
        - name: var-run-dbus
          hostPath:
            path: /var/run/dbus

joakimr-axis · July 2, 2020, 12:01pm

Addition: The cluster runs behind a network proxy, so I use linkerd inject --manual since the webhooks don’t play ball when there’s a proxy.

cpretzer · July 6, 2020, 7:23pm

@joakimr-axis I wonder if the iptables rules are being defined properly.

What is the output from the kubectl logs <pod-name> linkerd-init?

ssen-axis · July 8, 2020, 2:23pm

@cpretzer Hi, I’m a colleague of @joakimr-axis working on the same issue. He is currently on vacation but he advised me to follow up with you on this thread. I tried out the command you suggested and this is the output
error: container linkerd-init is not valid for the pod

ssen-axis · July 9, 2020, 2:12pm

@cpretzer Sorry for the previous post. I had a mistake while injecting the pod which is why the error was occurring

This is the actual output of the linkerd-init logs

2020/07/09 14:08:48 Tracing this script execution as [1594303728]
2020/07/09 14:08:48 State of iptables rules before run:
2020/07/09 14:08:48 > iptables -t nat -vnL
2020/07/09 14:08:48 < Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

2020/07/09 14:08:48 > iptables -t nat -F PROXY_INIT_REDIRECT
2020/07/09 14:08:48 < iptables: No chain/target/match by that name.

2020/07/09 14:08:48 > iptables -t nat -X PROXY_INIT_REDIRECT
2020/07/09 14:08:48 < iptables: No chain/target/match by that name.

2020/07/09 14:08:48 Will ignore port(s) [4190 4191] on chain PROXY_INIT_REDIRECT
2020/07/09 14:08:48 Will redirect all INPUT ports to proxy
2020/07/09 14:08:48 > iptables -t nat -F PROXY_INIT_OUTPUT
2020/07/09 14:08:49 < iptables: No chain/target/match by that name.

2020/07/09 14:08:49 > iptables -t nat -X PROXY_INIT_OUTPUT
2020/07/09 14:08:49 < iptables: No chain/target/match by that name.

2020/07/09 14:08:49 Ignoring uid 2102
2020/07/09 14:08:49 Redirecting all OUTPUT to 4140
2020/07/09 14:08:49 Executing commands:
2020/07/09 14:08:49 > iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -N PROXY_INIT_OUTPUT -m comment --comment proxy-init/redirect-common-chain/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -o lo ! -d 127.0.0.1/32 -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-non-loopback-local-traffic/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1594303728
2020/07/09 14:08:49 <
2020/07/09 14:08:49 > iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1594303728
2020/07/09 14:08:50 <
2020/07/09 14:08:50 > iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1594303728
2020/07/09 14:08:50 <
2020/07/09 14:08:50 > iptables -t nat -vnL
2020/07/09 14:08:50 < Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 PROXY_INIT_REDIRECT all – * * 0.0.0.0/0 0.0.0.0/0 /* proxy-init/install-proxy-init-prerouting/1594303728 */

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 PROXY_INIT_OUTPUT all – * * 0.0.0.0/0 0.0.0.0/0 /* proxy-init/install-proxy-init-output/1594303728 */

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain PROXY_INIT_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 PROXY_INIT_REDIRECT all – * lo 0.0.0.0/0 !127.0.0.1 owner UID match 2102 /* proxy-init/redirect-non-loopback-local-traffic/1594303728 /
0 0 RETURN all – * * 0.0.0.0/0 0.0.0.0/0 owner UID match 2102 / proxy-init/ignore-proxy-user-id/1594303728 /
0 0 RETURN all – * lo 0.0.0.0/0 0.0.0.0/0 / proxy-init/ignore-loopback/1594303728 /
0 0 REDIRECT tcp – * * 0.0.0.0/0 0.0.0.0/0 / proxy-init/redirect-all-outgoing-to-proxy-port/1594303728 */ redir ports 4140

Chain PROXY_INIT_REDIRECT (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp – * * 0.0.0.0/0 0.0.0.0/0 multiport dports 4190,4191 /* proxy-init/ignore-port-4190,4191/1594303728 /
0 0 REDIRECT tcp – * * 0.0.0.0/0 0.0.0.0/0 / proxy-init/redirect-all-incoming-to-proxy-port/1594303728 */ redir ports 4143

cpretzer · July 14, 2020, 7:44pm

hi @ssen-axis, I have to admit, I’m a bit baffled by this one. I see that the DBUS_SYSTEM_BUS_ADDRESS environment variable is set on the containers that are failing, but I don’t know enough about that environment variable to know whether it will affect Linkerd functionality. I did a bit of research and I don’t think that variable will do anything.

I think the next thing to do is use tshark in the Linkerd debug container in verbose mode to look at the packets going between the proxy and the service.

ssen-axis · July 21, 2020, 8:14am

Hi,

Here’s a snapshot of the network traffic from the Linkerd debug container in verbose mode

1448582 448784.621284138 10.42.1.1 → 10.42.1.3 HTTP 184 GET /ready HTTP/1.1
1448583 448784.621332587 10.42.1.3 → 10.42.1.1 TCP 68 4191 → 44624 [ACK] Seq=1 Ack=117 Win=64256 Len=0 TSval=2886559065 TSecr=808006533
1448584 448784.622296700 10.42.1.3 → 10.42.1.1 HTTP 149 HTTP/1.1 200 OK
1448585 448784.622391815 10.42.1.1 → 10.42.1.3 TCP 68 44624 → 4191 [ACK] Seq=117 Ack=82 Win=64896 Len=0 TSval=808006534 TSecr=2886559066
1448586 448784.622500184 10.42.1.3 → 10.42.1.1 TCP 68 4191 → 44624 [FIN, ACK] Seq=82 Ack=117 Win=64256 Len=0 TSval=2886559066 TSecr=808006534
1448587 448784.623031817 10.42.1.1 → 10.42.1.3 TCP 68 44624 → 4191 [FIN, ACK] Seq=117 Ack=83 Win=64896 Len=0 TSval=808006534 TSecr=2886559066
1448588 448784.623111088 10.42.1.3 → 10.42.1.1 TCP 68 4191 → 44624 [ACK] Seq=83 Ack=118 Win=64256 Len=0 TSval=2886559066 TSecr=808006534
1448589 448786.832996254 10.42.1.1 → 10.42.1.3 HTTP 119 GET /getprotocolversion HTTP/1.1
1448590 448786.835302713 127.0.0.1 → 127.0.0.1 TCP 76 [TCP Port numbers reused] 32830 → 8080 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=3209274411 TSecr=0 WS=128
1448591 448786.835365524 127.0.0.1 → 127.0.0.1 TCP 56 8080 → 32830 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
1448592 448786.836223700 10.42.1.3 → 10.42.1.1 HTTP 140 HTTP/1.1 502 Bad Gateway

I’ve highlighted the HTTP parts

cpretzer · July 21, 2020, 7:33pm

This is helpful @ssen-axis, the /ready/ request is the kubelet running the readiness check on the proxy container.

The output below appears as though the 502 bad gateway is coming from the service listening on 8080. Does that service set or manipulate headers in the response?

To get the packet level request for the request, can you run this tshark command using linkerd-debug against the failing container?

k exec -it <pod-name> -c linkerd-debug -- tshark -i any -f "tcp" -V \ -Y "tcp.port == 8080"

ssen-axis · July 22, 2020, 1:38pm

Thank you so much for the help @cpretzer

The output below appears as though the 502 bad gateway is coming from the service listening on 8080. Does that service set or manipulate headers in the response?

I’ll need to check with the application developer regarding this question. In the mean time, I ran the tshark command specified to get the packet level request. I’m providing the relevant parts below

Hypertext Transfer Protocol
GET /getprotocolversion HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET /getprotocolversion HTTP/1.1\r\n]
[GET /getprotocolversion HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: GET
Request URI: /getprotocolversion
Request Version: HTTP/1.1
Host: host ip:30080\r\n
\r\n
[Full request URI: http: //host ip:30080/getprotocolversion]
[HTTP request 1/1]

Hypertext Transfer Protocol
HTTP/1.1 502 Bad Gateway\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 502 Bad Gateway\r\n]
[HTTP/1.1 502 Bad Gateway\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.1
Status Code: 502
[Status Code Description: Bad Gateway]
Response Phrase: Bad Gateway
content-length: 0\r\n
[Content length: 0]
date: Wed, 22 Jul 2020 08:36:22 GMT\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.003058133 seconds]
[Request in frame: 1]
[Request URI: http: //host ip:30080/getprotocolversion]

ssen-axis · August 7, 2020, 8:28am

@cpretzer I was looking at the packet logs and was wondering about something. The URL in the request URI in the posted snippet is the local address (a 192.* address issued by my local router )of the device it’s trying to reach. Can that cause problems to access the device from within the cluster ?

joakimr-axis · August 25, 2020, 3:12pm

With the working echo-server example, I have successfully deployed it with node ports and meshed it. Then I can call the noded public IP (non-Kubernetes) and get the expected response. (Of course, calling the cluster internal addresses inside the cluster & mesh works too.)
Using the other container, where things do not work, in the same cluster & manner, I have now been able to get logs out from the web server that handles requests there. In the unmeshed case, where it works, it prints “I received request foobar” in the logs when it gets a request. When I mesh it, there are no such prints which would indicate that the requests never reach that webserver (or that it fails after having received it but before doing the print!?!).

joakimr-axis · August 26, 2020, 9:56am

I now tested with exactly the same manifest used for the non-working application (that uses port 8080 for its HTTP traffic) and replaced the image with the echo-server (also using port 8080). Everything with volumes, permissions, labels, etc. etc. exactly the same. And with the echo-server, it (again) works flawlessly.

linkerd-proxy will log

[ 195.837060318s] WARN ThreadId(02) inbound:accept{peer.addr=10.42.2.0:44854}:source{target.addr=10.42.1.19:8080}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)

To me it seems that for some reason, the webserver in the non-working application refuses to accept even looking at the request (which would explain why there is no logs there). Otherwise there would be no Connection refused, right?

cpretzer · August 28, 2020, 7:15pm

@joakimr-axis, thanks for following up on this. This is a very strange issue.

What protocol is the non-working application using? I’m beginning to wonder if there is something about the protocol that is preventing the request from being handled.

The other thing to do is set the proxy log level to trace for the non-working application and collect the log output.

Finally, if you can provide me with a reproducible test case, I can work to debug the issue. Do you have a stripped down version of the non-working application that I can deploy to my cluster? It looks like the image is image: internal.registry/experiment/arm32v7/foo:1234

joakimr-axis · September 7, 2020, 8:56am

Sorry about the delay, and thanks for helping out here!

It has not been possible for me to get my hands on the source code (so I have not even seen it myself and this component is still a black box for me), but I have now pinpointed the problem: the Connection Refused happens because that web server refuses incoming requests on e.g. 127.0.0.1 and localhost.

My workaround was to build a small relay server using the readily available standard go webserver component handling (any) incoming requests and then letting libcurl pass them on to the problematic component on an IP address that it will accept (I use the pod’s eth0 address). Problem “solved”.

Case closed.

joakimr-axis · September 7, 2020, 8:57am

(I forgot to mention: I let the relay server run in the same container, even though I of course don’t really want to run more than one process per container.)

cpretzer · September 7, 2020, 4:45pm

Wow @joakimr-axis! That’s some good work you did there. Sorry that it was so complex to figure out.

As I think through this, I can’t think of a different way to accomplish what you’ve done with your workaround. It’s really interesting that the service doesn’t accept connections on localhost or 127.0.0.1. It must be binding to the IP address of the pod, as you mentioned.

Thanks again for the update.

system · October 7, 2020, 4:45pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.