Targeting Pod IPs with TLS enabled linkerd routers in k8s


#1

I am running a k8s daemonset linkerd (v1.4.5) configuration with TLS encryption between nodes and it is working well. So services are listening on http and utilizing linkerd to handle encryption between the nodes using a mesh certificate.

However there is one application, which is a Spring Boot Admin (https://www.baeldung.com/spring-boot-admin) that services register with and attempts to connect with them directly via their pod IP address and port as opposed to the service name. Using the io.buoyant.rinet router unsecured this can connect just fine, but I want this communication to be secured. When I use io.buoyant.rinet router with TLS it upgrades the connection to https and it finds the IP, but that IP is only talking http so that does not work obviously.

I have a routing set up similar to servicemesh.yaml where I have a router that uses the daemonset to upgrade the connection and route to another localnode router that downgrades the TLS connection, however I do not know how I can route using the direct pod ip & port to this router to downgrade appropriately.

The issue is somewhat similar to Targeting a specific Kubernetes pod
I think if I can figure out a way to use a k8s based namer to route to the pod directly then I can get this working but I have been unable to do that so far. Kubernetes provides dns names like 9-8-7-6.ns.pod.cluster.local:5432 for pods and I figure that I the starting point of a dtab but nothing I have attempted so far along those lines has worked.
Does anyone have any help to offer?


#2

@jsherman It definitely looks like you are running into to the issue you referenced. Is there any way you can give each of those spring boot admin agents their own unique service? It’s not ideal but it does solve the issue.


#3

I am not sure what you mean by ‘spring boot admin agents’ - in this scenario there is a single Spring Boot Admin pod which serves a k8s service object (lets say its called ‘spring-boot-admin’), and then there are also several other k8s services which are each served by one or more pods.

I believe I can have spring boot admin attempt to hit endpoints via service requests, however it loses the granularity that hitting the endpoints on a per pod basis provides when a service is served by multiple pods.


#4

Ah, I misunderstood your setup. I think I get it now. You have a single spring boot admin server. That spring boot server sends requests to pods that have registered with this admin server. I think there may be a way to get TLS working but it might be complicated. In your setup, do send requests to services external to your cluster i.e. egress?


#5

Yes, I would not want to exclude the ability to egress externally to the cluster as well. What is your proposal?


#6

Well, it would have involved moving the daemonset transformer to the router instead of having it in the k8s namer. That way, when the rinet namer resolves the pod DNS name to an ip, the transformer will be applied and the request from the admin server will end up at the intended daemonset pod. But that would mean, you wouldn’t be able to do egress routing.


#7

Interesting - just spitballing - I wonder, if I did that, if then I could set up Kubernetes selector-free services that just point to an externalName and get outside that way so long as I knew in advance whatever I would be wanting to hit externally.


#8

Interesting indeed. I haven’t used an externalName service with Linkerd before but if the service is able to create endpoints that Linkerd can query through the endpoints API, that would be one way to reach external services. Definitely worth something to try out.


#9

Is there a reason one cannot specify apply special transformers to a prefix of io.buoyant.rinet? Why can io.buoyant.rinet be transformed by entire-router configuration and not by the namer configuration?


#10

I think the issue with that is the rinet namer can’t be configured as a “standalone” namer like the k8s or consul namer. Its tightly coupled in Linkerd’s namer core functionality. We could break it out into its own “standalone” namer but that would require some thinking of what implications that would cause.