Unable to configure Linkerd as gRPC load balancer

Hi,

I am trying to configure Linkerd as load balancer for my gRPC client-server communication.
All servers and the client are running into Kubernetes - 1 client and 2 (for now) servers. I’ve followed your startup guide.

gRPC server is listening on port 9050.
gRPC client is using blocking stub pointing to grpc-service host and port 9050

Linkerd finds the grpc pods in the mesh:
image

What happens is that no matter of how many requests (in the same gRPC ManagedChannel) I send to the server, only one of the PODs is receiving them. On next execution it could be the other node, it could be the same, but the once a Channel is created it does not “re-route” gRPC requests to different PODs.
It acts just like if there is no Linkerd and K8s service is doing it usual round robin stuff.

I couldn’t find anything else to configure apart from “…inject: enabled” annotation.
I am still using the “grpc-service” as hostname for the client - shall I use some other address?
Shall I add some rules somewhere?

Here are k8s deployment and service descriptors

apiVersion: apps/v1
kind: Deployment
metadata:
name: grpc
labels:
app: grpc
spec:
replicas: 2
selector:
matchLabels:
app: grpc
template:
metadata:
annotations:
linkerd.io/inject: enabled
labels:
app: grpc
spec:
containers:
- name: mda
image: xxxxxxx.azurecr.io/grpc-server:latest
imagePullPolicy: Always
ports:
- containerPort: 9050
protocol: TCP
name: grpc

apiVersion: v1
kind: Service
metadata:
name: grpc-service
spec:
type: ClusterIP
selector:
app: grpc
ports:
- protocol: TCP
port: 9050
targetPort: 9050
name: grpc

gRPC client code:
channel =
ManagedChannelBuilder.forAddress(“grpc-service”, 9050)
.usePlaintext()
.build();

gRPC server code:
ServerBuilder
.forPort(9050)
.addService(service)
.build()
.start()
.awaitTermination();

Hi @MOPT, I saw your Slack messages about this and am following up here as well.

Two things to check are:

  1. Whether the gRPC server uses unary requests or streaming API
  2. That the client sending the requests to the gRPC service is injected with the proxy

From the image that you posted, it looks like the client is not injected with the proxy. If the client is outside the cluster and you are routing traffic inbound through an ingress, then the ingress should also be injected with the linkerd proxy.

It works now - appeared I missed the instruction to inject side-car proxy on CLIENT pod too.
I needed some time to start working, but now it switches the pods as I expect.

Thank you.

Great news! I’m glad to hear that it’s working. Let us know here or in Slack if you have more questions.