Linkerd GRPC tutorial - wired behaviour


#1

hello, all not sure if thats the place to ask but ill try. we are trying to do the grpc linkerd example on kibernetes from the tutorial we see weird behvaiour as follows: we invoke docker run --rm --entrypoint=helloworld-client buoyantio/helloworld:0.1.3 10.152.179.205:4140 and on the kubetail of l5d pods we see: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/helloworld.Hello but when i take /svc/helloworld.Hello to the delagator on the admin console of linkerd on this very node (10.152.179.205) and try it i get the match and the list of endpoints. any ideas?


#2

Hi @iftachsc. Can you provide a bit more detail about your setup? Which commands did you run exactly? What does your config file look like? Are you on GKE or minikube? Are you using CNI?


#3

many thanks for responding.
we are using kubernetes and we followed the GRPC tutorial, as you can see.

the only command we are using after setting up the l5d daemon set from yaml and the hellowworld grpc app also from yaml is the docker run command to start the traffic flow. but we are stopped there since it seem that linkerD cannot find endpoint…how can it be inconsistent with the fact that on delegator we have a match for the endpoints. see the screen shot.


#4

this is the exception from linkerd when we try to invoker the docker run command to start the grpc traffic. we’re using linkerd 1.3.2:

[l5d-3m8m2] I 1129 11:34:37.427 UTC THREAD31: no available endpoints
[l5d-3m8m2] com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/helloworld.Hello, Dtab.base=[/srv=>/#/io.l5d.k8s/default/grpc;/grpc=>/srv;/grpc/World=>/srv/world-v1;/svc=>/$/io.buoyant.http.domainToPathPfx/grpc], Dtab.local=[]. Remote Info: Not Available
[l5d-3m8m2]
[l5d-3m8m2] E 1129 11:34:37.515 UTC THREAD31: [S L:/10.152.179.205:4140 R:/10.152.178.164:47034] dispatcher failed
[l5d-3m8m2] com.twitter.finagle.ChannelClosedException: Connection reset by peer at remote address: /10.152.178.164:47034. Remote Info: Not Available
[l5d-3m8m2] at com.twitter.finagle.ChannelException$.apply(Exceptions.scala:258)
[l5d-3m8m2] at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$1.exceptionCaught(ChannelTransport.scala:192)
[l5d-3m8m2] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
[l5d-3m8m2] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
[l5d-3m8m2] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)
[l5d-3m8m2] at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
[l5d-3m8m2] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
[l5d-3m8m2] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)


#5

@iftachsc That screenshot is consistent with the behavior you are seeing, it should be green like this:

Is this a default GKE cluster? Are you using CNI? Minikube? Depending on the answer, you may need to modify your config files, more info at: Flavors of Kubernetes


#6

many thanks for the response.
first i would say that i tested the grpc tutorial on GKE and added tracing and all was working fine. i am trying to do the same on our on prem.

  1. you said it should be green: what is the meaning when endpoints are found (ip addresses are shown) but its red?
    important to mention that linkerd configuration is the same as i used in the GKE where all worked.

  2. we are using Kubernetes 1.7.0 with Flannel CNI. we are using linkerd with many microservices that communicate succescfully so i am not sure whats the problm. it all started from not be able to do tracing so said - lets clean all and try to get the hellowolrld app so we can proof tracing on grpc works.

after reading a bit more i understood there is difference between the GKE flavour and my on prem which uses CNI (Flannel) so i updated the hostNetwork: true under the two transformers, added the NODE_NAME env var forthe daemon set l5d pods…
but i still get the same issues… i have the endpoints found with red color - and NoBrokersException when try to communicate with my app. i didnt c Flannel listed in the CNI guide only canal/calico/weave. does it mean flannel is not supported? because again we have13 microservices and the succesfully comminicate with one another. just when i try to run this tutorial not on GKE i fail.
any idea?


#7

To answer your questions:

  1. Green means it has gotten a Bound Path, successfully identifying IP addresses to route to. In your red screenshot, there are no IP addresses on the last hop, only in the middle when routing via the linkerd daemonset.
  2. This should still work on Flannel CNI.

Please try the following:

A) Provide us your linkerd and k8s config files.
B) Confirm that the default CNI example works: https://github.com/linkerd/linkerd-examples/blob/master/k8s-daemonset/k8s/linkerd-cni.yml


closed #8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.