Creating a new namer for the Rancher platform


#1

Hi,

I have been interested playing around with Linkerd, but we use Rancher for our orchestration which there currently isn’t a namer for. It seems I’m not alone, since there is an issue about adding support for it: https://github.com/linkerd/linkerd/issues/1146

I’ve never written a line of Scala - but I was fairly adamant about not letting small details like that stop me. So I set out yesterday, and I think I have something roughly in the right direction now. But there’s something I don’t understand about how Linkerd/Finagle works. But do keep in my that I have a veeeeery basic understanding of the intricacies of Scala when you reply :slight_smile:

So first things first - for those that don’t know, Rancher organises containers into stacks and services - a service consists of one or more instances of the same container, and stacks are used to organise one ore more services into a logical unit (and is basically what your describe using docker-compose.yml.)

So, I’ve created a case class for containing information about a single container running in Rancher (which includes the ip and the various exposed ports). I’ve also created a class that uses a Finagle Service to query the metadata-API and used the Jackson JSON parser to get the response into the above mentioned case class.
This class ends up exposing a Activity[List[RancherContainer]] instance based on Var.async - right now it simply requeries the API every n seconds, but in time should use the long-polling supported by the Rancher metadata-API to get more real-time updates.

In the namer itself, I’ve added the lookup(path: Path):Activity[NameTree[Name]] method so it uses a .transform() on the above mentioned Activity[List[RancherContainer]] to transform the List[RancherContainer]] into a NameTree[Name].
I do this by popping the first two parts of the path off into stack and service and then filter the list of contains down to those from that stack and service. I then create a set of Address's based on the ip and port of those containers.
My transform then simply returns

NameTree.Leaf(
  Name.Bound(
    Var.value(Addr.Bound(addresses)),
    prefix ++ path.take(2),
    path.drop(2)
  )
)

However, even when my set of addresses change change (I have a println inside my .transform which gets called every n seconds), Linkerd doesn’t update the destination of the path. I’ve been testing locally with a mock metadata-API and manually editing the response between two ports on my local host - so it’s easy to see that it keeps using the initial port.

Does anyone have any hints as to what I could be doing wrong? Is it because the Name.Bound I return ought to be a Var.async instead of a Var.value, or? The lookup-method doesn’t seem to be called again, so clearly the NameTree I return needs to update itself - but how do I achieve that?

I can also push my very Work-in-Progress work to a repo if that makes it easier to help out!

(Yes, I know it’s a bit silly implementing all these things for Rancher when their next version will be Kubernetes-based, but it seemed like a fun weekend project)

Kind regards
Morten


#2

Hi Morten!

Thanks for looking into this, having a Rancher namer would be super awesome! I’d definitely love to take a look at your branch; it should be much easier for me to point you in the right direction that way.


#3

Great. I’ll try and whip up a repo with a docker-compose.yml that contains Linkerd with the namer and a mock Rancher Metadata-API so it will be easy to see what’s going on…


#4

I’ve just done a standalone project because I didn’t want to figure out how to compile all of Linkerd - I used the example plugin as a starting-point.

The plugin is located in the following repo: https://github.com/fangel/linkerd-rancher-namer

Instructions for how to build and run the examples are given in the README. Let me know if something is unclear or doesn’t work out.


#5

@fangel very cool! Thanks for putting that together. We’ll check it out.


#6

Thanks for sharing the code. First of all, I have to say that I’m super impressed with what you have so far. Really nice work.

You’ve run into one of the most subtle and tricky aspects of Finagle/Linkerd which is the two-level caching that it does. Namers return an Activity[NameTree] which represents how a name resolves to a client (ie a Name.Bound). This Activity typically will only update if there is a dtab change or if a service stops existing or starts existing. Inside each Name.Bound is another Var[Addr] that represents the address set for that client. This inner Var will update each time the address set changes. The goal of this two-level caching is that we don’t want to rebuild the entire client every time the address set changes.

So you’ll need to do some gymnastics on your Activity[List[RancherContainer]] to get an Activity[NameTree[Name.Bound]] where the outer Activity only updates when the requested container is created or deleted and the inner Var only updates when the address set of that container changes. Thankfully, we have a utility method that does some of the heavy lifting.

I think what you want should be something like this (pseudo-code):

val container: Var[Option[RancherContainer]] = client.activity.map { allContainers =>
  // Some(container) if the requested container is in the list
  // None otherwise
}
val stabilized: Var[Option[Var[RancherContainer]]] = container.stabilizeExistence
stabilized.map {
  case Some(vContainer) => 
    val vaddr: Var[Addr] = vContainer.map(_.toAddr)
    NameTree.Leaf(Name.Bound(vaddr, ....))
  case None => NameTree.Neg
}

Hopefully this makes sense. Please let me know if you have any further questions!


#7

Thanks Alex!

I’ll see if I have some time to play around with it tonight - but a cursory read-through of the hint seems helpful, so I’ll give it a whirl.

So my initial idea that Name.Bound needs to be a async Var wasn’t completely off - and nice to hear that there is some infrastructure in place to help out with that part too.

Just to confirm that I’m not completely off on the architecture: The Activity that is returned from lookup is only sampled every so often for performance - but the Var inside Name.Bound is sampled at every request? And currently my thing changes the whole activity, but that isn’t resampled, and thus not registered? Is that a reasonable understanding of the setup?


#8

Both the outer Activity and inner Var are actually never sampled, but instead they are watched for updates. The difference between them is that the outer Activity only changes when the structure of the NameTree changes (eg when the dtab changes or when the service is created or removed) whereas the inner Var changes whenever the address set for that service changes.

In it’s current state, your code returns an Activity that changes each time the Rancher API is polled and an inner Var that never changes.


#9

Okay, I finally found some time to test out your suggestions - and there has been progress! The only thing with your pseudo-code is that the type is Activity[Option[Var[..]]] not Var[Option[Var[...]]].
And I had to figure out how to get ExistentialStability in as a dependency in sbt because the repo is done as a plugin, not a branch of Linkerd.

In my mock Docker-Compose based example it will now actually update the address for the endpoint(s), but I haven’t had time to experiment on running it in Rancher yet, but I have high hopes. I’ll see if I have an idle moment tomorrow - otherwise maybe over the weekend.


I have two further questions that I would love some input on:

  1. A service in Rancher is identified by a tuple of (stack, service) - what is the best way of providing those two parts? An identifier that takes a host-name of stack.service and transform it into svc/stack/service, or? (And if so, does such a identifier already exist?)
  2. How do you normally deal with a service that exposes multiple ports? Right now my prototype always just uses the first exposed one, but what is the “normal”/expected/suggested way of dealing with that? Should I actually identify a service by a tuple of (stack, service, port)?

And again, thanks for your help - I wouldn’t have figured it out myself!


#10

Excellent!

It’s helpful to remember that Activity is just a thin wrapper around Var[Activity.State]. You can always get the Var out of an Activity by calling activity.run and you can always build an Activity from the Var with Activity.apply.

I’d recommend that the namer accept names of the form /#/prefix/<port>/<stack>/<service>. This is very similar to the Kubernetes namer which accepts /#/io.l5d.k8s/<namespace>/<port>/<service>.

You can then use a dtab like:

/rancher => /#/io.l5d.rancher/http
/svc => /$/io.buoyant.http.domainToPathPfx/rancher

which will map names like /svc/service.stack to /#/io.l5d.rancher/http/stack/service.


#11

Okay, so Rancher doesn’t really have named ports. So I’ve added a mapping of service → port that users can extend using the namer-config. Otherwise, you can always do /#/io.l5d.rancher/80/stack/service. That seems to work quite well!

I’ve updated my mock docker-compose to have 2 services, one with two containers in it, and one with a single service in it.
I’ve also added in a Dtab-alias that load-balances between the two.

However, I then tried to do it as a fail-over using /s/sample-stack/sample-service => /s/sample-stack/sample-service2 | /s/sample-stack/sample-service1;.
Then all requests goes to the sample-service2-container. If I then stop that service using docker-compose stop sample-service2 I would expect Linkerd to consider it down and switch over to /s/sample-stack/sample-service1, however all I get is Linkerd complaining about the failure:

service failure: Failure(connection timed out: sample-service2/172.22.0.3:80 at remote address: sample-service2/172.22.0.3:80.
  Remote Info: Not Available, flags=0x09) with RemoteInfo → Upstream
  Address: Not Available, 
  Upstream id: Not Available,
  Downstream Address: sample-service2/172.22.0.3:80,
  Downstream label: #/io.l5d.rancher/http/sample-stack/sample-service2,

But it still doesn’t start routing the traffic to the next route…

Is that due to a mistake in my Namer, or in my understanding of how Linkerd configuration works? :slight_smile:


And now I just need to figure out how to best use the long-polling aspect of the metadata-API to get the updates to be more push than pull…


#12

What you are describing is the expected behavior. Linkerd will only evaluate the next branch of a fallback if the primary branch stops existing (ie that RancherContainer is no longer returned from the Rancher API).


#13

That explains it, then. I guess I just haven’t read the documentation carefully enough.

Is there anyway to get that behaviour? (I was thinking maybe for something like a canary deploy kinda situation where you want to old version to take over if the canary died)


Anyways, I’ll look into improving the API client, and then I guess I’ll try to update it to be an actual branch of Linkerd if this namer is something you would consider adding…


#14

Just a quick status update: I’ve actually gotten it running in Rancher now, and having it autodiscover the services and update the available endpoints when you scale a service up or down through the Rancher admin.

Still need to work on getting it to live-update using the long-polling API. So the long-polling API is actually a separate API-endpoint that tracks a arbitrary version-identifier for the current known data. You can then call an endpoint with the last version you know, and a max waiting time. It will then reply if the version-number changes or the wait is up.

My thinking was to create an Activity that simply updates every time this version-id changes - and then use .map to then query the endpoint I’m currently using to get the list of containers (so basically having a Activity[RancherVersion] and then map that into a Activity[List[RancherContainer]] and then continue like it currently does. Does that seem like a decent way of building this, or?


#15

Awesome to hear this is going well!

That sounds like a totally reasonable approach. I believe that the Consul namer does something very similar with respect to long-polling, although the API returns the actual objects rather than just the version.


#16

Having the long-polling return the actual objects would make sense, yes. But the Rancher Metadata API is… interesting. But to make up for it, is not very well documented either - so it got that going for it.

(As in, their documentation doesn’t even mention the latest version of the API that’s available - nor does it mention the long-polling functionality)


I’ve been doing a lot of poking around in the Scala REPL to make progress - but I can’t seem to get the Activity's to run continuously there - is there a weird trick I don’t know of? If I wanted to simply print something, whenever something changes, is that activity.run.respond { value => println(value) } or something to that effect?


#17

Vars that are created with Var.async are reference-counted. What this means is that the body of the Var.async doesn’t run until there’s an observation of that Var. Likewise, the closable doesn’t run until all observations have closed.

You can create an observation on a Var with something like:

val closable = myVar.changes.respond(println)
// Var.async should now be running
closable.close() // if there are no other observations, this will invoke the Var.async's closable

#18

Okay - it’s been some busy weeks at work, so I haven’t had a change to work more on this - till today! I have gotten the long-polling functionality working now, and it seems to be working out quite well.

What’s the best way to progress from here? Should I make a branch of Linkerd and move my work there, and then we can start a review-process so you can help me fix up all my Scala-rookie mistakes?


#19

Hi @fangel. Yup, we’re interested in seeing this get into linkerd proper. A few things to note:

  1. Yes, work on a branch of linkerd for us to review.
  2. Ensure your namer requires the experimental flag (https://linkerd.io/config/1.3.2/linkerd/index.html#namers).
  3. Minimize dependencies in your plugin, though this looks quite good if it’s the final form: https://github.com/fangel/linkerd-rancher-namer/blob/master/build.sbt
  4. Add a section to Namer docs: https://github.com/linkerd/linkerd/blob/master/linkerd/docs/namer.md
  5. Bonus points for a new config in the linkerd-examples.

Thanks for doing this, awesome work!


#20

Thanks for the checklist of things to do! I’ll try and take a look at it over the weekend and see if I can get a branch working.
And no - dependency-wise, I don’t think I’ve introduced anything that wasn’t already in Linkerd.