Unable to deploy a custom namer as plugin

bethleha · May 24, 2017, 11:51am

Hi,

I have followed linkerd-plugin-dev doc and developed a marathon-namer that resolves apps (services) based on the hostnames.
My configuration is as below:

namers:

kind: io.buoyant.marathonHostNamer
host: mesos.marathon
port: 8080

routers:

protocol: http
servers:

port: 4140
ip: 0.0.0.0
dtab: >-
/marathonId => /#/io.buoyant.marathonHostNamer;
/svc => /marathonId

I get the below log on starting linkerd.

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'io.buoyant.marathonHostNamer' into a subtype of [simple type, class io.buoyant.namer.NamerConfig]: known type ids = [NamerConfig, io.l5d.consul, io.l5d.curator, io.l5d.fs, io.l5d.k8s, io.l5d.k8s.external, io.l5d.marathon, io.l5d.rewrite, io.l5d.serversets, io.l5d.zkLeader]
 at [Source: java.io.StringReader@6b587673; line: 9, column: 9] (through reference chain: io.buoyant.linkerd.Linker$LinkerConfig["namers"]->com.fasterxml.jackson.module.scala.deser.BuilderWrapper[0])
	at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:42)
	at com.fasterxml.jackson.databind.DeserializationContext.unknownTypeIdException(DeserializationContext.java:1477)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1170)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:282)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:156)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:112)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:289)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
	at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:78)
	at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:61)
	at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:60)
	at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:11)
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:511)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:396)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1197)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2880)
	at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.readValue(ScalaObjectMapper.scala:190)
	at io.buoyant.config.Parser$$anon$1.readValue(Parser.scala:79)
	at io.buoyant.linkerd.Linker$.parse(Linker.scala:81)
	at io.buoyant.linkerd.Main$.loadLinker(Main.scala:61)
	at io.buoyant.linkerd.Main$.main(Main.scala:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:176)
	at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:175)
	at scala.Option.foreach(Option.scala:257)
	at com.twitter.app.App$class.nonExitingMain(App.scala:175)
	at io.buoyant.linkerd.Main$.nonExitingMain(Main.scala:20)
	at com.twitter.app.App$class.main(App.scala:141)
	at io.buoyant.linkerd.Main$.main(Main.scala:20)
	at io.buoyant.linkerd.Main.main(Main.scala)
Exception thrown in main on startup

Any help is much appreciated.

Thanks in advance.
Bethleharaja G.

Alex · May 24, 2017, 5:56pm

It looks like linkerd is not loading your plugin. Is your plugin jar in the $L5D_HOME/plugins directory? Are you passing the L5D_HOME env variable to linkerd?

To further debug, you can start linkerd with -log.level=DEBUG and it will print all loaded plugins on startup. Further information on plugin development is here: https://github.com/linkerd/linkerd-examples/tree/master/plugins/header-classifier

bethleha · May 25, 2017, 3:11am

Yes I have set $L5D_HOME and ensure that the jar is in $L5D_HOME/plugins. I have followed the header classfier example and deployed an identifier successfully. But unable to deploy a custom namer.

Alex · May 25, 2017, 4:43pm

Ok. The next thing I’d check is whether linkerd is loading the plugin at all. If you start linkerd with -log.level=DEBUG it will print all loaded plugins on startup. Beyond that, it’s hard to debug without being able to see the plugin jar or the source code.

olix0r · May 25, 2017, 4:48pm

@alex the source code is available on slack: https://files.slack.com/files-pri/T0JV2DX9R-F5HUGT0UW/download/marathon-namer-assembly-0.1-snapshot.jar

I took a brief look just now but nothing jumped out as obviously wrong to me.

bethleha · May 25, 2017, 5:07pm

I set log.level=DEBUG also passed L5D_HOME variable. But I don’t see any Load Service ... log for this namer.

bethleha · May 25, 2017, 6:07pm

@Alex Any update on this …

olix0r · May 31, 2017, 5:01pm

I just noticed that the META-INF file seems to start with an empty line:

$  jar xvf marathon-namer-assembly-0.1-SNAPSHOT.jar META-INF/services/io.buoyant.namer.NamerInitializer
$ cat META-INF/services/io.buoyant.namer.NamerInitializer

io.buoyant.namer.marathon.Marathon2Initializer

I suspect that this might be the source of the problem?

bethleha · June 1, 2017, 3:25am

I corrected the problem and recreated jar (The source is uploaded here https://linkerd.slack.com/files/bethleha/F5MNEPTP1/marathon-namer.zip). I am able to observe that the module is loading given -log.level=DEBUG

LoadService: loaded instance of class io.buoyant.namer.marathon.Marathon2Initializer for requested service io.buoyant.namer.NamerInitializer

But still I am getting the below error;

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id ‘io.buoyant.marathonHostNamer’ into a subtype of [simple type, class io.buoyant.namer.NamerConfig]: known type ids = [NamerConfig, io.l5d.consul, io.l5d.curator, io.l5d.fs, io.l5d.k8s, io.l5d.k8s.external, io.l5d.marathon, io.l5d.rewrite, io.l5d.serversets, io.l5d.zkLeader]

I observe that the service is loading but when I use the marathonHostNamer as in the below config

namers:

kind: io.buoyant.marathonHostNamer
host: marathon.mesos
port: 8080

routers:

protocol: http
identifier:
kind: io.l5d.methodAndHost
httpUriInDst: true
dtab: |
/marathonId => /#/io.l5d.marathon;
/svc/1.1/* => /marathonId;
servers:

port: 4140
ip: 0.0.0.0

Full stack trace of the errror is;

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id ‘io.buoyant.marathonHostNamer’ into a subtype of [simple type, class io.buoyant.namer.NamerConfig]: known type ids = [NamerConfig, io.l5d.consul, io.l5d.curator, io.l5d.fs, io.l5d.k8s, io.l5d.k8s.external, io.l5d.marathon, io.l5d.rewrite, io.l5d.serversets, io.l5d.zkLeader]
at [Source: java.io.StringReader@878537d; line: 2, column: 10] (through reference chain: io.buoyant.linkerd.Linker$LinkerConfig[“namers”]->com.fasterxml.jackson.module.scala.deser.BuilderWrapper[0])
at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:42)
at com.fasterxml.jackson.databind.DeserializationContext.unknownTypeIdException(DeserializationContext.java:1477)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1170)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:282)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:156)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:112)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:289)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:78)
at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:61)
at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:60)
at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:11)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:511)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:396)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1197)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2880)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.readValue(ScalaObjectMapper.scala:190)
at io.buoyant.config.Parser$$anon$1.readValue(Parser.scala:79)
at io.buoyant.linkerd.Linker$.parse(Linker.scala:81)
at io.buoyant.linkerd.Main$.loadLinker(Main.scala:61)
at io.buoyant.linkerd.Main$.main(Main.scala:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:176)
at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:175)
at scala.Option.foreach(Option.scala:257)
at com.twitter.app.App$class.nonExitingMain(App.scala:175)
at io.buoyant.linkerd.Main$.nonExitingMain(Main.scala:20)
at com.twitter.app.App$class.main(App.scala:141)
at io.buoyant.linkerd.Main$.main(Main.scala:20)
at io.buoyant.linkerd.Main.main(Main.scala)
Exception thrown in main on startup

Thanks in advance,
Bethleharaja G.

olix0r · June 1, 2017, 3:42pm

It’s great that we’re seeing the LoadService line now!

It strikes me that you’re defining a io.buoyant.namer.marathon.MarathonConfig type, which conflicts with another class on the classpath (defined in linkerd). I’d try moving the whole project out of io.buoyant to a separate namespace like com.bethleha.linkerd to avoid namespace collision. At the very least, it will make things slightly simpler to debug

bethleha · June 2, 2017, 3:25am

When I changed the namespace, I didn’t see the LoadService line of the module in the log and facing the same problem.

klingerf · June 2, 2017, 6:13pm

Hey @bethleha – I grabbed your source code and was able to successfully build and load it as a linkerd plugin. To do so, I had to add an sbt build file and an sbt plugin for generating an assembly jar. I also reorganized your project to put everything in a separate com.bethleha package. Here’s the updated source code: https://linkerd.slack.com/files/kl/F5NQUPD2B/marathon-namer.zip

Once you’ve downloaded the update source code, build the plugin jar by running: ./sbt assembly. That will produce a jar file in target/scala-2.12/marathon-namer-assembly-0.1-SNAPSHOT.jar.

After that, download the the latest linkerd release tarball, and extract it to a new directory, with the following commands:

wget https://github.com/linkerd/linkerd/releases/download/1.0.2/linkerd-1.0.2.tgz
tar xzvf linkerd-1.0.2.tgz
cd linkerd-1.0.2

Once you’re in the linkerd directory, you’ll need to create a new directory, plugins, and copy the plugin jar that you build into that directory. Something like:

mkdir plugins
cp <marathon-namer-project-path>/target/scala-2.12/marathon-namer-assembly-0.1-SNAPSHOT.jar plugins

And finally, start linkerd with the L5D_HOME environment variable set and debug logging enabled:

L5D_HOME=`pwd` ./linkerd-1.0.2-exec config/linkerd.yaml -log.level=DEBUG

When I run that command, I see the following line in the linkerd log:

D 0602 18:11:19.009 UTC THREAD1: LoadService: loaded instance of class com.bethleha.linkerd.marathon.Marathon2Initializer for requested service io.buoyant.namer.NamerInitializer

And that indicates that the plugin has successfully loaded. Give it a shot and let us know if you’re able to get everything working.

bethleha · June 7, 2017, 9:10am

@klingerf Thanks a lot it worked finally for me. Special thanks to @olix0r and Alex .