Unable to deploy a custom namer as plugin

Hi,

I have followed linkerd-plugin-dev doc and developed a marathon-namer that resolves apps (services) based on the hostnames.
My configuration is as below:

namers:

  • kind: io.buoyant.marathonHostNamer
    host: mesos.marathon
    port: 8080

routers:

  • protocol: http
    servers:
    • port: 4140
      ip: 0.0.0.0
      dtab: >-
      /marathonId => /#/io.buoyant.marathonHostNamer;
      /svc => /marathonId

I get the below log on starting linkerd.

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'io.buoyant.marathonHostNamer' into a subtype of [simple type, class io.buoyant.namer.NamerConfig]: known type ids = [NamerConfig, io.l5d.consul, io.l5d.curator, io.l5d.fs, io.l5d.k8s, io.l5d.k8s.external, io.l5d.marathon, io.l5d.rewrite, io.l5d.serversets, io.l5d.zkLeader]
 at [Source: java.io.StringReader@6b587673; line: 9, column: 9] (through reference chain: io.buoyant.linkerd.Linker$LinkerConfig["namers"]->com.fasterxml.jackson.module.scala.deser.BuilderWrapper[0])
	at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:42)
	at com.fasterxml.jackson.databind.DeserializationContext.unknownTypeIdException(DeserializationContext.java:1477)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1170)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:282)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:156)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:112)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:289)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
	at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:78)
	at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:61)
	at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:60)
	at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:11)
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:511)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:396)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1197)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2880)
	at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.readValue(ScalaObjectMapper.scala:190)
	at io.buoyant.config.Parser$$anon$1.readValue(Parser.scala:79)
	at io.buoyant.linkerd.Linker$.parse(Linker.scala:81)
	at io.buoyant.linkerd.Main$.loadLinker(Main.scala:61)
	at io.buoyant.linkerd.Main$.main(Main.scala:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:176)
	at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:175)
	at scala.Option.foreach(Option.scala:257)
	at com.twitter.app.App$class.nonExitingMain(App.scala:175)
	at io.buoyant.linkerd.Main$.nonExitingMain(Main.scala:20)
	at com.twitter.app.App$class.main(App.scala:141)
	at io.buoyant.linkerd.Main$.main(Main.scala:20)
	at io.buoyant.linkerd.Main.main(Main.scala)
Exception thrown in main on startup

Any help is much appreciated.

Thanks in advance.
Bethleharaja G.

It looks like linkerd is not loading your plugin. Is your plugin jar in the $L5D_HOME/plugins directory? Are you passing the L5D_HOME env variable to linkerd?

To further debug, you can start linkerd with -log.level=DEBUG and it will print all loaded plugins on startup. Further information on plugin development is here: https://github.com/linkerd/linkerd-examples/tree/master/plugins/header-classifier

Yes I have set $L5D_HOME and ensure that the jar is in $L5D_HOME/plugins. I have followed the header classfier example and deployed an identifier successfully. But unable to deploy a custom namer.

Ok. The next thing I’d check is whether linkerd is loading the plugin at all. If you start linkerd with -log.level=DEBUG it will print all loaded plugins on startup. Beyond that, it’s hard to debug without being able to see the plugin jar or the source code.

@alex the source code is available on slack: https://files.slack.com/files-pri/T0JV2DX9R-F5HUGT0UW/download/marathon-namer-assembly-0.1-snapshot.jar

I took a brief look just now but nothing jumped out as obviously wrong to me.

I set log.level=DEBUG also passed L5D_HOME variable. But I don’t see any Load Service ... log for this namer.

@Alex Any update on this …

I just noticed that the META-INF file seems to start with an empty line:

$  jar xvf marathon-namer-assembly-0.1-SNAPSHOT.jar META-INF/services/io.buoyant.namer.NamerInitializer
$ cat META-INF/services/io.buoyant.namer.NamerInitializer

io.buoyant.namer.marathon.Marathon2Initializer

I suspect that this might be the source of the problem?

I corrected the problem and recreated jar (The source is uploaded here https://linkerd.slack.com/files/bethleha/F5MNEPTP1/marathon-namer.zip). I am able to observe that the module is loading given -log.level=DEBUG

LoadService: loaded instance of class io.buoyant.namer.marathon.Marathon2Initializer for requested service io.buoyant.namer.NamerInitializer

But still I am getting the below error;

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id ‘io.buoyant.marathonHostNamer’ into a subtype of [simple type, class io.buoyant.namer.NamerConfig]: known type ids = [NamerConfig, io.l5d.consul, io.l5d.curator, io.l5d.fs, io.l5d.k8s, io.l5d.k8s.external, io.l5d.marathon, io.l5d.rewrite, io.l5d.serversets, io.l5d.zkLeader]

I observe that the service is loading but when I use the marathonHostNamer as in the below config

namers:

  • kind: io.buoyant.marathonHostNamer
    host: marathon.mesos
    port: 8080

routers:

  • protocol: http
    identifier:
    kind: io.l5d.methodAndHost
    httpUriInDst: true
    dtab: |
    /marathonId => /#/io.l5d.marathon;
    /svc/1.1/* => /marathonId;
    servers:
    • port: 4140
      ip: 0.0.0.0

Full stack trace of the errror is;

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id ‘io.buoyant.marathonHostNamer’ into a subtype of [simple type, class io.buoyant.namer.NamerConfig]: known type ids = [NamerConfig, io.l5d.consul, io.l5d.curator, io.l5d.fs, io.l5d.k8s, io.l5d.k8s.external, io.l5d.marathon, io.l5d.rewrite, io.l5d.serversets, io.l5d.zkLeader]
at [Source: java.io.StringReader@878537d; line: 2, column: 10] (through reference chain: io.buoyant.linkerd.Linker$LinkerConfig[“namers”]->com.fasterxml.jackson.module.scala.deser.BuilderWrapper[0])
at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:42)
at com.fasterxml.jackson.databind.DeserializationContext.unknownTypeIdException(DeserializationContext.java:1477)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1170)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:282)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:156)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:112)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:289)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:78)
at com.fasterxml.jackson.module.scala.deser.SeqDeserializer.deserialize(SeqDeserializerModule.scala:61)
at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:60)
at com.fasterxml.jackson.module.scala.deser.OptionDeserializer.deserialize(OptionDeserializerModule.scala:11)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:511)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:396)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1197)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2880)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.readValue(ScalaObjectMapper.scala:190)
at io.buoyant.config.Parser$$anon$1.readValue(Parser.scala:79)
at io.buoyant.linkerd.Linker$.parse(Linker.scala:81)
at io.buoyant.linkerd.Main$.loadLinker(Main.scala:61)
at io.buoyant.linkerd.Main$.main(Main.scala:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:176)
at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:175)
at scala.Option.foreach(Option.scala:257)
at com.twitter.app.App$class.nonExitingMain(App.scala:175)
at io.buoyant.linkerd.Main$.nonExitingMain(Main.scala:20)
at com.twitter.app.App$class.main(App.scala:141)
at io.buoyant.linkerd.Main$.main(Main.scala:20)
at io.buoyant.linkerd.Main.main(Main.scala)
Exception thrown in main on startup

Thanks in advance,
Bethleharaja G.

It’s great that we’re seeing the LoadService line now!

It strikes me that you’re defining a io.buoyant.namer.marathon.MarathonConfig type, which conflicts with another class on the classpath (defined in linkerd). I’d try moving the whole project out of io.buoyant to a separate namespace like com.bethleha.linkerd to avoid namespace collision. At the very least, it will make things slightly simpler to debug :slight_smile:

When I changed the namespace, I didn’t see the LoadService line of the module in the log and facing the same problem.

Hey @bethleha – I grabbed your source code and was able to successfully build and load it as a linkerd plugin. To do so, I had to add an sbt build file and an sbt plugin for generating an assembly jar. I also reorganized your project to put everything in a separate com.bethleha package. Here’s the updated source code: https://linkerd.slack.com/files/kl/F5NQUPD2B/marathon-namer.zip

Once you’ve downloaded the update source code, build the plugin jar by running: ./sbt assembly. That will produce a jar file in target/scala-2.12/marathon-namer-assembly-0.1-SNAPSHOT.jar.

After that, download the the latest linkerd release tarball, and extract it to a new directory, with the following commands:

wget https://github.com/linkerd/linkerd/releases/download/1.0.2/linkerd-1.0.2.tgz
tar xzvf linkerd-1.0.2.tgz
cd linkerd-1.0.2

Once you’re in the linkerd directory, you’ll need to create a new directory, plugins, and copy the plugin jar that you build into that directory. Something like:

mkdir plugins
cp <marathon-namer-project-path>/target/scala-2.12/marathon-namer-assembly-0.1-SNAPSHOT.jar plugins

And finally, start linkerd with the L5D_HOME environment variable set and debug logging enabled:

L5D_HOME=`pwd` ./linkerd-1.0.2-exec config/linkerd.yaml -log.level=DEBUG

When I run that command, I see the following line in the linkerd log:

D 0602 18:11:19.009 UTC THREAD1: LoadService: loaded instance of class com.bethleha.linkerd.marathon.Marathon2Initializer for requested service io.buoyant.namer.NamerInitializer

And that indicates that the plugin has successfully loaded. Give it a shot and let us know if you’re able to get everything working.

1 Like

@klingerf Thanks a lot it worked finally for me. Special thanks to @olix0r and Alex .