Client TLS with kind io.l5d.static can not work on K8S


#1

Hey guys,

Linkerd version: 1.3.6

I found Client TLS with kind io.l5d.static can’t work on K8S, only kind io.l5d.global works. Here is my Linkerd config:

    admin:
      ip: 0.0.0.0
      port: 9990
    namers:
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.out
      host: localhost
      port: 8001
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: default
        port: incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.in
      host: localhost
      port: 8001
    - kind: io.l5d.rewrite
      prefix: /portNsSvcToK8s
      pattern: "/{port}/{ns}/{svc}"
      name: "/k8s/{ns}/{port}/{svc}"
    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25
    usage:
      enabled: false
    routers:
    - protocol: http
      label: outgoing
      dtab: |
        /k8s          =>   /#/io.l5d.k8s.out;
        /portNsSvc    =>   /#/portNsSvcToK8s;
        /host         =>   /portNsSvc/http/default;
        /host         =>   /portNsSvc/http;
        /svc          =>   /$/io.buoyant.http.domainToPathPfx/host;
      servers:
      - port: 4140
        ip: 0.0.0.0
      client:
        kind: io.l5d.static
        configs:
        - prefix: /#/io.l5d.k8s.out
          tls:
            commonName: linkerd
            trustCerts:
            - /io.buoyant/linkerd/tls/ca.crt
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX
    - protocol: http
      label: incoming
      dtab: |
        /k8s          =>   /#/io.l5d.k8s.in;
        /portNsSvc    =>   /#/portNsSvcToK8s;
        /host         =>   /portNsSvc/http/default;
        /host         =>   /portNsSvc/http;
        /svc          =>   /$/io.buoyant.http.domainToPathPfx/host;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      servers:
      - port: 4141
        ip: 0.0.0.0
        tls:
          certPath: /io.buoyant/linkerd/tls/linkerd.crt
          keyPath: /io.buoyant/linkerd/tls/linkerd.key

After executing this command:

curl -v -H "Host: user" http://192.168.1.12:4140/users/tom/bookings
* About to connect() to 192.168.1.12 port 4140 (#0)
*   Trying 192.168.1.12...
* Connected to 192.168.1.12 (192.168.1.12) port 4140 (#0)
> GET /users/tom/bookings HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: user
>
< HTTP/1.1 502 Bad Gateway
< l5d-err: ChannelException+at+remote+address%3A+%2F192.168.1.12%3A4141+from+service%3A+%25%2Fio.l5d.k8s.daemonset%2Fdefault%2Fincoming%2Fl5d%2F%23%2Fio.l5d.k8s.out%2Fdefault%2Fhttp%2Fuser.+Remote+Info%3A+Upstream+Address%3A+Not+Available%2C+Upstream+id%3A+Not+Available%2C+Downstream+Address%3A+%2F192.168.1.12%3A4141%2C+Downstream+label%3A+%25%2Fio.l5d.k8s.daemonset%2Fdefault%2Fincoming%2Fl5d%2F%23%2Fio.l5d.k8s.out%2Fdefault%2Fhttp%2Fuser%2C+Trace+Id%3A+b5359d4235712c1a.ddfcdc9089e81fb1%3C%3Ab5359d4235712c1a
< Content-Type: text/plain
< Content-Length: 422
<
* Connection #0 to host 192.168.1.12 left intact
ChannelException at remote address: /192.168.1.12:4141 from service: %/io.l5d.k8s.daemonset/default/incoming/l5d/#/io.l5d.k8s.out/default/http/user. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: /192.168.1.12:4141, Downstream label: %/io.l5d.k8s.daemonset/default/incoming/l5d/#/io.l5d.k8s.out/default/http/user, Trace Id: b5359d4235712c1a.ddfcdc9089e81fb1<:b5

Linkerd logs like:

W 0427 06:56:41.435 UTC THREAD25: Unhandled exception in connection with /192.168.1.12:55624, shutting down connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f75736572732f746f6d2f626f6f6b696e677320485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a4163636570743a202a2f2a0d0a486f73743a20757365720d0a636f6e74656e742d6c656e6774683a20300d0a6c35642d6473742d736572766963653a202f7376632f757365720d0a5669613a20312e31206c696e6b6572640d0a6c35642d6473742d636c69656e743a202f252f696f2e6c35642e6b38732e6461656d6f6e7365742f64656661756c742f696e636f6d696e672f6c35642f232f696f2e6c35642e6b38732e6f75742f64656661756c742f687474702f757365720d0a6c35642d6374782d74726163653a206a30766d764831566c6a4c717777633078575144692b7244427a54465a414f4c41414141414141414141413d0d0a6c35642d72657169643a20656163333037333463353634303338620d0a0d0a
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f75736572732f746f6d2f626f6f6b696e677320485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a4163636570743a202a2f2a0d0a486f73743a20757365720d0a636f6e74656e742d6c656e6774683a20300d0a6c35642d6473742d736572766963653a202f7376632f757365720d0a5669613a20312e31206c696e6b6572640d0a6c35642d6473742d636c69656e743a202f252f696f2e6c35642e6b38732e6461656d6f6e7365742f64656661756c742f696e636f6d696e672f6c35642f232f696f2e6c35642e6b38732e6f75742f64656661756c742f687474702f757365720d0a6c35642d6374782d74726163653a206a30766d764831566c6a4c717777633078575144692b7244427a54465a414f4c41414141414141414141413d0d0a6c35642d72657169643a20656163333037333463353634303338620d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	... 18 more
WARN 0427 06:56:41.471 UTC finagle/netty4-3: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f75736572732f746f6d2f626f6f6b696e677320485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a4163636570743a202a2f2a0d0a486f73743a20757365720d0a636f6e74656e742d6c656e6774683a20300d0a6c35642d6473742d736572766963653a202f7376632f757365720d0a5669613a20312e31206c696e6b6572640d0a6c35642d6473742d636c69656e743a202f252f696f2e6c35642e6b38732e6461656d6f6e7365742f64656661756c742f696e636f6d696e672f6c35642f232f696f2e6c35642e6b38732e6f75742f64656661756c742f687474702f757365720d0a6c35642d6374782d74726163653a202b503937586e3972774c7a717777633078575144692b7244427a54465a414f4c41414141414141414141413d0d0a6c35642d72657169643a20656163333037333463353634303338620d0a0d0a
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

It seems client doesn’t send unencrypted traffic, i don’t know why this happened, is something wrong?

Thanks,
Clare


#2

Hi @yangzhares, it looks like linkerd is unable to coincide the client name generated through the routing process and the prefix name set in your linkerd config. As you can tell in the error message, linkerd is trying to use the name %/io.l5d.k8s.daemonset/default/incoming/l5d/#/io.l5d.k8s.out to initiate TLS but your config says that all client TLS should be applied on names that start with /#/l5d.k8s.out. The former name is a result of transformers being prepended during the request identification phase. Because of this, Linkerd won’t send encrypted traffic to the user service since the prefix is set to initiate TLS for service names that begin with /#io.l5d.k8s.out. What you would need to do is prepend all transformers used in the config in the prefix value, in the order in which they appear. That might hopefully resolve the issue.

This behavior is pretty subtle, so, I went ahead and created an issue so it could be highlighted in the Linkerd documentation. Feel free to add any additional information that you may find useful.


#3

thanks @dennis.ab, follow your advice, update client tls configuration like:

client:
        kind: io.l5d.static
        configs:
        - prefix: /#/io.l5d.k8s
          tls:
            commonName: linkerd
            trustCerts:
            - /io.buoyant/linkerd/tls/ca.crt
        - prefix: "%/io.l5d.k8s.daemonset"
          tls:
            commonName: linkerd
            trustCerts:
            - /io.buoyant/linkerd/tls/ca.crt

it definitely works. As you said, this should be explicitly included in docs, what’s more, i think if Transformer is applied on outgoing router and TLS is enabled, when configure client tls as kind io.l5d.static, prefix should be started with "%/$transformer_kind".


#4

Awesome! Glad to hear that you got everything working. Be sure to look out for the update in the linkerd configuration docs.