K8s: Unable to route to endpoints with TLS configuration

https://buoyant.io/2016/10/24/a-service-mesh-for-kubernetes-part-iii-encrypting-all-the-things

I followed this blog post on how to set up the TLS config using openssl and linkerd on k8s environment. I am able to generate the certificates and sign them with no issues. I did the Secrets deployment and verified if certs are copied to the l5d container in k8s.
I have a service with name “service1” so I created the the certificate with this name.

I am unable to route through service1 over http/https. I am able to request and get the response with the dns of the service and I checked the dtabs on the admin, they are resolvable.

http_proxy=$L5D_INGRESS:4140 curl https://service1
when I did this, I see below error
null at remote address: /10.31.64.5:4141. Remote Info: Not Available

Hi @zshaik

I believe most libraries will ignore the http_proxy variable for HTTPS requests. Some may respect a https_proxy variable but this is not universal.

Taking a step back though, the benefit of Linkerd in this case is that it can do the TLS upgrade for you and your app can speak plain HTTP to Linkerd. To simulate this, the curl command would be http_proxy=$L5D_INGRESS:4140 curl http://service1.

Does this work for you? Or is there a reason why you need your application to speak HTTPS to Linkerd? If that’s the case, you’ll need to add a new server to your Linkerd config that accepts TLS traffic.

1 Like

Hey @alex, I am unable to route through http as well. I get the same response.
http_proxy=$L5D_INGRESS:4140 curl http://service1
null at remote address: /10.31.64.5:4141. Remote Info: Not Available

I don’t know it this will help, but here is my version of the TLS yaml that works with a cluster that has Calico installed (ie CNI).

Note: Don’t forget to uncomment the LoadBalancer section the first time you run this.

# runs linkerd in a daemonset, in linker-to-linker mode
# for CNI setups such as Calico and Weave
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: l5d-config
data:
  config.yaml: |-
    admin:
      port: 9990

    namers:
    - kind: io.l5d.k8s
      experimental: true
      host: localhost
      port: 8001

    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25

    usage:
      orgId: linkerd-examples-daemonset-tls-cni

    routers:
    - protocol: http
      label: outgoing
      dtab: |
        /srv        => /#/io.l5d.k8s/uodonle/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.daemonset
          namespace: default
          port: incoming
          service: l5d
          hostNetwork: true
      servers:
      - port: 4140
        ip: 0.0.0.0
      client:
        tls:
          commonName: linkerd
          trustCerts:
          - /io.buoyant/linkerd/certs/cacertificate.pem
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX

    - protocol: http
      label: incoming
      dtab: |
        /srv        => /#/io.l5d.k8s/uodonle/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      servers:
      - port: 4141
        ip: 0.0.0.0
        tls:
          certPath: /io.buoyant/linkerd/certs/certificate.pem
          keyPath: /io.buoyant/linkerd/certs/key.pem
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: l5d
  name: l5d
spec:
  template:
    metadata:
      labels:
        app: l5d
    spec:
      hostNetwork: true
      volumes:
      - name: l5d-config
        configMap:
          name: "l5d-config"
      - name: certificates
        secret:
          secretName: certificates
      containers:
      - name: l5d
        image: buoyantio/linkerd:1.1.2
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - /io.buoyant/linkerd/config/config.yaml
        ports:
        - name: outgoing
          containerPort: 4140
          hostPort: 4140
        - name: incoming
          containerPort: 4141
          hostPort: 4141
        - name: admin
          containerPort: 9990
          hostPort: 9990
        volumeMounts:
        - name: "l5d-config"
          mountPath: "/io.buoyant/linkerd/config"
          readOnly: true
        - name: "certificates"
          mountPath: "/io.buoyant/linkerd/certs"
          readOnly: true

      - name: kubectl
        image: buoyantio/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
# Uncomment this to create the LoadBalancer
# ---
# apiVersion: v1
# kind: Service
# metadata:
#   name: l5d
# spec:
#   selector:
#     app: l5d
#   type: LoadBalancer
#   ports:
#   - name: outgoing
#     port: 4140
#   - name: incoming
#     port: 4141
#   - name: admin
#     port: 9990

Is there anything in the Linkerd logs? Either the Linkerd that you’re sending to, or the one running on 10.31.64.5?

Hey @Alex ! There is a log which I just checked says that certificate is invalid!?

WARN 0721 17:42:21.908 UTC finagle/netty4-3: Failed to initialize a channel. Closing: [id: 0x76a6a818]
java.lang.IllegalArgumentException: File does not contain valid certificates: /tmp/certCollection7634975048145713661.tmp
        at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:179)
        at com.twitter.finagle.netty4.ssl.Netty4SslConfigurations$.configureTrust(Netty4SslConfigurations.scala:40)
        at com.twitter.finagle.netty4.ssl.client.Netty4ClientEngineFactory.apply(Netty4ClientEngineFactory.scala:63)
        at com.twitter.finagle.netty4.ssl.client.Netty4ClientSslHandler.$anonfun$initChannel$1(Netty4ClientSslHandler.scala:114)
        at com.twitter.finagle.netty4.ssl.client.Netty4ClientSslHandler.$anonfun$initChannel$1$adapted(Netty4ClientSslHandler.scala:111)
        at scala.Option.foreach(Option.scala:257)
        at com.twitter.finagle.netty4.ssl.client.Netty4ClientSslHandler.initChannel(Netty4ClientSslHandler.scala:111)
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:113)
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:105)
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:597)
        at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:178)
        at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:143)
        at com.twitter.finagle.netty4.channel.AbstractNetty4ClientChannelInitializer.initChannel(AbstractNetty4ClientChannelInitializer.scala:77)
        at com.twitter.finagle.netty4.channel.RawNetty4ClientChannelInitializer.initChannel(RawNetty4ClientChannelInitializer.scala:18)
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:113)
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:105)
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:597)
        at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:44)
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1387)
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1122)
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:647)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:506)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:419)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:478)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.cert.CertificateException: found no certificates in input stream
        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:98)
        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:64)
        at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1049)
        at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:177)

Are you specifying trustCerts? It looks like the file(s) you point to with that may not be valid CA certificates.

Hmm… I signed it myself, here is what I did exactly,

openssl and …

req -x509 -nodes -newkey rsa:2048 -keyout cakey.pem -out cacertificate.pem

req -new -nodes -keyout service2_key.pem -out service2_req.pem

ca -batch -keyfile cakey.pem -cert cacertificate.pem -out service2_certificate.pem -infiles service2_req.pem

I set stateOrProvince and OrganisationName as optional in openssl.cnf

and my k8s Secret deployment had the following,

kind: Secret
apiVersion: v1
metadata:
  name: certificates
namespace: prod
type: Opaque
data:
  certificate.pem: <service2_certificate.pem String>
  key.pem: <service2_key.pem String>
  cacertificate.pem: <cacertificate.pem String>

Would you mind sharing your linkerd config as well?

1 Like

Sure! Here it is.


We discussed this in slack. I am marking this as answered for now, but @zshaik please let me know if you still can’t get this working.

1 Like