Support Mutual TLS setup with self signed certificates

zshaik · September 27, 2017, 12:47pm

Summary

We have endpoints with self signed certificates & looking to setup mutual TLS.
After testing and referring docs I realized that linkerd doesn’t support clientAuth with disableValidation:true which means that I cannot setup tls with self signed certs.

So, are there any alternatives like enforcing finagle not to use jdk ssl when disableValidation is true etc… ?

Details

curl(with client cert & skip TLS validation) -> https://foo.bar.com SUCCESS

curl -> l5d(with client auth) -> https://foo.bar.com FAILS

Config:
version: 1.2.0

  client:
    kind: io.l5d.static
    configs:
    - prefix: "/$/io.buoyant.rinet/9083/{service}"
      tls:
        commonName: "{service}.bar.com"
        clientAuth:
          certPath: /io.buoyant/linkerd/certs/my-certificate.pem
          keyPath: /io.buoyant/linkerd/certs/my-key.pem

Request

https_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -vk https://foo.bar.com

Response

 TCP_NODELAY set
Connected to a4120bf07a2xxxxxx-2092924292.us-east-1.elb.amazonaws.com (36.226.29.237) port 4140 (#0)
allocate connect buffer!
Establish HTTP proxy tunnel to foo.bar.com:9083
CONNECT foo.bar.com:9083 HTTP/1.1
Host: foo.bar.com:9083
User-Agent: curl/7.55.0
Proxy-Connection: Keep-Alive

HTTP/1.1 502 Bad Gateway
l5d-err: General+OpenSslEngine+problem+at+remote+address%3A+foo.bar.com%2F165.237.45.25%3A9083.+Remote+Info%3A+Not+Available
Content-Type: text/plain
l5d-retryable: true
Content-Length: 117Preformatted text
Received HTTP code 502 from proxy after CONNECT
CONNECT phase completed!
Closing connection 0
curl: (56) Received HTTP code 502 from proxy after CONNECT

Log

    I 0927 11:15:13.640 UTC THREAD38 TraceId:306b260f5ec5fbeb: FailureAccrualFactory marking connection to "$/io.buoyant.rinet/9083/foo.bar.com" as dead. Remote Address: Inet(foo.bar.com/165.237.45.25:9083,Map())
    E 0927 11:15:13.643 UTC THREAD38 TraceId:306b260f5ec5fbeb: service failure: Failure(General OpenSslEngine problem at remote address: foo.bar.com/165.237.45.25:9083. Remote Info: Not Available, flags=0x09) with RemoteInfo -> Upstream Address: /172.28.166.212:53188, Upstream Client Id: Not Available, Downstream Address: foo.bar.com/165.237.45.25:9083, Downstream Client Id: $/io.buoyant.rinet/9083/foo.bar.com, Trace Id: 306b260f5ec5fbeb.306b260f5ec5fbeb<:306b260f5ec5fbeb
    WARN 0927 11:15:13.646 UTC finagle/netty4-7: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24)
    at java.lang.Thread.run(Thread.java:748)
    Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:649)
        at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:450)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:878)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:981)
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:200)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    ... 18 more
    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:223)
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:645)
    ... 26 more
    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
    ... 33 more
    W 0927 11:15:13.645 UTC THREAD38: ChannelStatsHandler caught an exception
        io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24)
    at java.lang.Thread.run(Thread.java:748)
    Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:649)
        at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:450)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:878)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:981)
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:200)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    ... 18 more
    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:223)
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:645)
    ... 26 more
    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
    ... 33 more
    W 0927 11:15:13.645 UTC THREAD38: Unhandled exception in connection with foo.bar.com/165.237.45.25:9083, shutting down connection
        io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24)
    at java.lang.Thread.run(Thread.java:748)
    Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:649)
        at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:450)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:878)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:981)
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:200)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    ... 18 more
    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:223)
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:645)
    ... 26 more
    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
    ... 33 more

Test to make sure that key & cert is valid and deployed appropriately

shell into l5d pod at io.buoyant/linkerd/certs

curl -vk https://foo.bar.com -E ./my-certificate.pem ./my-key.pem

able to get the response back successfully

Alex · September 27, 2017, 5:53pm

Hi @zshaik,

You should be able to use mutual TLS with self-signed certs but you will need to turn the disableValidation property off (ie validation must be on) and common names in your Linkerd configs must match the common names of your self-signed certificates.

zshaik · September 27, 2017, 6:54pm

Hey @Alex , sounds great! I thought I had to disable the validation. Hmm… but with,
enableValidation & clientAuth I get 502 Badgateway (response & log is in the question)

are my keys valid?
Yes I tested with plain curl
is my common name in cert matching with l5d config?
yes

zshaik · October 10, 2017, 7:20pm

Resolved!
added trustcerts and appropriate secret deployment.

   trustCert:
    - /io.buoyant/linkerd/certs/cacert.pem
    clientAuth:
      certPath: /io.buoyant/linkerd/certs/certificate.pem
      keyPath: /io.buoyant/linkerd/certs/key.pem

william · October 11, 2017, 4:14pm

Glad you got it working!

sbehnke · January 3, 2018, 3:04pm

We are also trying to get mutual self-signed TLS working. The services share the self-signed cert for both client auth and server auth. Using these certs we are able to get the services to client auth to linkerd and the requests correctly routed. However, when making the outbound client connection, we are getting the same errors as above.

servers:
tls:
certPath: /certificates/cert.pem
keyPath: /certificates/key.pem
requireClientAuth: true
caCertPath: /certificates/cert.pem
client:
kind: io.l5d.static
configs:
- prefix: /#/io.l5d.fs/{service}
tls:
commonName: "{service}.foo.com"
trustCerts:
- /certificates/cert.pem
clientAuth:
certPath: /certificates/cert.pem
keyPath: /certificates/key.pem