Mutual auth configured for external endpoint (IP address) fails to get response

Help Kubernetes
#1

Setup

l5d: v1.3.5
env: aws
k8s: v1.7.x

issue in a nutshell

curl -v https://52.xx.xx.xx:8443 -E ./client-cert.pem --key ./client-key.pem --cacert tomcat-cacertificate.pem => success

http_proxy=$l5d:4140 curl -v http://52.xx.xx.xx:8443  => 502 bad gateway

Details

We have an endpoint on aws instance configured with mutual auth, (self-signed cert cacert & client cert), added cert configuration as below in linkerd.yml.
The certs are configured through secret deployment where,
client-cert & cacert are in p12 base64 encoded in secret
key is in pk8 format base64 encoded in secret.

    - prefix: "/$/io.buoyant.rinet/8443/{service}"
      tls:
        trustCerts:
        - /io.buoyant/linkerd/test-certs/tomcat-cacertificate.pem
        - /etc/ssl/certs/ca-certificates.crt
        commonName: "{service}"
        clientAuth:
          certPath: /io.buoyant/linkerd/test-certs/client-cert.pem
          keyPath: /io.buoyant/linkerd/test-certs/client-key.pem

testing to see if certs created in pod are valid

kubectl exec -it l5d-3n0xk – /bin/bash

curl -v https://52.xx.xx.xx:8443 -E ./client-cert.pem --key ./client-key.pem --cacert tomcat-cacertificate.pem

* Rebuilt URL to: https://52.xx.xx.xx:8443/
*   Trying 52.xx.xx.xx...
* TCP_NODELAY set
* Connected to 52.xx.xx.xx (52.xx.xx.xx) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: tomcat.pem
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=IN; ST=Telengana; L=Hyderabad; O=zillani; OU=Developer; CN=52.xx.xx.xx
*  start date: Mar 14 09:31:34 2018 GMT
*  expire date: Mar 11 09:31:34 2028 GMT
*  common name: 52.xx.xx.xx (matched)
*  issuer: C=IN; ST=Telengana; L=Hyderabad; O=zillani; OU=Developer; CN=52.xx.xx.xx
*  SSL certificate verify ok.* GET / HTTP/1.1
* Host: 52.xx.xx.xx:8443
* User-Agent: curl/7.56.1
* Accept: */*
* 
* HTTP/1.1 200 
* Content-Type: application/json
* Content-Length: 28
* Date: Wed, 14 Mar 2018 10:45:23 GMT  {"message": "Hello, World!"}

Testing through linkerd and linkerd logs

http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -v http://52.xx.xx.xx:8443
* Rebuilt URL to: http://52.xx.xx.xx:8443/
*   Trying 35.xx.xx.xx...
* TCP_NODELAY set
* Connected to xxxx-1997826825.us-west-2.elb.amazonaws.com port 4140 (#0)
> GET http://52.xx.xx.xx:8443/ HTTP/1.1
> Host: 52.xx.xx.xx:8443
> User-Agent: curl/7.56.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: null+at+remote+address%3A+%2F52.xx.xx.xx%3A8443.+Remote+Info%3A+Not+Available+at+remote+address%3A+%2F52.xx.xx.xx%3A8443.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< l5d-retryable: true
< Content-Length: 138
< 
* Connection #0 to host a22569e3e230911e886c6061ad424cbf-1997826825.us-west-2.elb.amazonaws.com left intact
null at remote address: /52.xx.xx.xx:8443. Remote Info: Not Available at remote address: /52.xx.xx.xx:8443. Remote Info: Not Available

linkerd log

E 0314 12:46:50.786 UTC THREAD54 TraceId:b0366745d8c7a013: service failure: Failure(null at remote address: /52.xxx.xx.xx:8443. Remote Info: Not Available at remote address: /52.xxx.xx.xx:8443. Remote Info: Not Available, flags=0x09) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: /52.xxx.xx.xx:8443, Downstream label: $/io.buoyant.rinet/8443/52.xxx.xx.xx, Trace Id: b0366745d8c7a013.b0366745d8c7a013<:b0366745d8c7a013

We have experience of configuring for external endpoints(domain based) with mutual tls setup with SUCCESS, this endpoint is NOT a domain but an ipaddress, is the issue indicating that linkerd doesn’t allow such endpoints?

#2

Hi @zshaik, in the linkerd config you posted above, is {service} the IP address of the service?

#3

yes, actually tried both ways using {service} & ip as below.

    - prefix: "/$/io.buoyant.rinet/8443/52.xx.xx.xx"
      tls:
        trustCerts:
        - /io.buoyant/linkerd/test-certs/tomcat-cacertificate.pem
        - /etc/ssl/certs/ca-certificates.crt
        commonName: "52.xx.xx.xx"
#4

Hi @zshaik,
How are you generating your certs? If we can generate them the same way as you, we can try and reproduce the issue.

Thanks

#5

Sure,
generating server key in pkcs12 format & encoding in base64.

Step-1 server key

keytool -genkey -v -alias tomcat -keyalg RSA -validity 3650 -keystore tomcat.keystore -dname "CN=xx.xx.xx.xx, OU=Developer, O=xx.xx.xx.xx, L=Hyderabad, ST=Telengana, C=IN" -storepass changeit -keypass changeit

keytool -importkeystore -srckeystore tomcat.keystore -destkeystore tomcat.p12 -srcstoretype jks -deststoretype pkcs12

openssl pkcs12 -in tomcat.p12 -out tomcat.pem

openssl base64 -in tomcat.pem -out tomcat-base64.pem

Step-2 client-key

generating key in pkcs8 format and encoding in base64

keytool -genkey -v -alias clientkey -keyalg RSA -storetype PKCS12 -keystore client.p12 -dname "CN=xx.xx.xx.xx, OU=Developer, O=xx.xx.xx.xx, L=Hyderabad, ST=Telengana, C=IN" -storepass changeit -keypass changeit

keytool -export -alias clientkey -keystore client.p12 -storetype PKCS12 -storepass changeit -rfc -file client.cer

keytool -import -v -file client.cer -keystore tomcat.keystore -storepass changeit -alias clientkey

openssl pkcs12 -in client.p12 -out client-cert.pem -clcerts -nokeys

openssl pkcs12 -in client.p12 -out client-key.pem -nocerts -nodes

openssl pkcs8 -in client-key.pem -topk8 -nocrypt -out client-key.pk8

openssl base64 -in client.pem -out client-base64.pem
#6

Hi! Thank you for providing the extra context!

I ran through your set of commands for generating the keys and noticed that not all file names match up, so it’s a bit unclear which keys you’re actually using in the config.

When trying to reproduce it I accidentally added a type to the cert file and got a service failure error, however Linkerd provided a long stack trace. Did you get a stack trace from Linkerd and if so, what does it say?

Further it’s hard to get a complete picture without your server-side tls config. So if your stack trace doesn’t help you debug your issue, it would be helpful for us to see the server-side config as well.
Here are the docs including required fields: https://linkerd.io/config/1.3.6/linkerd/index.html#server-tls

Let us know how it goes!

1 Like