Mutual auth configured for external endpoint (IP address) fails to get response

Setup

l5d: v1.3.5
env: aws
k8s: v1.7.x

issue in a nutshell

curl -v https://52.xx.xx.xx:8443 -E ./client-cert.pem --key ./client-key.pem --cacert tomcat-cacertificate.pem => success

http_proxy=$l5d:4140 curl -v http://52.xx.xx.xx:8443  => 502 bad gateway

Details

We have an endpoint on aws instance configured with mutual auth, (self-signed cert cacert & client cert), added cert configuration as below in linkerd.yml.
The certs are configured through secret deployment where,
client-cert & cacert are in p12 base64 encoded in secret
key is in pk8 format base64 encoded in secret.

    - prefix: "/$/io.buoyant.rinet/8443/{service}"
      tls:
        trustCerts:
        - /io.buoyant/linkerd/test-certs/tomcat-cacertificate.pem
        - /etc/ssl/certs/ca-certificates.crt
        commonName: "{service}"
        clientAuth:
          certPath: /io.buoyant/linkerd/test-certs/client-cert.pem
          keyPath: /io.buoyant/linkerd/test-certs/client-key.pem

testing to see if certs created in pod are valid

kubectl exec -it l5d-3n0xk – /bin/bash

curl -v https://52.xx.xx.xx:8443 -E ./client-cert.pem --key ./client-key.pem --cacert tomcat-cacertificate.pem

* Rebuilt URL to: https://52.xx.xx.xx:8443/
*   Trying 52.xx.xx.xx...
* TCP_NODELAY set
* Connected to 52.xx.xx.xx (52.xx.xx.xx) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: tomcat.pem
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=IN; ST=Telengana; L=Hyderabad; O=zillani; OU=Developer; CN=52.xx.xx.xx
*  start date: Mar 14 09:31:34 2018 GMT
*  expire date: Mar 11 09:31:34 2028 GMT
*  common name: 52.xx.xx.xx (matched)
*  issuer: C=IN; ST=Telengana; L=Hyderabad; O=zillani; OU=Developer; CN=52.xx.xx.xx
*  SSL certificate verify ok.* GET / HTTP/1.1
* Host: 52.xx.xx.xx:8443
* User-Agent: curl/7.56.1
* Accept: */*
* 
* HTTP/1.1 200 
* Content-Type: application/json
* Content-Length: 28
* Date: Wed, 14 Mar 2018 10:45:23 GMT  {"message": "Hello, World!"}

Testing through linkerd and linkerd logs

http_proxy=$(kubectl get svc l5d -o jsonpath="{.status.loadBalancer.ingress[0].*}"):4140 curl -v http://52.xx.xx.xx:8443
* Rebuilt URL to: http://52.xx.xx.xx:8443/
*   Trying 35.xx.xx.xx...
* TCP_NODELAY set
* Connected to xxxx-1997826825.us-west-2.elb.amazonaws.com port 4140 (#0)
> GET http://52.xx.xx.xx:8443/ HTTP/1.1
> Host: 52.xx.xx.xx:8443
> User-Agent: curl/7.56.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: null+at+remote+address%3A+%2F52.xx.xx.xx%3A8443.+Remote+Info%3A+Not+Available+at+remote+address%3A+%2F52.xx.xx.xx%3A8443.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< l5d-retryable: true
< Content-Length: 138
< 
* Connection #0 to host a22569e3e230911e886c6061ad424cbf-1997826825.us-west-2.elb.amazonaws.com left intact
null at remote address: /52.xx.xx.xx:8443. Remote Info: Not Available at remote address: /52.xx.xx.xx:8443. Remote Info: Not Available

linkerd log

E 0314 12:46:50.786 UTC THREAD54 TraceId:b0366745d8c7a013: service failure: Failure(null at remote address: /52.xxx.xx.xx:8443. Remote Info: Not Available at remote address: /52.xxx.xx.xx:8443. Remote Info: Not Available, flags=0x09) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: /52.xxx.xx.xx:8443, Downstream label: $/io.buoyant.rinet/8443/52.xxx.xx.xx, Trace Id: b0366745d8c7a013.b0366745d8c7a013<:b0366745d8c7a013

We have experience of configuring for external endpoints(domain based) with mutual tls setup with SUCCESS, this endpoint is NOT a domain but an ipaddress, is the issue indicating that linkerd doesn’t allow such endpoints?

Hi @zshaik, in the linkerd config you posted above, is {service} the IP address of the service?

yes, actually tried both ways using {service} & ip as below.

    - prefix: "/$/io.buoyant.rinet/8443/52.xx.xx.xx"
      tls:
        trustCerts:
        - /io.buoyant/linkerd/test-certs/tomcat-cacertificate.pem
        - /etc/ssl/certs/ca-certificates.crt
        commonName: "52.xx.xx.xx"

Hi @zshaik,
How are you generating your certs? If we can generate them the same way as you, we can try and reproduce the issue.

Thanks

Sure,
generating server key in pkcs12 format & encoding in base64.

Step-1 server key

keytool -genkey -v -alias tomcat -keyalg RSA -validity 3650 -keystore tomcat.keystore -dname "CN=xx.xx.xx.xx, OU=Developer, O=xx.xx.xx.xx, L=Hyderabad, ST=Telengana, C=IN" -storepass changeit -keypass changeit

keytool -importkeystore -srckeystore tomcat.keystore -destkeystore tomcat.p12 -srcstoretype jks -deststoretype pkcs12

openssl pkcs12 -in tomcat.p12 -out tomcat.pem

openssl base64 -in tomcat.pem -out tomcat-base64.pem

Step-2 client-key

generating key in pkcs8 format and encoding in base64

keytool -genkey -v -alias clientkey -keyalg RSA -storetype PKCS12 -keystore client.p12 -dname "CN=xx.xx.xx.xx, OU=Developer, O=xx.xx.xx.xx, L=Hyderabad, ST=Telengana, C=IN" -storepass changeit -keypass changeit

keytool -export -alias clientkey -keystore client.p12 -storetype PKCS12 -storepass changeit -rfc -file client.cer

keytool -import -v -file client.cer -keystore tomcat.keystore -storepass changeit -alias clientkey

openssl pkcs12 -in client.p12 -out client-cert.pem -clcerts -nokeys

openssl pkcs12 -in client.p12 -out client-key.pem -nocerts -nodes

openssl pkcs8 -in client-key.pem -topk8 -nocrypt -out client-key.pk8

openssl base64 -in client.pem -out client-base64.pem

Hi! Thank you for providing the extra context!

I ran through your set of commands for generating the keys and noticed that not all file names match up, so it’s a bit unclear which keys you’re actually using in the config.

When trying to reproduce it I accidentally added a type to the cert file and got a service failure error, however Linkerd provided a long stack trace. Did you get a stack trace from Linkerd and if so, what does it say?

Further it’s hard to get a complete picture without your server-side tls config. So if your stack trace doesn’t help you debug your issue, it would be helpful for us to see the server-side config as well.
Here are the docs including required fields: https://linkerd.io/config/1.3.6/linkerd/index.html#server-tls

Let us know how it goes!

1 Like