Linkerd 502 badgateway for external endpoint


#1

I am unable to get the expected response from our endpoint through the linkerd, the endpoint is external to the cluster and configured as below,

config:

    - prefix: "/$/io.buoyant.rinet/443/{service}"
      tls:
        trustCertsBundle: /etc/ssl/certs/ca-certificates.crt
        commonName: "{service}"

    - prefix: "/$/io.buoyant.rinet/443/sec.qa.go2bb.com"
      tls:
        disableValidation: true

individual curl:
the endpoint responds with 403 badgateway, this is tested from linkerd pod.

> curl -v https://sec.qa.go2bb.com:443/services/msolookupservice.asmx
> *   Trying 52.203.103.200...
> * TCP_NODELAY set
> * Connected to sec.qa.go2bb.com (52.203.103.200) port 443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /etc/ssl/certs/ca-certificates.crt
> CApath: /etc/ssl/certs
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
> * ALPN, server did not agree to a protocol
> * Server certificate:
> *  subject: OU=Domain Control Validated; CN=*.qa.go2bb.com
> *  start date: Mar 23 15:02:01 2017 GMT
> *  expire date: Apr  8 22:57:20 2019 GMT
> *  subjectAltName: host "sec.qa.go2bb.com" matched cert's "*.qa.go2bb.com"
> *  issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy sec Certificate Authority - G2
> *  SSL certificate verify ok.
>  GET /services/msolookupservice.asmx HTTP/1.1
>  Host: sec.qa.go2bb.com
>  User-Agent: curl/7.56.1
>  Accept: */*
> 
> * TLSv1.2 (IN), TLS handshake, Hello request (0):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Request CERT (13):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Certificate (11):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
>  HTTP/1.1 403 Forbidden
>  Content-Type: text/html
>  Server: Microsoft-IIS/8.5
>  Host: E1d-Web-QA1
>  X-Frame-Options: SAMEORIGIN
>  X-Content-Type-Options: nosniff
>  X-Xss-Protection: 1; mode=block
>  Referrer-Policy: strict-origin
>  Strict-Transport-Security: max-age=31536000; includeSubdomains
>  Content-Security-Policy: default-src 'self' 'unsafe-inline' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval' https:;
>  Date: Thu, 09 Aug 2018 15:14:19 GMT
>  Content-Length: 1233
> 
> !DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"
> html xmlns="http://www.w3.org/1999/xhtml"
> head
> meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"
> 403 - Forbidden: Access is denied
> 
> style type="text/css"
> !--
> body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
> fieldset{padding:0 15px 10px 15px;}
> h1{font-size:2.4em;margin:0;color:#FFF;}
> h2{font-size:1.7em;margin:0;color:#CC0000;}
> h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
> #header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
> background-color:#555555;}
> #content{margin:0 0 0 2%;position:relative;}
> .content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
> --
> /style
> /head
> body
> div id="header"h1Server Error/h1/div
> div id="content"
> div class="content-container"fieldset
> h2403 - Forbidden: Access is denied./h2
> h3You do not have permission to view this directory or page using the credentials that you supplied./h3
> /fieldset/div
> /div
> /body
> /html
> * Connection #0 to host sec.qa.go2bb.com left intact

curl through linkerd:

returns 502 badgateway

http_proxy=http://a91dd1aca9d88a01fb-2036530607.us-west-2.elb.amazonaws.com:4140 curl -v http://sec.qa.go2bb.com:443/services/msolookupservice.asmx
*   Trying 54.191.14.1...
* TCP_NODELAY set
* Connected to a91dd1aca9b3b021d88a01fb-2036530607.us-west-2.elb.amazonaws.com (54.191.149.1) port 4140 (#0)
> GET http://sec.qa.go2bb.com:443/services/msolookupservice.asmx HTTP/1.1
> Host: sec.qa.go2bb.com:443
> User-Agent: curl/7.56.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< l5d-err: ChannelException+at+remote+address%3A+sec.qa.go2bb.com%2F52.203.103.200%3A443+from+service%3A+%24%2Fio.buoyant.rinet%2F443%2Fsec.qa.go2bb.com.+Remote+Info%3A+Upstream+Address%3A+Not+Available%2C+Upstream+id%3A+Not+Available%2C+Downstream+Address%3A+sec.qa.go2bb.com%2F52.203.103.200%3A443%2C+Downstream+label%3A+%24%2Fio.buoyant.rinet%2F443%2Fsec.qa.go2bb.com%2C+Trace+Id%3A+832b7402cb7dbba6.f8f31fbc2c759b04%3C%3A832b7402cb7dbba6
< Content-Length: 418
< Content-Type: text/plain
< 
* Connection #0 to host a91dd1aca9b3b11e8ada5021d88a01fb-2036530607.us-west-2.elb.amazonaws.com left intact
ChannelException at remote address: sec.qa.go2bb.com/52.203.103.200:443 from service: $/io.buoyant.rinet/443/sec.qa.go2bb.com. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: sec.qa.go2bb.com/52.203.103.200:443, Downstream label: $/io.buoyant.rinet/443/sec.qa.go2bb.com, Trace Id: 832b7402cb7dbba6.f8f31fbc2c759b04<:832b7402cb7dbba6

#2

Just highlighting, I found that the endpoint is HSTS (Http Strict Transport Security) based.
Do these headers prevent the endpoint from getting proxied?

Host: E1d-Web-QA1
>  X-Frame-Options: SAMEORIGIN
>  X-Content-Type-Options: nosniff
>  X-Xss-Protection: 1; mode=block
>  Referrer-Policy: strict-origin
>  Strict-Transport-Security: max-age=31536000; includeSubdomains
>  Content-Security-Policy: default-src 'self' 'unsafe-inline' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval' https:;
>  Date: Thu, 09 Aug 2018 15:14:19 GMT
>  Content-Length: 1233

#3

@zshaik, I am not aware of Linkerd modifying its proxying behavior based on the contents of the headers. It could be that the server is requiring something that the client is not fulfilling TLS wise. I am not an expert at TLS, out of curiosity, I noticed that disableValidation is set to true. What is the reason for having that set to true?


#4

In addition to my previous question, is there a stacktrace in Linkerd that highlights why Linkerd gives a 502 bad gateway?


#5

Hey @dennis.ab ,
I wanted to isolate the issue to check if its related to cacerts so disabled tls validation, but the response or linkerd logs/metrics are the same even if the CN validation is enabled/disabled.
(I updated the question with full l5d proxy curl and response.)

linkerd logs
E 0810 12:10:53.958 UTC THREAD32 TraceId:9d65c52beb1438f0: service failure: com.twitter.finagle.ChannelClosedException: ChannelException at remote address: sec.qa.go2bb.com/52.203.103.200:443 from service: $/io.buoyant.rinet/443/sec.qa.go2bb.com. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: sec.qa.go2broadband.com/52.203.103.200:443, Downstream label: $/io.buoyant.rinet/443/sec.qa.go2broadband.com, Trace Id: 9d65c52beb1438f0.3cbb36c592732f3465c52beb1438f0

linkerd metrics
metrics.json (7.0 KB)


#6

also, below is how we tested with CN validation enabled,

- prefix: "/$/io.buoyant.rinet/443/sec.qa.go2bb.com"
  tls:
    trustCertsBundle: /io.buoyant/linkerd/l5d-general/sec.qa.go2bb.com.pem
    commonName: "sec.qa.go2bb.com"

note: The common name of the certificate has a wildcard CN=*.qa.go2bb.com, this can be observed in the response posted in the question.


#7

This indicates that the server sec.qa.go2bb.com is closing the connection somehow, can you verify via ssldump or some other method that shows that the Linkerd client is performing a TLS handshake with the server?


#8

Hi @dennis.ab
I got the ssldump, looks like its performing the handshake, below is the log.

root@dev-infra--kubernetes-10-2-130-170:/io.buoyant/linkerd/l5d-general# ssldump -Ad -i ens5 host sec.qa.go2bb.com

New TCP connection #1: ip-10-2-131-171.us-west-2.compute.internal(35538) <-> ec2-52-203-103-200.compute-1.amazonaws.com(443)
1 1  0.0900 (0.0900)  C>SV3.1(173)  Handshake
      ClientHello
        Version 3.3 
        random[32]=
          a0 4b 9d 4f 05 78 05 bc 5a 64 1e 4e 42 40 56 a9 
          70 59 2c c1 f3 e5 77 ea ef b9 b9 af 11 60 53 24 
        cipher suites
        Unknown value 0xc02b
        Unknown value 0xc02f
        Unknown value 0xc02c
        Unknown value 0xc030
        Unknown value 0xcca9
        Unknown value 0xcca8
        Unknown value 0xc009
        Unknown value 0xc023
        Unknown value 0xc013
        Unknown value 0xc027
        Unknown value 0xc00a
        Unknown value 0xc024
        Unknown value 0xc014
        Unknown value 0xc028
        Unknown value 0x9c
        Unknown value 0x9d
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0x3c
        TLS_RSA_WITH_AES_256_CBC_SHA
        Unknown value 0x3d
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        compression methods
                  NULL
1 2  0.1829 (0.0929)  S>CV3.3(3023)  Handshake
      ServerHello
        Version 3.3 
        random[32]=
          5b 71 94 cc bb ce 97 b1 ce 4e 04 55 ae bd 5b 13 
          da 83 18 38 94 0b a5 aa e7 4a 25 8d 62 32 a3 9c 
        session_id[32]=
          4d 13 00 00 3b 43 33 f4 80 45 c8 e1 26 3c cc 20 
          94 67 bb 22 83 99 ea 1b 89 d2 9b 84 5e 69 6f 3a 
        cipherSuite         Unknown value 0xc028
        compressionMethod                   NULL
      Certificate
      ServerKeyExchange
      ServerHelloDone
1 3  0.1882 (0.0052)  C>SV3.3(70)  Handshake
      ClientKeyExchange
1 4  0.1882 (0.0000)  C>SV3.3(1)  ChangeCipherSpec
1 5  0.1882 (0.0000)  C>SV3.3(96)  Handshake
1 6  0.2795 (0.0913)  S>CV3.3(1)  ChangeCipherSpec
1 7  0.2795 (0.0000)  S>CV3.3(96)  Handshake
1 8  0.2819 (0.0023)  C>SV3.3(448)  application_data
1 9  0.3721 (0.0902)  S>CV3.3(80)  Handshake
1 10 0.3725 (0.0003)  C>SV3.3(80)  Alert
1    0.4629 (0.0904)  S>C  TCP FIN
1    0.4632 (0.0002)  C>S  TCP FIN

#9

Yea, looks like the handshake happens successfully. Is there anyway you can debug the server side to see if you can get any useful information?


#10

We found that, the client IP wasn’t whitelisted in the elb at the server end, that is why the request is getting cancelled,

Thank you for the support :blush: @dennis.ab