502 bad gateway through linkerd

#1

Issue type: bug
Linkerd version: 1.6.0
Environment: k8s on AWS
Endpoint type: One way SSL configuration.

Request

http_proxy=$<l5d>:4140 curl -v http://foo.bar.com:443

Response

*Trying IP-Addr …
*TCP_NODELAY set
Connected to (nil) () port 4140 (#0)
GET http://foo.bar.com:443
HTTP/1.1
Host: :443
User-Agent: curl/
Accept: /
Proxy-Connection: Keep-Alive
HTTP/1.1 502 Bad Gateway
l5d-err: Unable+to+establish+connection+to+< list-of-svc >
Content-Length: 560
Content-Type: text/plain
Unable to establish connection to /443
service name: /svc/:443
client name: //io.buoyant.rinet/443/<svc-name> addresses: [<svc-name>/<IP-Addr>:443] selected address: <domain-name>/<IP-Addr>:443 dtab resolution: /svc/<svc-name>:443 /ph/443/<svc-name> (/svc=&gt;//io.buoyant.porthostPfx/ph) //io.buoyant.rinet/443/<svc-name> (/ph=&gt;//io.buoyant.rinet)
*Curl_http_done: called premature == 0
*Connection #0 to host (nil) left intact

Expected Response

Endpoint when requested individually, responds with 200 status

200 ok response

Logs form linkerd

k8s ns default service com:443 endpoints resource does not exist, assuming it has yet to be created
I FailureAccrualFactory marking connection to “/io.buoyant.rinet/443/foo.bar.com" as dead. Remote Address: Inet(<domain-name>/<IP-Addr>:443,Map()) I FailureAccrualFactory marking connection to "/io.buoyant.rinet/443/foo.bar.com” as dead. Remote Address: Inet(/:443,Map())

Configuration

        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            trustCertsBundle: /etc/ssl/certs/ca-certificates.crt
            commonName: "{service}"

        - prefix: "/$/io.buoyant.rinet/443/foo.bar.com"
          tls:
            trustCertsBundle: /io.buoyant/linkerd/certs/foo.bar.com.pem
            commonName: "foo.bar.com"

please let me know if I can provide with additional info

#2

@zshaik this message is a little suspect.

“ endpoints resource does not exist, assuming it has yet to be created“

Can you confirm if you can resolve that service name in the dtab playground?

#3

Hey @zshaik, it looks like there’s a bunch of data in the response log you provided that got mangled, for example a bunch of newlines have been stripped, the > character has been escaped, I’m guessing that the $ character has been removed, and there are inconsistencies where the Host header is listed as :433, the service name is listed as /svc/:443, but the resolution starts with the service name of /svc/<svc-name>:443. Can you please provide the output as it actually appears? Redactions of service names or IP addresses are fine, as long as they are consistent and clearly marked.

Your log message indicates a failure to look up a name in Kubernetes, but the Kubernetes namer does not appear in your resolution at all.

Can you verify that <IP-Addr>:443 is reachable and that you are able to establish connections manually.

#4

I’ll also second @dennis.ab’s suggestion to verify the resolution in the dtab UI.

closed #5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.