Linkerd in EKS having trouble with auth (keyclock) container

Hi!

I installed Linkerd 2.9.2 using CLI in AWS EKS and injected to your application. Its working fine for all the services with mtls but i am having trouble with auth container used for keycloak authentication.

Process i followed, injected linkerd to auth pod contains postgresdb check init container and auth container with liveness and readiness probe. Once linkerd injected it created linkerd-init and linkerd-proxy containers. When I describe auth pod linkerd-proxy container is running and true state, auth container is running but in false state and after sometime its going terminating state and events i got
Readiness probe failed: HTTP probe failed with statuscode: 502
Readiness probe failed: HTTP probe failed with statuscode: 404
Liveness probe failed: HTTP probe failed with statuscode: 404

Auth pod status 2/3
When i check auth container logs no errors
Linkerd-proxy container logs
time=“2021-02-15T16:22:17Z” level=info msg=“running version stable-2.9.2”
[ 0.000951s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.001770s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.001826s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.001833s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.001837s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[ 0.001840s] INFO ThreadId(01) linkerd2_proxy: Local identity is default.new.serviceaccount.identity.linkerd.cluster.local
[ 0.001849s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.001855s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.002162s] INFO ThreadId(01) outbound: linkerd2_app: listen.addr=127.0.0.1:4140 ingress_mode=false
[ 0.002235s] INFO ThreadId(01) inbound: linkerd2_app: listen.addr=0.0.0.0:4143
[ 0.329291s] WARN ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=ip:40836 target.addr=podip:4191}: rustls::session: Sending fatal alert AccessDenied
[ 0.329481s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=ip:40836 target.addr=podip:4191}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
[ 0.543760s] INFO ThreadId(02) daemon:identity: linkerd2_app: Certified identity: default.new.serviceaccount.identity.linkerd.cluster.local
[ 332.166074s] WARN ThreadId(01) inbound:accept{peer.addr=ip:60710 target.addr=podip:9990}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 662.166313s] WARN ThreadId(01) inbound:accept{peer.addr=ip:36420 target.addr=podip:9990}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 992.166169s] WARN ThreadId(01) inbound:accept{peer.addr=ip:40328 target.addr=podip:9990}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 1322.166103s] WARN ThreadId(01) inbound:accept{peer.addr=ip:44232 target.addr=podip:9990}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 1652.166186s] WARN ThreadId(01) inbound:accept{peer.addr=ip:48138 target.addr=podip:9990}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 1982.166321s] WARN ThreadId(01) inbound:accept{peer.addr=ip:52064 target.addr=podip:9990}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)

ip : nodeip, podip: auth podip, auth port: 9990

When exec into auth container and run curl getting below output
curl -I http://localhost:9990/auth/
HTTP/1.1 405 Method Not Allowed
Connection : keep-alive
Content-Length : 83
Content-Type : text/html

Tried skipping ports didn’t work

podAnnotations:
      config.linkerd.io/skip-inbound-ports: "7600" # skip keycloak cluster discovery
      config.linkerd.io/skip-outbound-ports: "7600"

So in this scenario which one is not working auth container or linkerd-proxy. Can anyone help with this issue. Do you guys have any input on how I should proceed here? Thought its simple setup works well but having issues.

@pachalk I saw that you posted a similar issue on GitHub, so I’ll reply over there.