Container linkerd-proxy is not ready

Hi all!
I try to run linkerd 2.8.1 from helm chart with identity.issuer.scheme=kubernetes.io/tls as in this instruction , but data plane is not ready, all control plane containers is up and ready, but data plane (linkerd-proxy) is not
linkerd-controller-5b79949c8-977kl 1/2 Running 0 11m
linkerd-controller-5b79949c8-pth8z 1/2 Running 0 11m
linkerd-controller-5b79949c8-xb9lv 1/2 Running 0 11m
linkerd-destination-6d566d86bd-2x8vw 1/2 Running 0 11m
linkerd-destination-6d566d86bd-fhstm 1/2 Running 1 11m
linkerd-destination-6d566d86bd-pm6fl 1/2 Running 0 11m
linkerd-grafana-59649f5b45-sdpqh 1/2 Running 0 11m
linkerd-identity-cd6d749f6-2tgbs 1/2 Running 0 11m
linkerd-identity-cd6d749f6-fd5lz 1/2 Running 0 11m
linkerd-identity-cd6d749f6-t4nds 1/2 Running 0 11m
linkerd-prometheus-7dd7455655-hktjk 1/2 Running 0 11m
linkerd-proxy-injector-9749b5498-6rqxn 1/2 Running 0 11m
linkerd-proxy-injector-9749b5498-85wwb 1/2 Running 0 11m
linkerd-proxy-injector-9749b5498-qcc9l 1/2 Running 0 11m
linkerd-sp-validator-776d8d649f-52qh7 1/2 Running 0 11m
linkerd-sp-validator-776d8d649f-v26h2 1/2 Running 0 11m
linkerd-sp-validator-776d8d649f-vh4sv 1/2 Running 0 11m
linkerd-tap-56cf7fbbd7-298vl 1/2 Running 0 11m
linkerd-tap-56cf7fbbd7-8xjq8 1/2 Running 0 11m
linkerd-tap-56cf7fbbd7-wkbzm 1/2 Running 0 11m
linkerd-web-df89b496f-kxp8r 1/2 Running 0 11m

# linkerd check --proxy -L pfm-admins
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2020-08-09T00:30:32Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

linkerd-api
-----------
\ pod/linkerd-controller-5b79949c8-pth8z container linkerd-proxy is not ready

I try to set log level to debug/trace in linkerd-proxy and don’t see queries to http://:4191/ready only to http://:9995/ready (200 ready)

How to understand way http://:4191/ready returned 503 not ready ?

Hi @sbvitok, can you share the logs from the linkerd-controller or linkerd-identity components?

My best guess is that the proxies cannot communicate with the linkerd-identity component, so the certificates used for mTLS are not being issued.

Where is this cluster running? In a cloud provider? Can you also share the output from kubectl get ev -L pfm-admins?

# kubectl logs -n pfm-admins linkerd-controller-7766b66bfb-84tf8 -c public-api
time="2020-08-10T17:45:43Z" level=info msg="running version stable-2.8.1"
time="2020-08-10T17:45:43Z" level=info msg="Using cluster domain: cluster.local"
time="2020-08-10T17:45:43Z" level=info msg="waiting for caches to sync"
time="2020-08-10T17:45:43Z" level=info msg="caches synced"
time="2020-08-10T17:45:43Z" level=info msg="starting admin server on :9995"
time="2020-08-10T17:45:43Z" level=info msg="starting HTTP server on :8085"

# kubectl logs -n pfm-admins linkerd-identity-5fb4f45888-55ws6 -c identity
time="2020-08-10T17:45:42Z" level=info msg="running version stable-2.8.1"
time="2020-08-10T17:45:42Z" level=info msg="starting admin server on :9990"
time="2020-08-10T17:45:42Z" level=info msg="starting gRPC server on :8080"
time="2020-08-10T17:45:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:46:04 +0000 UTC"
time="2020-08-10T17:45:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:46:14 +0000 UTC"
time="2020-08-10T17:46:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:46:24 +0000 UTC"
time="2020-08-10T17:46:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:46:34 +0000 UTC"
time="2020-08-10T17:46:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:46:44 +0000 UTC"
time="2020-08-10T17:46:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:46:54 +0000 UTC"
time="2020-08-10T17:46:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:47:04 +0000 UTC"
time="2020-08-10T17:46:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:47:14 +0000 UTC"
time="2020-08-10T17:47:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:47:24 +0000 UTC"
time="2020-08-10T17:47:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:47:34 +0000 UTC"
time="2020-08-10T17:47:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:47:44 +0000 UTC"
time="2020-08-10T17:47:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:47:54 +0000 UTC"
time="2020-08-10T17:47:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:48:04 +0000 UTC"
time="2020-08-10T17:47:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:48:14 +0000 UTC"
time="2020-08-10T17:48:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:48:24 +0000 UTC"
time="2020-08-10T17:48:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:48:34 +0000 UTC"
time="2020-08-10T17:48:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:48:44 +0000 UTC"
time="2020-08-10T17:48:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:48:54 +0000 UTC"
time="2020-08-10T17:48:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:49:04 +0000 UTC"
time="2020-08-10T17:48:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:49:14 +0000 UTC"
time="2020-08-10T17:49:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:49:24 +0000 UTC"
time="2020-08-10T17:49:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:49:34 +0000 UTC"
time="2020-08-10T17:49:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:49:44 +0000 UTC"
time="2020-08-10T17:49:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:49:54 +0000 UTC"
time="2020-08-10T17:49:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:50:04 +0000 UTC"
time="2020-08-10T17:49:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:50:14 +0000 UTC"
time="2020-08-10T17:50:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:50:24 +0000 UTC"
time="2020-08-10T17:50:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:50:34 +0000 UTC"
time="2020-08-10T17:50:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:50:44 +0000 UTC"
time="2020-08-10T17:50:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:50:54 +0000 UTC"
time="2020-08-10T17:50:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:51:04 +0000 UTC"
time="2020-08-10T17:50:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:51:14 +0000 UTC"
time="2020-08-10T17:51:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:51:24 +0000 UTC"
time="2020-08-10T17:51:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:51:34 +0000 UTC"
time="2020-08-10T17:51:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:51:44 +0000 UTC"
time="2020-08-10T17:51:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:51:54 +0000 UTC"
time="2020-08-10T17:51:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:52:04 +0000 UTC"
time="2020-08-10T17:51:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:52:14 +0000 UTC"
time="2020-08-10T17:52:04Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:52:24 +0000 UTC"
time="2020-08-10T17:52:14Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:52:34 +0000 UTC"
time="2020-08-10T17:52:24Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:52:44 +0000 UTC"
time="2020-08-10T17:52:34Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:52:54 +0000 UTC"
time="2020-08-10T17:52:44Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:53:04 +0000 UTC"
time="2020-08-10T17:52:54Z" level=info msg="certifying linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local until 2020-08-11 17:53:14 +0000 UTC"
# kubectl get events --sort-by=.metadata.creationTimestamp -n pfm-admins
19m         Normal    Requested           certificate/linkerd-identity-issuer                     Created new CertificateRequest resource "linkerd-identity-issuer-1356821256"
19m         Normal    Issued              certificate/linkerd-identity-issuer                     Certificate issued successfully
19m         Normal    GeneratedKey        certificate/linkerd-identity-issuer                     Generated a new private key
19m         Normal    PrivateKeyLost      certificate/linkerd-identity-issuer                     Lost private key for CertificateRequest "linkerd-identity-issuer-1356821256", deleting old resource
19m         Normal    CertificateIssued   certificaterequest/linkerd-identity-issuer-1356821256   Certificate fetched from issuer successfully
17m         Normal    SuccessfulCreate    replicaset/linkerd-grafana-76677ddd47                   Created pod: linkerd-grafana-76677ddd47-d7fsl
17m         Normal    SuccessfulCreate    replicaset/linkerd-prometheus-6c78f59c6                 Created pod: linkerd-prometheus-6c78f59c6-bnxsm
17m         Normal    ScalingReplicaSet   deployment/linkerd-grafana                              Scaled up replica set linkerd-grafana-76677ddd47 to 1
<unknown>   Normal    Scheduled           pod/linkerd-prometheus-6c78f59c6-bnxsm                  Successfully assigned pfm-admins/linkerd-prometheus-6c78f59c6-bnxsm to vkp-devkube-node-2.i
<unknown>   Normal    Scheduled           pod/linkerd-identity-5fb4f45888-55ws6                   Successfully assigned pfm-admins/linkerd-identity-5fb4f45888-55ws6 to vkp-devkube-node-1.i
<unknown>   Normal    Scheduled           pod/linkerd-sp-validator-594d84d4c7-tnnbd               Successfully assigned pfm-admins/linkerd-sp-validator-594d84d4c7-tnnbd to vkp-devkube-node-2.i
17m         Normal    SuccessfulCreate    replicaset/linkerd-tap-68b4c8c997                       Created pod: linkerd-tap-68b4c8c997-dhd46
17m         Normal    ScalingReplicaSet   deployment/linkerd-controller                           Scaled up replica set linkerd-controller-7766b66bfb to 1
17m         Normal    ScalingReplicaSet   deployment/linkerd-web                                  Scaled up replica set linkerd-web-6895444795 to 1
17m         Normal    SuccessfulCreate    replicaset/linkerd-identity-5fb4f45888                  Created pod: linkerd-identity-5fb4f45888-55ws6
<unknown>   Normal    Scheduled           pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Successfully assigned pfm-admins/linkerd-proxy-injector-5f5dc455f8-xspjt to vkp-devkube-node-3.i
<unknown>   Normal    Scheduled           pod/linkerd-web-6895444795-4jh5g                        Successfully assigned pfm-admins/linkerd-web-6895444795-4jh5g to vkp-devkube-node-1.i
17m         Normal    ScalingReplicaSet   deployment/linkerd-tap                                  Scaled up replica set linkerd-tap-68b4c8c997 to 1
<unknown>   Normal    Scheduled           pod/linkerd-tap-68b4c8c997-dhd46                        Successfully assigned pfm-admins/linkerd-tap-68b4c8c997-dhd46 to vkp-devkube-node-1.i
17m         Normal    ScalingReplicaSet   deployment/linkerd-proxy-injector                       Scaled up replica set linkerd-proxy-injector-5f5dc455f8 to 1
17m         Normal    SuccessfulCreate    replicaset/linkerd-sp-validator-594d84d4c7              Created pod: linkerd-sp-validator-594d84d4c7-tnnbd
17m         Normal    SuccessfulCreate    replicaset/linkerd-controller-7766b66bfb                Created pod: linkerd-controller-7766b66bfb-84tf8
<unknown>   Normal    Scheduled           pod/linkerd-controller-7766b66bfb-84tf8                 Successfully assigned pfm-admins/linkerd-controller-7766b66bfb-84tf8 to vkp-devkube-node-3.i
17m         Normal    SuccessfulCreate    replicaset/linkerd-destination-fbfb8dc98                Created pod: linkerd-destination-fbfb8dc98-j7t2b
17m         Normal    ScalingReplicaSet   deployment/linkerd-sp-validator                         Scaled up replica set linkerd-sp-validator-594d84d4c7 to 1
<unknown>   Normal    Scheduled           pod/linkerd-destination-fbfb8dc98-j7t2b                 Successfully assigned pfm-admins/linkerd-destination-fbfb8dc98-j7t2b to vkp-devkube-node-1.i
17m         Normal    SuccessfulCreate    replicaset/linkerd-web-6895444795                       Created pod: linkerd-web-6895444795-4jh5g
17m         Normal    SuccessfulCreate    replicaset/linkerd-proxy-injector-5f5dc455f8            Created pod: linkerd-proxy-injector-5f5dc455f8-xspjt
17m         Normal    ScalingReplicaSet   deployment/linkerd-destination                          Scaled up replica set linkerd-destination-fbfb8dc98 to 1
<unknown>   Normal    Scheduled           pod/linkerd-grafana-76677ddd47-d7fsl                    Successfully assigned pfm-admins/linkerd-grafana-76677ddd47-d7fsl to vkp-devkube-node-3.i
17m         Normal    ScalingReplicaSet   deployment/linkerd-prometheus                           Scaled up replica set linkerd-prometheus-6c78f59c6 to 1
17m         Normal    ScalingReplicaSet   deployment/linkerd-identity                             Scaled up replica set linkerd-identity-5fb4f45888 to 1
17m         Normal    Pulled              pod/linkerd-prometheus-6c78f59c6-bnxsm                  Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Created             pod/linkerd-grafana-76677ddd47-d7fsl                    Created container linkerd-init
17m         Normal    Pulled              pod/linkerd-grafana-76677ddd47-d7fsl                    Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-grafana-76677ddd47-d7fsl                    Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulled              pod/linkerd-sp-validator-594d84d4c7-tnnbd               Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-tap-68b4c8c997-dhd46                        Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Created             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Created container linkerd-init
17m         Normal    Pulled              pod/linkerd-controller-7766b66bfb-84tf8                 Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-controller-7766b66bfb-84tf8                 Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-destination-fbfb8dc98-j7t2b                 Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-web-6895444795-4jh5g                        Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulled              pod/linkerd-web-6895444795-4jh5g                        Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulling             pod/linkerd-identity-5fb4f45888-55ws6                   Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Created             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Created container linkerd-init
17m         Normal    Created             pod/linkerd-web-6895444795-4jh5g                        Created container linkerd-init
17m         Normal    Created             pod/linkerd-tap-68b4c8c997-dhd46                        Created container linkerd-init
17m         Normal    Started             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Started container linkerd-init
17m         Normal    Started             pod/linkerd-tap-68b4c8c997-dhd46                        Started container linkerd-init
17m         Normal    Pulled              pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Started             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Started container linkerd-init
17m         Normal    Started             pod/linkerd-identity-5fb4f45888-55ws6                   Started container linkerd-init
17m         Normal    Created             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Created container linkerd-init
17m         Normal    Pulled              pod/linkerd-identity-5fb4f45888-55ws6                   Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Pulled              pod/linkerd-tap-68b4c8c997-dhd46                        Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Started             pod/linkerd-controller-7766b66bfb-84tf8                 Started container linkerd-init
17m         Normal    Created             pod/linkerd-controller-7766b66bfb-84tf8                 Created container linkerd-init
17m         Normal    Started             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Started container linkerd-init
17m         Normal    Started             pod/linkerd-destination-fbfb8dc98-j7t2b                 Started container linkerd-init
17m         Normal    Created             pod/linkerd-identity-5fb4f45888-55ws6                   Created container linkerd-init
17m         Normal    Created             pod/linkerd-destination-fbfb8dc98-j7t2b                 Created container linkerd-init
17m         Normal    Pulled              pod/linkerd-destination-fbfb8dc98-j7t2b                 Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init:v1.3.3"
17m         Normal    Started             pod/linkerd-web-6895444795-4jh5g                        Started container linkerd-init
17m         Normal    Started             pod/linkerd-grafana-76677ddd47-d7fsl                    Started container linkerd-init
17m         Normal    Pulling             pod/linkerd-grafana-76677ddd47-d7fsl                    Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/grafana:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-controller-7766b66bfb-84tf8                 Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-controller-7766b66bfb-84tf8                 Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Created             pod/linkerd-controller-7766b66bfb-84tf8                 Created container public-api
17m         Normal    Pulling             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/prom/prometheus:v2.15.2"
17m         Normal    Pulled              pod/linkerd-prometheus-6c78f59c6-bnxsm                  Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/prom/prometheus:v2.15.2"
17m         Normal    Created             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Created container prometheus
17m         Normal    Pulling             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-grafana-76677ddd47-d7fsl                    Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-prometheus-6c78f59c6-bnxsm                  Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Created             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Created container linkerd-proxy
17m         Normal    Started             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Started container linkerd-proxy
17m         Normal    Pulling             pod/linkerd-controller-7766b66bfb-84tf8                 Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-controller-7766b66bfb-84tf8                 Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Created             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Created container sp-validator
17m         Normal    Created             pod/linkerd-controller-7766b66bfb-84tf8                 Created container linkerd-proxy
17m         Normal    Started             pod/linkerd-controller-7766b66bfb-84tf8                 Started container linkerd-proxy
17m         Normal    Pulling             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-identity-5fb4f45888-55ws6                   Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-sp-validator-594d84d4c7-tnnbd               Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Started             pod/linkerd-prometheus-6c78f59c6-bnxsm                  Started container prometheus
17m         Normal    Created             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Created container proxy-injector
17m         Normal    Started             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Started container proxy-injector
17m         Normal    Pulled              pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-tap-68b4c8c997-dhd46                        Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Created             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Created container linkerd-proxy
17m         Normal    Pulling             pod/linkerd-destination-fbfb8dc98-j7t2b                 Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-grafana-76677ddd47-d7fsl                    Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/grafana:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-web-6895444795-4jh5g                        Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/web:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Pulling             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Created             pod/linkerd-grafana-76677ddd47-d7fsl                    Created container grafana
17m         Normal    Started             pod/linkerd-grafana-76677ddd47-d7fsl                    Started container linkerd-proxy
17m         Normal    Created             pod/linkerd-grafana-76677ddd47-d7fsl                    Created container linkerd-proxy
17m         Normal    Pulled              pod/linkerd-grafana-76677ddd47-d7fsl                    Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Started             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Started container sp-validator
17m         Normal    Pulling             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Started             pod/linkerd-grafana-76677ddd47-d7fsl                    Started container grafana
17m         Normal    Started             pod/linkerd-controller-7766b66bfb-84tf8                 Started container public-api
17m         Normal    Pulling             pod/linkerd-destination-fbfb8dc98-j7t2b                 Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Created             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Created container linkerd-proxy
17m         Normal    Pulled              pod/linkerd-sp-validator-594d84d4c7-tnnbd               Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Started             pod/linkerd-identity-5fb4f45888-55ws6                   Started container identity
17m         Normal    Pulling             pod/linkerd-identity-5fb4f45888-55ws6                   Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Created             pod/linkerd-identity-5fb4f45888-55ws6                   Created container identity
17m         Normal    Pulled              pod/linkerd-web-6895444795-4jh5g                        Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/web:stable-2.8.1"
17m         Normal    Created             pod/linkerd-web-6895444795-4jh5g                        Created container web
17m         Normal    Pulled              pod/linkerd-identity-5fb4f45888-55ws6                   Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Created             pod/linkerd-tap-68b4c8c997-dhd46                        Created container tap
17m         Normal    Started             pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Started container linkerd-proxy
17m         Normal    Started             pod/linkerd-web-6895444795-4jh5g                        Started container web
17m         Normal    Pulling             pod/linkerd-web-6895444795-4jh5g                        Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-tap-68b4c8c997-dhd46                        Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Started             pod/linkerd-sp-validator-594d84d4c7-tnnbd               Started container linkerd-proxy
17m         Normal    Started             pod/linkerd-tap-68b4c8c997-dhd46                        Started container tap
17m         Normal    Pulling             pod/linkerd-tap-68b4c8c997-dhd46                        Pulling image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-destination-fbfb8dc98-j7t2b                 Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller:stable-2.8.1"
17m         Normal    Created             pod/linkerd-destination-fbfb8dc98-j7t2b                 Created container destination
17m         Normal    Started             pod/linkerd-destination-fbfb8dc98-j7t2b                 Started container destination
17m         Normal    Pulled              pod/linkerd-web-6895444795-4jh5g                        Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Pulled              pod/linkerd-destination-fbfb8dc98-j7t2b                 Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Created             pod/linkerd-destination-fbfb8dc98-j7t2b                 Created container linkerd-proxy
17m         Normal    Started             pod/linkerd-destination-fbfb8dc98-j7t2b                 Started container linkerd-proxy
17m         Normal    Started             pod/linkerd-web-6895444795-4jh5g                        Started container linkerd-proxy
17m         Normal    Pulled              pod/linkerd-tap-68b4c8c997-dhd46                        Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Created             pod/linkerd-tap-68b4c8c997-dhd46                        Created container linkerd-proxy
17m         Normal    Created             pod/linkerd-web-6895444795-4jh5g                        Created container linkerd-proxy
17m         Normal    Created             pod/linkerd-identity-5fb4f45888-55ws6                   Created container linkerd-proxy
17m         Normal    Pulled              pod/linkerd-identity-5fb4f45888-55ws6                   Successfully pulled image "registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy:stable-2.8.1"
17m         Normal    Started             pod/linkerd-identity-5fb4f45888-55ws6                   Started container linkerd-proxy
2m32s       Warning   Unhealthy           pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Readiness probe failed: HTTP probe failed with statuscode: 503
2m32s       Warning   Unhealthy           pod/linkerd-controller-7766b66bfb-84tf8                 Readiness probe failed: HTTP probe failed with statuscode: 503
17m         Normal    Started             pod/linkerd-tap-68b4c8c997-dhd46                        Started container linkerd-proxy
2m31s       Warning   Unhealthy           pod/linkerd-grafana-76677ddd47-d7fsl                    Readiness probe failed: HTTP probe failed with statuscode: 503
17m         Warning   Unhealthy           pod/linkerd-web-6895444795-4jh5g                        Readiness probe failed: Get http://10.212.25.148:9994/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2m29s       Warning   Unhealthy           pod/linkerd-prometheus-6c78f59c6-bnxsm                  Readiness probe failed: HTTP probe failed with statuscode: 503
17m         Warning   Unhealthy           pod/linkerd-sp-validator-594d84d4c7-tnnbd               Readiness probe failed: Get http://10.212.27.28:9997/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
17m         Warning   Unhealthy           pod/linkerd-controller-7766b66bfb-84tf8                 Readiness probe failed: Get http://10.212.30.152:9995/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
17m         Warning   Unhealthy           pod/linkerd-proxy-injector-5f5dc455f8-xspjt             Readiness probe failed: Get http://10.212.30.139:9995/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2m36s       Warning   Unhealthy           pod/linkerd-tap-68b4c8c997-dhd46                        Readiness probe failed: HTTP probe failed with statuscode: 503
2m36s       Warning   Unhealthy           pod/linkerd-destination-fbfb8dc98-j7t2b                 Readiness probe failed: HTTP probe failed with statuscode: 503
2m36s       Warning   Unhealthy           pod/linkerd-web-6895444795-4jh5g                        Readiness probe failed: HTTP probe failed with statuscode: 503
17m         Warning   Unhealthy           pod/linkerd-tap-68b4c8c997-dhd46                        Readiness probe failed: Get http://10.212.25.184:9998/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2m34s       Warning   Unhealthy           pod/linkerd-identity-5fb4f45888-55ws6                   Readiness probe failed: HTTP probe failed with statuscode: 503
17m         Warning   Unhealthy           pod/linkerd-grafana-76677ddd47-d7fsl                    Readiness probe failed: Get http://10.212.30.147:3000/api/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2m33s       Warning   Unhealthy           pod/linkerd-sp-validator-594d84d4c7-tnnbd               Readiness probe failed: HTTP probe failed with statuscode: 503
17m         Warning   Unhealthy           pod/linkerd-identity-5fb4f45888-55ws6                   Readiness probe failed: Get http://10.212.25.131:9990/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
17m         Warning   Unhealthy           pod/linkerd-prometheus-6c78f59c6-bnxsm                  Liveness probe failed: HTTP probe failed with statuscode: 503

Bare metal.
If I install linkerd2 by command:

linkerd install -L pfm-admins|sed 's/gcr.io/registry-gitlab.corp.domain.com\/rb-adm/g' | kubectl apply -f 

All good

But from helm chart by this instruction don’t work.

# kubectl logs -n pfm-admins linkerd-web-6895444795-4jh5g -c linkerd-proxy
time="2020-08-10T17:45:43Z" level=info msg="running version stable-2.8.1"
[     0.14133218s]  INFO linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.14184658s]  INFO linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.14194496s]  INFO linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.14205998s]  INFO linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.14216166s]  INFO linkerd2_proxy: Local identity is linkerd-web.pfm-admins.serviceaccount.identity.pfm-admins.cluster.local
[     0.14228634s]  INFO linkerd2_proxy: Identity verified via linkerd-identity.pfm-admins.svc.cluster.local:8080 (linkerd-identity.pfm-admins.serviceaccount.identity.pfm-admins.cluster.loca
l)
[     0.14250089s]  INFO linkerd2_proxy: Destinations resolved via linkerd-dst.pfm-admins.svc.cluster.local:8086 (linkerd-destination.pfm-admins.serviceaccount.identity.pfm-admins.cluster.lo
cal)
[     0.15538873s]  INFO linkerd2_app_inbound: Serving listen.addr=0.0.0.0:4143
[     4.139331218s]  WARN inbound:accept{peer.addr=10.10.19.2:58398}:source{target.addr=10.212.25.148:9994}: linkerd2_app_core::errors: Failed to proxy request: request timed out

@sbvitok thanks for sharing these logs.

It’s interesting that the command line install works without any issues. Have you made any changes to the values.yaml file that is used by helm during installation? Please share the helm install command.

Another thing to try is to run a curl container, exec into it, and attempt to connect to the readiness probe endpoints to see what response is returned: curl -v http://linker-controller.pfm-admins:9995/ready

linkerd services has no endpoints because linkerd-proxy container is not ready in each linkerd pod

# kubectl get endpoints -n pfm-admins 
NAME                                           ENDPOINTS                                                              AGE
alert-webhook                                  10.212.25.188:5000                                                     62d
alert-webhook-headless                         10.212.25.188:5000                                                     62d
alertmanager-operated                          10.212.25.186:9094,10.212.25.186:9093,10.212.25.186:9094               18d
cert-manager                                   10.212.27.17:9402                                                      63d
cert-manager-webhook                           10.212.25.167:8443                                                     63d
filebeat-metrics                               10.212.25.173:9479,10.212.25.224:9479,10.212.26.28:9479 + 3 more...    26d
gateway                                        10.212.25.177:8443                                                     63d
gateway-proxy                                  10.212.25.190:443,10.212.27.40:443,10.212.25.190:80 + 1 more...        63d
gloo                                           10.212.25.176:9966,10.212.25.176:9977,10.212.25.176:9988 + 1 more...   63d
internal-proxy                                 10.212.27.29:443,10.212.30.175:443,10.212.27.29:80 + 1 more...         20d
linkerd-controller-api                                                                                                66m
linkerd-dst                                                                                                           66m
linkerd-grafana                                                                                                       66m
linkerd-identity                                                                                                      66m
linkerd-prometheus                                                                                                    66m
linkerd-proxy-injector                                                                                                66m
linkerd-sp-validator                                                                                                  66m
linkerd-tap                                                                                                           66m
linkerd-web                                                                                                           66m
prometheus-operated                            10.212.30.180:9090                                                     18d
prometheus-operator-alertmanager               10.212.25.186:9093                                                     18d
prometheus-operator-kube-state-metrics         10.212.27.1:8080                                                       18d
prometheus-operator-operator                   10.212.27.45:8080                                                      18d
prometheus-operator-prometheus                 10.212.30.180:9090                                                     18d
prometheus-operator-prometheus-node-exporter   10.10.18.2:9100,10.10.18.3:9100,10.10.18.4:9100 + 3 more...            18d
redash                                         10.212.30.168:5000                                                     31d
redash-redis-headless                          10.212.25.147:6379                                                     31d
redash-redis-master                            10.212.25.147:6379                                                     31d
redis                                          10.212.25.147:6379,10.212.30.172:6379                                  62d

But if got directly to pod ip

linkerd-controller-7766b66bfb-v4c29                       1/2     Running   1          64m    10.212.27.8     vkp-devkube-node-2.i     <none>           <none>

public-api probe is ok, but linkerd-proxy not ready

# kubectl run  --generator=run-pod/v1 tmp-shell --rm -i --tty -n pfm-admins  --image registry-gitlab.corp.domain.com/rb-adm/hub/nicolaka/netshoot -- /bin/bash
Flag --generator has been deprecated, has no effect and will be removed in the future.
If you don't see a command prompt, try pressing enter.
bash-5.0# curl -v http://10.212.27.8:9995/ready
*   Trying 10.212.27.8:9995...
* TCP_NODELAY set
* Connected to 10.212.27.8 (10.212.27.8) port 9995 (#0)
> GET /ready HTTP/1.1
> Host: 10.212.27.8:9995
> User-Agent: curl/7.65.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Mon, 10 Aug 2020 19:54:01 GMT
< content-length: 3
< content-type: text/plain; charset=utf-8
< 
ok
* Connection #0 to host 10.212.27.8 left intact
bash-5.0# curl -v http://10.212.27.8:4191/ready
*   Trying 10.212.27.8:4191...
* TCP_NODELAY set
* Connected to 10.212.27.8 (10.212.27.8) port 4191 (#0)
> GET /ready HTTP/1.1
> Host: 10.212.27.8:4191
> User-Agent: curl/7.65.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 10
< date: Mon, 10 Aug 2020 19:54:10 GMT
< 
not ready
* Connection #0 to host 10.212.27.8 left intact
bash-5.0# 

helm values:

# cat values-pfm.yaml
global:
  namespace: pfm-admins
  # proxy configuration
  proxy:
    image:
      name: registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy
    #waitBeforeExitSeconds: 10
    #logLevel: linkerd=debug

  # proxy-init configuration
  proxyInit:
    image:
      name: registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/proxy-init

controllerImage: registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/controller

identity:
  issuer:
    scheme: kubernetes.io/tls


# prometheus configuration
prometheusImage: registry-gitlab.corp.domain.com/rb-adm/hub/prom/prometheus:v2.15.2

# web configuration
webImage: registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/web

# If the namespace is controlled by an external tool or can't be installed with Helm
# you can disable its installation. In this case:
# - The namespace created by the external tool must match the namespace value above
# - The external tool needs to create the namespace with the label:
#     config.linkerd.io/admission-webhooks: disabled
installNamespace: false


grafana:
  image:
    name: registry-gitlab.corp.domain.com/rb-adm/hub/linkerd-io/grafana

helm3 command:

helm install --set-file global.identityTrustAnchorsPEM=ca.crt -f values-pfm.yaml -n pfm-admins linkerd2 .
helm install -f values-pfm.yaml --set-file global.identityTrustAnchorsPEM=ca.crt --set-file identity.issuer.tls.crtPEM=issuer.crt --set-file identity.issuer.tls.keyPEM=issuer.key --set identity.issuer.crtExpiry=$(date -d '+8760 hour' +"%Y-%m-%dT%H:%M:%SZ") linkerd2 .

Also works.
The problem in this

identity:
  issuer:
    scheme: kubernetes.io/tls

But linkerd check says that all good whit identity…

# linkerd check -L pfm-admins
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-api
-----------
/ pod/linkerd-controller-7766b66bfb-v4c29 container linkerd-proxy is not ready

@sbvitok I think we need to get more information from the proxies about why the requests are timing out.

Can you update the proxy log level to debug and restart one of the control-plane deployments so that we can collect the proxy logs from startup?

You might consider taking a look at the trace level logging as well. It’s pretty verbose, but will be helpful in understanding what’s happening.

Here is the trace log
https://file.io/198BSHzZSoZu

@sbvitok the link to the file isn’t working for me

When you say that kubernetes.io/tls is the problem, what do you mean?

that all combinations work except this

https://0x0.st/i35J.log

@sbvitok this is the issue:

[     0.28856748s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[     4.415878178s]  WARN inbound:accept{peer.addr=10.10.19.2:57626}:source{target.addr=10.212.25.135:9990}: linkerd2_app_core::errors: Failed to proxy request: request timed out
[    10.37330210s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    20.45677001s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    30.54428508s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    40.63847652s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    50.72222146s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    60.80715344s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    70.89467054s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    80.98817403s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[    90.106976372s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   100.116719414s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   110.124077728s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   120.134673177s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   130.143319573s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   140.151627032s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   150.161173116s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   160.169114736s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   170.178509053s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   180.187087215s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   190.196055051s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   200.205261446s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   210.214203441s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   220.223777189s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   230.232965512s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   240.244685375s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   250.254078502s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   260.262886576s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   270.272947326s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   280.283699518s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   290.292152280s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   300.301041001s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   310.309429938s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   320.317266480s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   330.326646021s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   340.334920193s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   350.343284577s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   360.354048433s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   370.363367244s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   380.372279656s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   390.380778324s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer
[   400.389204299s] ERROR daemon:identity: linkerd2_proxy_identity::certify: Received invalid ceritficate: invalid certificate: UnknownIssuer

When you install Linkerd using the CLI, do you specify the same certs that you use when installing with helm? Have a look here for reference.

I resolve problem!
The problem was in settings ClusterIssuer of cert-manager
I set path for vault to “pki/sign/[role-name]” - by this path vault sign non CA certs, co linkerd can’t sign certs for proxy container
To make it works I change path to “pki/root/sign-intermediate” and now all works. linkerd-identity-issuer secret has CA: TRUE cert and can sign certs for proxy container.

thanks for you help!

@sbvitok that’s great news, I’m glad to hear that you solved it and that it wasn’t an issue with Linkerd.

Feel free to join us on https://slack.linkerd.io if you have more questions in the future. There are lots of friendly people there who have experience using Linkerd.