503 when clicking Grafana link in linkerd 2.7.0

I have a strange problem on one of my k8s clusters.

I am running 4 identical clusters and 3 of them are working nicely however on a single cluster I have a problem where I click on the Grafana link within Linkerd and I get a 503 error.

I am just running a standard ha install of Linkerd, I have not changed anything from the default options.

I have tried a complete uninstall and reinstall but I still get the same issue, the error in the web log file is (IP addresses removed):

44.784989922s] WARN outbound:accept{peer.addr=x.x.x.x:35660}:source{target.addr=x.x.x.x:3000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion

Does anybody have any ideas on this?

Hi @jonbbbb,

We can take a look at the output from the linkerd check command to make sure the control plane is healthy.

Also, check the grafana logs with kubetcl logs -f <linkerd grafana pod> -n linkerd.

If you can share that output, that will help.

Charles

Thanks for the help, linkerd check appears all fine and dandy, I’ve not looked at the grafana logs before so no idea if there is something in there obvious or not

kubernetes-api

√ can initialize the client
√ can query the Kubernetes API

kubernetes-version

√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence

√ ‘linkerd-config’ config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config

√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity

√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-api

√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version

√ can determine the latest version
√ cli is up-to-date

control-plane-version

√ control plane is up-to-date
√ control plane and cli versions match

linkerd-ha-checks

√ pod injection disabled on kube-system

Status check results are √

t=2020-03-13T09:11:19+0000 lvl=info msg=“Starting Grafana” logger=server version=6.2.5 commit=6082d19 branch=HEAD compiled=2019-06-25T17:56:19+0000
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/data”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.provisioning=/etc/grafana/provisioning”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.log.mode=console”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from Environment variable” logger=settings var=“GF_PATHS_DATA=/data”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from Environment variable” logger=settings var=“GF_PATHS_LOGS=/var/log/grafana”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from Environment variable” logger=settings var=“GF_PATHS_PLUGINS=/var/lib/grafana/plugins”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Config overridden from Environment variable” logger=settings var=“GF_PATHS_PROVISIONING=/etc/grafana/provisioning”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2020-03-13T09:11:19+0000 lvl=info msg=“Path Data” logger=settings path=/data
t=2020-03-13T09:11:19+0000 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2020-03-13T09:11:19+0000 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2020-03-13T09:11:19+0000 lvl=info msg=“Path Provisioning” logger=settings path=/etc/grafana/provisioning
t=2020-03-13T09:11:19+0000 lvl=info msg=“App mode production” logger=settings
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing SqlStore” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2020-03-13T09:11:19+0000 lvl=info msg=“Starting DB migration” logger=migrator
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create migration_log table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create user table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index user.login”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index user.email”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index UQE_user_login - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index UQE_user_email - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Rename table user to user_v1 - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create user table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_user_login - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_user_email - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“copy data_source v1 to v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop old table user_v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column help_flags1 to user table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update user table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add last_seen_at column to user”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add missing user data”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create temp user table v1-7”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_temp_user_email - v1-7”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_temp_user_org_id - v1-7”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_temp_user_code - v1-7”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_temp_user_status - v1-7”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update temp_user table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create star table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index star.user_id_dashboard_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create org table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_org_name - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create org_user table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_org_user_org_id - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_org_user_org_id_user_id - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update org table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update org_user table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Migrate all Read Only Viewers to Viewers”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index dashboard.account_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index dashboard_account_id_slug”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard_tag table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index dashboard_tag.dasboard_id_term”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index UQE_dashboard_tag_dashboard_id_term - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Rename table dashboard to dashboard_v1 - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_dashboard_org_id - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_dashboard_org_id_slug - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“copy dashboard v1 to v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop table dashboard_v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“alter dashboard.data to mediumtext v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column updated_by in dashboard - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column created_by in dashboard - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column gnetId in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add index for gnetId in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column plugin_id in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add index for plugin_id in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add index for dashboard_id in dashboard_tag”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard_tag table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column folder_id in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column isFolder in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column has_acl in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column uid in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update uid column values in dashboard”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add unique index dashboard_org_id_uid”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Remove unique index org_id_slug”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard title length”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add unique index for dashboard_org_id_title_folder_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard_provisioning”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard_provisioning v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_dashboard_provisioning_dashboard_id - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_dashboard_provisioning_dashboard_id_name - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“copy dashboard_provisioning v1 to v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop dashboard_provisioning_tmp_qwerty”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add check_sum column”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create data_source table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index data_source.account_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index data_source.account_id_name”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index IDX_data_source_account_id - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index UQE_data_source_account_id_name - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Rename table data_source to data_source_v1 - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create data_source table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_data_source_org_id - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_data_source_org_id_name - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“copy data_source v1 to v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop old table data_source_v1 #2
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column with_credentials”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add secure json data column”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update data_source table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update initial version to 1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add read_only data column”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Migrate logging ds to loki ds”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update json_data with nulls”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create api_key table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index api_key.account_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index api_key.key”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index api_key.account_id_name”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index IDX_api_key_account_id - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index UQE_api_key_key - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index UQE_api_key_account_id_name - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Rename table api_key to api_key_v1 - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create api_key table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_api_key_org_id - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_api_key_key - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_api_key_org_id_name - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“copy api_key v1 to v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop old table api_key_v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update api_key table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard_snapshot table v4”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop table dashboard_snapshot_v4 #1
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard_snapshot table v5 #2
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_dashboard_snapshot_key - v5”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_dashboard_snapshot_delete_key - v5”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_dashboard_snapshot_user_id - v5”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“alter dashboard_snapshot to mediumtext v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard_snapshot table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column external_delete_url to dashboard_snapshots table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create quota table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_quota_org_id_user_id_target - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update quota table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create plugin_setting table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index UQE_plugin_setting_org_id_plugin_id - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column plugin_version to plugin_settings”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update plugin_setting table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create session table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop old table playlist table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop old table playlist_item table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create playlist table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create playlist item table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update playlist table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update playlist_item table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop preferences table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop preferences table v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create preferences table v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update preferences table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column team_id in preferences”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update team_id column values in preferences”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create alert table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id="add index alert org_id & id "
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index alert state”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index alert dashboard_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create alert_notification table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column is_default”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column frequency”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column send_reminder”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column disable_resolve_message”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index alert_notification org_id & name”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update alert table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update alert_notification table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create notification_journal table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index notification_journal org_id & alert_id & notifier_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop alert_notification_journal”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create alert_notification_state table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index alert_notification_state org_id & alert_id & notifier_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add for to alert table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column uid in alert_notification”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update uid column values in alert_notification”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add unique index alert_notification_org_id_uid”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Remove unique index org_id_name”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop old annotation table v4”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create annotation table v5”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 0 v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 1 v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 2 v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 3 v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 4 v3”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update annotation table charset”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column region_id to annotation table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Drop category_id index”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column tags to annotation table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Create annotation_tag table v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add unique index annotation_tag.annotation_id_tag_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Update alert annotations and set TEXT to empty”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add created time to annotation table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add updated time to annotation table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add index for created in annotation table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add index for updated in annotation table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Convert existing annotations from seconds to milliseconds”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create test_data table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard_version table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index dashboard_version.dashboard_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index dashboard_version.dashboard_id and dashboard_version.version”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Set dashboard version to 1 where 0”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“save existing dashboard data in dashboard_version table v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“alter dashboard_version.data to mediumtext v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create team table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index team.org_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index team_org_id_name”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create team member table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index team_member.org_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index team_member_org_id_team_id_user_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column email to team table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column external to team_member table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add column permission to team_member table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create dashboard acl table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index dashboard_acl_dashboard_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index dashboard_acl_dashboard_id_user_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index dashboard_acl_dashboard_id_team_id”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“save default acl rules in dashboard_acl table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create tag table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index tag.key_value”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create login attempt table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index login_attempt.username”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop index IDX_login_attempt_username - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Rename table login_attempt to login_attempt_tmp_qwerty - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create login_attempt v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_login_attempt_username - v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“copy login_attempt v1 to v2”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“drop login_attempt_tmp_qwerty”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create user auth table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create index IDX_user_auth_auth_module_auth_id - v1”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“alter user_auth.auth_id to length 190”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add OAuth access token to user_auth”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add OAuth refresh token to user_auth”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add OAuth token type to user_auth”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add OAuth expiry to user_auth”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“Add index to user_id column in user_auth”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create server_lock table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add index server_lock.operation_uid”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create user auth token table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index user_auth_token.auth_token”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index user_auth_token.prev_auth_token”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“create cache_data table”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Executing migration” logger=migrator id=“add unique index cache_data.cache_key”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Created default admin” logger=sqlstore user=admin
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing HTTPServer” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing InternalMetricsService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing RemoteCache” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing QuotaService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing LoginService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing PluginManager” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Starting plugin search” logger=plugins
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing RenderingService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing AlertingService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing DatasourceCacheService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing HooksService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing SearchService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing ServerLockService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing TracingService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing UsageStatsService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing UserAuthTokenService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing CleanUpService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing NotificationService” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing provisioningServiceImpl” logger=server
t=2020-03-13T09:11:19+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=prometheus
t=2020-03-13T09:11:19+0000 lvl=eror msg=“Can’t read alert notification provisioning files from directory” logger=provisioning.notifiers path=/etc/grafana/provisioning/notifiers error=“open /etc/grafana/provisioning/notifiers: no such file or directory”
t=2020-03-13T09:11:19+0000 lvl=info msg=“Initializing Stream Manager”
t=2020-03-13T09:11:19+0000 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3000 protocol=http subUrl=/grafana socket=

When you click on the grafana link it isn’t adding to that error log nor the linkerd-proxy log of grafana.
it does however add to the linkerd-proxy log of the web:

kubectl logs -f linkerd-web-744b87dd79-w6jtv -n linkerd linkerd-proxy
time=“2020-03-13T09:06:27Z” level=info msg=“running version stable-2.7.0”
[ 0.6829566s] INFO linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.6850019s] INFO linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.6855811s] INFO linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.6859141s] INFO linkerd2_proxy: Tap interface on 0.0.0.0:4190
[ 0.6861816s] INFO linkerd2_proxy: Local identity is linkerd-web.linkerd.serviceaccount.identity.linkerd.cluster.local
[ 0.6867087s] INFO linkerd2_proxy: Identity verified via linkerd-identity.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.6874647s] INFO linkerd2_proxy: Destinations resolved via linkerd-dst.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.7162928s] INFO inbound: linkerd2_app_inbound: serving listen.addr=0.0.0.0:4143
[ 0.20706763s] INFO daemon:identity: linkerd2_app: Certified identity: linkerd-web.linkerd.serviceaccount.identity.linkerd.cluster.local
[ 44.784989922s] WARN outbound:accept{peer.addr=10.29.95.103:35660}:source{target.addr=172.20.95.193:3000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion
[ 27633.195253516s] WARN outbound:accept{peer.addr=10.29.95.103:46722}:source{target.addr=172.20.95.193:3000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion
[ 27696.423738719s] WARN outbound:accept{peer.addr=10.29.95.103:46722}:source{target.addr=172.20.95.193:3000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion
[ 27721.980557631s] WARN outbound:accept{peer.addr=10.29.95.103:46722}:source{target.addr=172.20.95.193:3000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion
[ 27759.82438142s] WARN outbound:accept{peer.addr=10.29.95.103:46722}:source{target.addr=172.20.95.193:3000}: linkerd2_app_core::errors: Failed to proxy request: the request could not be dispatched in a timely fashion

@jonbbbb this looks suspect:

Can you try to restart the grafana deployment and check whether the same errors occur? I think the configuration for the notifiers is in a ConfigMap

hi @jonbbbb just checking to see if this is resolved.

Have you been able to access Grafana?

I have redeployed linkerd from scratch but still can’t access grafana I’m afraid.
Tried a kill of the grafana pod and let it re-create, still can’t access it.

It is strange as this installation was working fine ~ a week ago and we have 3 identical installations that work perfectly (all using the same scripts to install)

Sorry to hear that you haven’t been able to access Grafana on this instance. That’s frustrating.

It definitely sounds like a problem with network access for this particular cluster. Is the cluster a managed cluster like GKE, AKS, or EKS?

Any chance that there is a firewall configured for port 3000?

This sounds like a good opportunity to use the debug sidecar: https://linkerd.io/2/tasks/using-the-debug-container/

You can inject it into the linkerd-web deployment and use it to try to connect to the grafana pod, as well as monitor traffic to the grafana pod.

Charles

Yep a managed EKS cluster, I’ll give the debug container a go thanks!

Do you have any suggestions on which commands I should be running within the debug sidecar?
I can curl the grafana endpoint from within the debug sidecar on linkerd-grafana.linkerd.svc.cluster.local:3000 without it complaining that it can’t resolve it (if I curl an endpoint that doesn’t exist it complains that the host couldn’t be resolved)

root@linkerd-web-67d4fcf966-2k97n:/# curl linkerd-controller-api.linkerd.svc.cluster.local
root@linkerd-web-67d4fcf966-2k97n:/# curl grafana.linkerd.svc.cluster.local
curl: (6) Could not resolve host: grafana.linkerd.svc.cluster.local
root@linkerd-web-67d4fcf966-2k97n:/# curl linkerd-grafana.linkerd.svc.cluster.local
root@linkerd-web-67d4fcf966-2k97n:/# curl linkerd-grafana.linkerd.svc.cluster.local:3000

Error log when I click the link to grafana within web

6044 436.865078625 10.29.29.181 → 10.29.29.248 TCP 76 54458 → 4191 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=2084600226 TSecr=0 WS=512
6045 436.865178278 10.29.29.248 → 10.29.29.181 TCP 76 4191 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=8961 SACK_PERM=1 TSval=3536337201 TSecr=2084600226 WS=512
6046 436.865211575 10.29.29.181 → 10.29.29.248 TCP 68 54458 → 4191 [ACK] Seq=1 Ack=1 Win=27136 Len=0 TSval=2084600226 TSecr=3536337201
6047 436.865322609 10.29.29.181 → 10.29.29.248 HTTP 188 GET /ready HTTP/1.1
6048 436.865331201 10.29.29.248 → 10.29.29.181 TCP 68 4191 → 54458 [ACK] Seq=1 Ack=121 Win=27136 Len=0 TSval=3536337201 TSecr=2084600226
6049 436.865414165 10.29.29.248 → 10.29.29.181 HTTP 149 HTTP/1.1 200 OK
6050 436.865429267 10.29.29.181 → 10.29.29.248 TCP 68 54458 → 4191 [ACK] Seq=121 Ack=82 Win=27136 Len=0 TSval=2084600226 TSecr=3536337201
6051 436.865451403 10.29.29.248 → 10.29.29.181 TCP 68 4191 → 54458 [FIN, ACK] Seq=82 Ack=121 Win=27136 Len=0 TSval=3536337201 TSecr=2084600226
6052 436.865514596 10.29.29.181 → 10.29.29.248 TCP 68 54458 → 4191 [FIN, ACK] Seq=121 Ack=83 Win=27136 Len=0 TSval=2084600226 TSecr=3536337201
6053 436.865522495 10.29.29.248 → 10.29.29.181 TCP 68 4191 → 54458 [ACK] Seq=83 Ack=122 Win=27136 Len=0 TSval=3536337201 TSecr=2084600226
6054 436.949508282 10.29.29.248 → 172.20.0.10 DNS 147 Standard query 0x7526 A linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6055 436.949975427 172.20.0.10 → 10.29.29.248 DNS 240 Standard query response 0x7526 No such name A linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6056 436.950168675 10.29.29.248 → 172.20.0.10 DNS 147 Standard query 0x13e3 AAAA linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6057 436.950506812 172.20.0.10 → 10.29.29.248 DNS 240 Standard query response 0x13e3 No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6058 436.950617788 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0xa4d0 A linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local OPT
6059 436.950863207 172.20.0.10 → 10.29.29.248 DNS 232 Standard query response 0xa4d0 No such name A linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6060 436.950954488 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0x3714 AAAA linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local OPT
6061 436.951590598 172.20.0.10 → 10.29.29.248 DNS 232 Standard query response 0x3714 No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6062 436.951685116 10.29.29.248 → 172.20.0.10 DNS 135 Standard query 0x3c25 A linkerd-controller-api.linkerd.svc.cluster.local.cluster.local OPT
6063 436.952212534 172.20.0.10 → 10.29.29.248 DNS 228 Standard query response 0x3c25 No such name A linkerd-controller-api.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6064 436.952308216 10.29.29.248 → 172.20.0.10 DNS 135 Standard query 0x6ca6 AAAA linkerd-controller-api.linkerd.svc.cluster.local.cluster.local OPT
6065 436.952588850 172.20.0.10 → 10.29.29.248 DNS 228 Standard query response 0x6ca6 No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6066 436.952673149 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0x927b A linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6067 436.953125171 172.20.0.10 → 10.29.29.248 DNS 253 Standard query response 0x927b No such name A linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-ev2vegdtb9d.aws.escloud.co.uk OPT
6068 436.953213708 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0xba7d AAAA linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6069 436.954039155 172.20.0.10 → 10.29.29.248 DNS 253 Standard query response 0xba7d No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-ev2vegdtb9d.aws.escloud.co.uk OPT
6070 436.954174775 10.29.29.248 → 172.20.0.10 DNS 148 Standard query 0x117c A linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6071 436.954894058 172.20.0.10 → 10.29.29.248 DNS 148 Standard query response 0x117c No such name A linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6072 436.955004037 10.29.29.248 → 172.20.0.10 DNS 148 Standard query 0xf4dd AAAA linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6073 436.955935502 172.20.0.10 → 10.29.29.248 DNS 148 Standard query response 0xf4dd No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6074 436.956032983 10.29.29.248 → 172.20.0.10 DNS 121 Standard query 0xbddd A linkerd-controller-api.linkerd.svc.cluster.local OPT
6075 436.956578548 172.20.0.10 → 10.29.29.248 DNS 185 Standard query response 0xbddd A linkerd-controller-api.linkerd.svc.cluster.local A 172.20.185.231 OPT
6076 437.276116826 10.29.29.110 → 10.29.29.248 HTTP 982 GET /api/tps-reports?resource_type=namespace&namespace=all&tcp_stats=true&window=1m HTTP/1.1
6077 437.276141668 10.29.29.248 → 10.29.29.110 TCP 68 8084 → 33346 [ACK] Seq=188 Ack=2004 Win=31232 Len=0 TSval=4247232007 TSecr=4076823145
6078 437.276445718 127.0.0.1 → 127.0.0.1 HTTP 920 GET /api/tps-reports?resource_type=namespace&namespace=all&tcp_stats=true&window=1m HTTP/1.1
6079 437.276689373 10.29.29.248 → 127.0.0.1 HTTP 363 POST /api/v1/StatSummary HTTP/1.1
6080 437.276992680 10.29.29.248 → 10.29.28.60 TLSv1.2 204 Application Data
6081 437.296122569 10.29.28.60 → 10.29.29.248 TLSv1.2 140 Application Data
6082 437.296158319 10.29.29.248 → 10.29.28.60 TCP 68 54022 → 8085 [ACK] Seq=1945 Ack=2696 Win=31744 Len=0 TSval=1843271875 TSecr=1154942137
6083 437.296128712 10.29.28.60 → 10.29.29.248 TLSv1.2 1276 Application Data
6084 437.296164722 10.29.29.248 → 10.29.28.60 TCP 68 54022 → 8085 [ACK] Seq=1945 Ack=3904 Win=34304 Len=0 TSval=1843271875 TSecr=1154942137
6085 437.296407190 172.20.185.231 → 10.29.29.248 HTTP 1373 HTTP/1.1 200 OK
6086 437.296427623 10.29.29.248 → 127.0.0.1 TCP 68 52600 → 4140 [ACK] Seq=2056 Ack=9073 Win=108032 Len=0 TSval=1347275259 TSecr=1275662989
6087 437.298166079 127.0.0.1 → 127.0.0.1 TCP 4164 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6088 437.298184009 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=7006 Ack=60939 Win=464896 Len=0 TSval=1323613080 TSecr=1323613080
6089 437.298196298 127.0.0.1 → 127.0.0.1 TCP 5483 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6090 437.298203414 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=7006 Ack=66354 Win=464896 Len=0 TSval=1323613080 TSecr=1323613080
6091 437.298234270 127.0.0.1 → 127.0.0.1 HTTP 75 HTTP/1.1 200 OK (application/json)
6092 437.298237445 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=7006 Ack=66361 Win=464896 Len=0 TSval=1323613080 TSecr=1323613080
6093 437.298288452 10.29.29.248 → 10.29.29.110 TCP 9017 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6094 437.298331329 10.29.29.110 → 10.29.29.248 TCP 68 33346 → 8084 [ACK] Seq=2004 Ack=9137 Win=46080 Len=0 TSval=4076823167 TSecr=4247232029
6095 437.298340755 10.29.29.248 → 10.29.29.110 TCP 632 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6096 437.298352490 10.29.29.110 → 10.29.29.248 TCP 68 33346 → 8084 [ACK] Seq=2004 Ack=9701 Win=64000 Len=0 TSval=4076823167 TSecr=4247232029
6097 437.298411175 10.29.29.248 → 10.29.29.110 HTTP 73 HTTP/1.1 200 OK (application/json)
6098 437.298423827 10.29.29.110 → 10.29.29.248 TCP 68 33346 → 8084 [ACK] Seq=2004 Ack=9706 Win=64000 Len=0 TSval=4076823167 TSecr=4247232029
6099 437.557725948 10.29.29.181 → 10.29.29.248 TCP 76 54512 → 4191 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=2084600918 TSecr=0 WS=512
6100 437.557818925 10.29.29.248 → 10.29.29.181 TCP 76 4191 → 54512 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=8961 SACK_PERM=1 TSval=3536337894 TSecr=2084600918 WS=512
6101 437.557841624 10.29.29.181 → 10.29.29.248 TCP 68 54512 → 4191 [ACK] Seq=1 Ack=1 Win=27136 Len=0 TSval=2084600919 TSecr=3536337894
6102 437.557982664 10.29.29.181 → 10.29.29.248 HTTP 190 GET /metrics HTTP/1.1
6103 437.558001189 10.29.29.248 → 10.29.29.181 TCP 68 4191 → 54512 [ACK] Seq=1 Ack=123 Win=27136 Len=0 TSval=3536337894 TSecr=2084600919
6104 437.560234429 10.29.29.248 → 10.29.29.181 HTTP 7273 HTTP/1.1 200 OK (text/plain)
6105 437.560257453 10.29.29.181 → 10.29.29.248 TCP 68 54512 → 4191 [ACK] Seq=123 Ack=7206 Win=45056 Len=0 TSval=2084600921 TSecr=3536337896
6106 437.560276774 10.29.29.248 → 10.29.29.181 TCP 68 4191 → 54512 [FIN, ACK] Seq=7206 Ack=123 Win=27136 Len=0 TSval=3536337896 TSecr=2084600921
6107 437.560568337 10.29.29.181 → 10.29.29.248 TCP 68 54512 → 4191 [FIN, ACK] Seq=123 Ack=7207 Win=45056 Len=0 TSval=2084600921 TSecr=3536337896
6108 437.560578268 10.29.29.248 → 10.29.29.181 TCP 68 4191 → 54512 [ACK] Seq=7207 Ack=124 Win=27136 Len=0 TSval=3536337896 TSecr=2084600921
6109 437.680065591 10.29.29.248 → 10.29.30.56 TCP 84 33662 → 3000 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=4225733567 TSecr=0 WS=512
6110 439.182381891 10.29.29.248 → 10.29.30.56 TCP 84 33752 → 3000 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=4225735069 TSecr=0 WS=512
6111 439.271898462 10.29.31.129 → 10.29.29.248 HTTP 982 GET /api/tps-reports?resource_type=namespace&namespace=all&tcp_stats=true&window=1m HTTP/1.1
6112 439.272226268 127.0.0.1 → 127.0.0.1 HTTP 920 GET /api/tps-reports?resource_type=namespace&namespace=all&tcp_stats=true&window=1m HTTP/1.1
6113 439.272473631 10.29.29.248 → 127.0.0.1 HTTP 363 POST /api/v1/StatSummary HTTP/1.1
6114 439.272775939 10.29.29.248 → 10.29.95.225 TLSv1.2 203 Application Data
6115 439.296858783 10.29.95.225 → 10.29.29.248 TLSv1.2 140 Application Data
6116 439.296904892 10.29.29.248 → 10.29.95.225 TCP 68 41614 → 8085 [ACK] Seq=2216 Ack=5251 Win=36864 Len=0 TSval=435375939 TSecr=3316798433
6117 439.296864939 10.29.95.225 → 10.29.29.248 TLSv1.2 1276 Application Data
6118 439.296911008 10.29.29.248 → 10.29.95.225 TCP 68 41614 → 8085 [ACK] Seq=2216 Ack=6459 Win=39424 Len=0 TSval=435375939 TSecr=3316798433
6119 439.297125130 172.20.185.231 → 10.29.29.248 HTTP 1373 HTTP/1.1 200 OK
6120 439.297151762 10.29.29.248 → 127.0.0.1 TCP 68 52600 → 4140 [ACK] Seq=2351 Ack=10378 Win=108032 Len=0 TSval=1347277260 TSecr=1275664990
6121 439.298816701 127.0.0.1 → 127.0.0.1 TCP 4164 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6122 439.298835625 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=7858 Ack=70457 Win=464896 Len=0 TSval=1323615080 TSecr=1323615080
6123 439.298848344 127.0.0.1 → 127.0.0.1 TCP 5483 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6124 439.298853344 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=7858 Ack=75872 Win=464896 Len=0 TSval=1323615081 TSecr=1323615081
6125 439.298884908 127.0.0.1 → 127.0.0.1 HTTP 75 HTTP/1.1 200 OK (application/json)
6126 439.298888349 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=7858 Ack=75879 Win=464896 Len=0 TSval=1323615081 TSecr=1323615081
6127 439.298982103 10.29.29.248 → 10.29.31.129 TCP 9581 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6128 439.299053348 10.29.29.248 → 10.29.31.129 HTTP 73 HTTP/1.1 200 OK (application/json)
6129 439.299225949 10.29.31.129 → 10.29.29.248 TCP 68 55770 → 8084 [ACK] Seq=2743 Ack=28551 Win=99328 Len=0 TSval=3406454121 TSecr=1164502472
6130 439.299230570 10.29.31.129 → 10.29.29.248 TCP 68 55770 → 8084 [ACK] Seq=2743 Ack=28556 Win=99328 Len=0 TSval=3406454121 TSecr=1164502472
6131 439.421633243 10.29.29.248 → 172.20.0.10 DNS 140 Standard query 0x12cd A linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6132 439.422332025 172.20.0.10 → 10.29.29.248 DNS 233 Standard query response 0x12cd No such name A linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6133 439.422531745 10.29.29.248 → 172.20.0.10 DNS 140 Standard query 0x2e7b AAAA linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6134 439.423111284 172.20.0.10 → 10.29.29.248 DNS 233 Standard query response 0x2e7b No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6135 439.423214086 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0xef7b A linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local OPT
6136 439.423653344 172.20.0.10 → 10.29.29.248 DNS 225 Standard query response 0xef7b No such name A linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6137 439.423737015 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0x709f AAAA linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local OPT
6138 439.424228832 172.20.0.10 → 10.29.29.248 DNS 225 Standard query response 0x709f No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6139 439.424319257 10.29.29.248 → 172.20.0.10 DNS 128 Standard query 0x9a8b A linkerd-grafana.linkerd.svc.cluster.local.cluster.local OPT
6140 439.424785762 172.20.0.10 → 10.29.29.248 DNS 221 Standard query response 0x9a8b No such name A linkerd-grafana.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6141 439.424883139 10.29.29.248 → 172.20.0.10 DNS 128 Standard query 0x83f9 AAAA linkerd-grafana.linkerd.svc.cluster.local.cluster.local OPT
6142 439.425177325 172.20.0.10 → 10.29.29.248 DNS 221 Standard query response 0x83f9 No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6143 439.425399158 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0xdef2 A linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6144 439.425853380 172.20.0.10 → 10.29.29.248 DNS 246 Standard query response 0xdef2 No such name A linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-89kv0s8od0m.aws.escloud.co.uk OPT
6145 439.425983465 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0x4600 AAAA linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6146 439.426220284 172.20.0.10 → 10.29.29.248 DNS 246 Standard query response 0x4600 No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-ev2vegdtb9d.aws.escloud.co.uk OPT
6147 439.426321471 10.29.29.248 → 172.20.0.10 DNS 141 Standard query 0xec5a A linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6148 439.427587532 172.20.0.10 → 10.29.29.248 DNS 141 Standard query response 0xec5a No such name A linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6149 439.427678248 10.29.29.248 → 172.20.0.10 DNS 141 Standard query 0xf3df AAAA linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6150 439.428532903 172.20.0.10 → 10.29.29.248 DNS 141 Standard query response 0xf3df No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6151 439.428626861 10.29.29.248 → 172.20.0.10 DNS 114 Standard query 0x4a7d A linkerd-grafana.linkerd.svc.cluster.local OPT
6152 439.428921791 172.20.0.10 → 10.29.29.248 DNS 171 Standard query response 0x4a7d A linkerd-grafana.linkerd.svc.cluster.local A 172.20.95.193 OPT
6153 440.000014862 10.29.29.181 → 10.29.29.248 TCP 76 47798 → 9994 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=2084603361 TSecr=0 WS=512
6154 440.000113811 10.29.29.248 → 10.29.29.181 TCP 76 9994 → 47798 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=8961 SACK_PERM=1 TSval=3536340336 TSecr=2084603361 WS=512
6155 440.000151222 10.29.29.181 → 10.29.29.248 TCP 68 47798 → 9994 [ACK] Seq=1 Ack=1 Win=27136 Len=0 TSval=2084603361 TSecr=3536340336
6156 440.000268406 10.29.29.181 → 10.29.29.248 HTTP 187 GET /ping HTTP/1.1
6157 440.000281430 10.29.29.248 → 10.29.29.181 TCP 68 9994 → 47798 [ACK] Seq=1 Ack=120 Win=27136 Len=0 TSval=3536340336 TSecr=2084603361
6158 440.000584984 127.0.0.1 → 127.0.0.1 HTTP 168 GET /ping HTTP/1.1
6159 440.000698635 127.0.0.1 → 127.0.0.1 HTTP 189 HTTP/1.1 200 OK (text/plain)
6160 440.000709152 127.0.0.1 → 127.0.0.1 TCP 68 55744 → 9994 [ACK] Seq=21441 Ack=116692 Win=908 Len=0 TSval=1323615782 TSecr=1323615782
6161 440.000777893 10.29.29.248 → 10.29.29.181 HTTP 189 HTTP/1.1 200 OK (text/plain)
6162 440.000788709 10.29.29.181 → 10.29.29.248 TCP 68 47798 → 9994 [ACK] Seq=120 Ack=122 Win=27136 Len=0 TSval=2084603361 TSecr=3536340336
6163 440.000809096 10.29.29.248 → 10.29.29.181 TCP 68 9994 → 47798 [FIN, ACK] Seq=122 Ack=120 Win=27136 Len=0 TSval=3536340336 TSecr=2084603361
6164 440.000885661 10.29.29.181 → 10.29.29.248 TCP 68 47798 → 9994 [FIN, ACK] Seq=120 Ack=123 Win=27136 Len=0 TSval=2084603362 TSecr=3536340336
6165 440.000894681 10.29.29.248 → 10.29.29.181 TCP 68 9994 → 47798 [ACK] Seq=123 Ack=121 Win=27136 Len=0 TSval=3536340337 TSecr=2084603362
6166 440.057467569 10.29.95.234 → 10.29.29.248 HTTP 1157 GET /grafana/dashboard/db/linkerd-namespace?var-namespace=&var-namespace=linkerd HTTP/1.1
6167 440.057793106 127.0.0.1 → 127.0.0.1 HTTP 1095 GET /grafana/dashboard/db/linkerd-namespace?var-namespace=&var-namespace=linkerd HTTP/1.1
6168 440.058007191 10.29.29.248 → 127.0.0.1 HTTP 1118 GET /dashboard/db/linkerd-namespace?var-namespace=&var-namespace=linkerd HTTP/1.1
6169 440.058020242 172.20.95.193 → 10.29.29.248 TCP 68 3000 → 58600 [ACK] Seq=93 Ack=2101 Win=48128 Len=0 TSval=1275665751 TSecr=1447490960
6170 440.098128119 10.29.29.248 → 10.29.95.234 TCP 68 8084 → 59252 [ACK] Seq=18593 Ack=2932 Win=32768 Len=0 TSval=3210983603 TSecr=58307892
6171 440.098122223 127.0.0.1 → 127.0.0.1 TCP 68 8084 → 60694 [ACK] Seq=75879 Ack=8885 Win=62976 Len=0 TSval=1323615880 TSecr=1323615839
6172 440.684320811 10.29.29.248 → 10.29.30.56 TCP 84 33862 → 3000 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=4225736571 TSecr=0 WS=512
6173 441.364438864 10.29.29.181 → 10.29.29.248 TCP 76 47898 → 9994 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=2084604725 TSecr=0 WS=512
6174 441.364529302 10.29.29.248 → 10.29.29.181 TCP 76 9994 → 47898 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=8961 SACK_PERM=1 TSval=3536341700 TSecr=2084604725 WS=512
6175 441.364562139 10.29.29.181 → 10.29.29.248 TCP 68 47898 → 9994 [ACK] Seq=1 Ack=1 Win=27136 Len=0 TSval=2084604725 TSecr=3536341700
6176 441.364680064 10.29.29.181 → 10.29.29.248 HTTP 188 GET /ready HTTP/1.1
6177 441.364692176 10.29.29.248 → 10.29.29.181 TCP 68 9994 → 47898 [ACK] Seq=1 Ack=121 Win=27136 Len=0 TSval=3536341700 TSecr=2084604725
6178 441.364975841 127.0.0.1 → 127.0.0.1 HTTP 169 GET /ready HTTP/1.1
6179 441.365106869 127.0.0.1 → 127.0.0.1 HTTP 187 HTTP/1.1 200 OK (text/plain)
6180 441.365118803 127.0.0.1 → 127.0.0.1 TCP 68 55744 → 9994 [ACK] Seq=21542 Ack=116811 Win=908 Len=0 TSval=1323617147 TSecr=1323617147
6181 441.365192733 10.29.29.248 → 10.29.29.181 HTTP 187 HTTP/1.1 200 OK (text/plain)
6182 441.365203649 10.29.29.181 → 10.29.29.248 TCP 68 47898 → 9994 [ACK] Seq=121 Ack=120 Win=27136 Len=0 TSval=2084604726 TSecr=3536341701
6183 441.365221862 10.29.29.248 → 10.29.29.181 TCP 68 9994 → 47898 [FIN, ACK] Seq=120 Ack=121 Win=27136 Len=0 TSval=3536341701 TSecr=2084604726
6184 441.365278748 10.29.29.181 → 10.29.29.248 TCP 68 47898 → 9994 [FIN, ACK] Seq=121 Ack=121 Win=27136 Len=0 TSval=2084604726 TSecr=3536341701
6185 441.365288147 10.29.29.248 → 10.29.29.181 TCP 68 9994 → 47898 [ACK] Seq=121 Ack=122 Win=27136 Len=0 TSval=3536341701 TSecr=2084604726
6186 441.390886834 10.29.30.189 → 10.29.29.248 HTTP 352 GET /metrics HTTP/1.1
6187 441.391187746 127.0.0.1 → 127.0.0.1 HTTP 352 GET /metrics HTTP/1.1
6188 441.392497263 127.0.0.1 → 127.0.0.1 HTTP 2574 HTTP/1.1 200 OK (text/plain)
6189 441.392618788 10.29.29.248 → 10.29.30.189 TCP 2287 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6190 441.392679085 10.29.29.248 → 10.29.30.189 TCP 350 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
6191 441.392731416 10.29.29.248 → 10.29.30.189 HTTP 73 HTTP/1.1 200 OK (text/plain)
6192 441.392831912 10.29.30.189 → 10.29.29.248 TCP 68 39706 → 9994 [ACK] Seq=12781 Ack=108230 Win=211 Len=0 TSval=3998110381 TSecr=3562804392
6193 441.392836598 10.29.30.189 → 10.29.29.248 TCP 68 39706 → 9994 [ACK] Seq=12781 Ack=108512 Win=211 Len=0 TSval=3998110381 TSecr=3562804392
6194 441.392854789 10.29.30.189 → 10.29.29.248 TCP 68 39706 → 9994 [ACK] Seq=12781 Ack=108517 Win=211 Len=0 TSval=3998110381 TSecr=3562804392
6195 441.434133443 127.0.0.1 → 127.0.0.1 TCP 68 55744 → 9994 [ACK] Seq=21826 Ack=119317 Win=908 Len=0 TSval=1323617216 TSecr=1323617174
6196 441.957799911 10.29.29.248 → 172.20.0.10 DNS 147 Standard query 0x1abf A linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6197 441.958233603 172.20.0.10 → 10.29.29.248 DNS 240 Standard query response 0x1abf No such name A linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6198 441.958424304 10.29.29.248 → 172.20.0.10 DNS 147 Standard query 0x445b AAAA linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6199 441.958765361 172.20.0.10 → 10.29.29.248 DNS 240 Standard query response 0x445b No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6200 441.958920485 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0xb208 A linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local OPT
6201 441.959568570 172.20.0.10 → 10.29.29.248 DNS 232 Standard query response 0xb208 No such name A linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6202 441.959688821 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0x7908 AAAA linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local OPT
6203 441.960278195 172.20.0.10 → 10.29.29.248 DNS 232 Standard query response 0x7908 No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6204 441.960433785 10.29.29.248 → 172.20.0.10 DNS 135 Standard query 0xdc81 A linkerd-controller-api.linkerd.svc.cluster.local.cluster.local OPT
6205 441.960709021 172.20.0.10 → 10.29.29.248 DNS 228 Standard query response 0xdc81 No such name A linkerd-controller-api.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6206 441.960820880 10.29.29.248 → 172.20.0.10 DNS 135 Standard query 0x4b45 AAAA linkerd-controller-api.linkerd.svc.cluster.local.cluster.local OPT
6207 441.961333112 172.20.0.10 → 10.29.29.248 DNS 228 Standard query response 0x4b45 No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6208 441.961431096 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0x758f A linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6209 441.961862527 172.20.0.10 → 10.29.29.248 DNS 253 Standard query response 0x758f No such name A linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-ev2vegdtb9d.aws.escloud.co.uk OPT
6210 441.961953620 10.29.29.248 → 172.20.0.10 DNS 139 Standard query 0x386e AAAA linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6211 441.962189395 172.20.0.10 → 10.29.29.248 DNS 253 Standard query response 0x386e No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-89kv0s8od0m.aws.escloud.co.uk OPT
6212 441.962330071 10.29.29.248 → 172.20.0.10 DNS 148 Standard query 0xf97b A linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6213 441.963274952 172.20.0.10 → 10.29.29.248 DNS 148 Standard query response 0xf97b No such name A linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6214 441.963380665 10.29.29.248 → 172.20.0.10 DNS 148 Standard query 0x372f AAAA linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6215 441.964250396 172.20.0.10 → 10.29.29.248 DNS 148 Standard query response 0x372f No such name AAAA linkerd-controller-api.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6216 441.964341697 10.29.29.248 → 172.20.0.10 DNS 121 Standard query 0xc273 A linkerd-controller-api.linkerd.svc.cluster.local OPT
6217 441.964601547 172.20.0.10 → 10.29.29.248 DNS 185 Standard query response 0xc273 A linkerd-controller-api.linkerd.svc.cluster.local A 172.20.185.231 OPT
6218 442.186179409 10.29.29.248 → 10.29.30.56 TCP 84 33954 → 3000 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=4225738073 TSecr=0 WS=512
6219 443.050135141 10.29.29.248 → 172.20.220.44 TCP 68 [TCP Keep-Alive] 46412 → 8080 [ACK] Seq=1 Ack=1 Win=62 Len=0 TSval=3573722458 TSecr=2238113733
6220 443.050565222 172.20.220.44 → 10.29.29.248 TCP 68 [TCP Keep-Alive ACK] 8080 → 46412 [ACK] Seq=1 Ack=2 Win=62 Len=0 TSval=2238189510 TSecr=3573648281
6221 443.059246593 172.20.95.193 → 10.29.29.248 HTTP 160 HTTP/1.1 503 Service Unavailable
6222 443.059266294 10.29.29.248 → 127.0.0.1 TCP 68 58600 → 4140 [ACK] Seq=2101 Ack=185 Win=27136 Len=0 TSval=1447493961 TSecr=1275668752
6223 443.059426702 127.0.0.1 → 127.0.0.1 HTTP 255 HTTP/1.1 503 Service Unavailable
6224 443.059438565 127.0.0.1 → 127.0.0.1 TCP 68 60694 → 8084 [ACK] Seq=8885 Ack=76066 Win=464896 Len=0 TSval=1323618841 TSecr=1323618841
6225 443.059532632 10.29.29.248 → 10.29.95.234 HTTP 255 HTTP/1.1 503 Service Unavailable
6226 443.059969863 10.29.95.234 → 10.29.29.248 TCP 68 59252 → 8084 [ACK] Seq=2932 Ack=18780 Win=82432 Len=0 TSval=58310895 TSecr=3210986564
6227 443.306127673 10.29.29.248 → 10.29.157.60 TCP 68 [TCP Keep-Alive] 42426 → 8085 [ACK] Seq=928 Ack=2803 Win=33280 Len=0 TSval=2554157045 TSecr=4171786481
6228 443.306644005 10.29.157.60 → 10.29.29.248 TCP 68 [TCP Keep-Alive ACK] 8085 → 42426 [ACK] Seq=2803 Ack=929 Win=30208 Len=0 TSval=4171796504 TSecr=2554147022
6229 443.348313737 10.29.157.60 → 10.29.29.248 TCP 68 [TCP Keep-Alive] 8085 → 42426 [ACK] Seq=2802 Ack=929 Win=30208 Len=0 TSval=4171796545 TSecr=2554147022
6230 443.348327966 10.29.29.248 → 10.29.157.60 TCP 68 [TCP Keep-Alive ACK] 42426 → 8085 [ACK] Seq=929 Ack=2803 Win=33280 Len=0 TSval=2554157087 TSecr=4171796504
6231 443.689631264 10.29.29.248 → 10.29.30.56 TCP 84 34040 → 3000 [SYN] Seq=0 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=4225739576 TSecr=0 WS=512
6232 443.875635297 10.29.30.189 → 10.29.29.248 HTTP 352 GET /metrics HTTP/1.1
6233 443.877864230 10.29.29.248 → 10.29.30.189 HTTP 7280 HTTP/1.1 200 OK (text/plain)
6234 443.878171989 10.29.30.189 → 10.29.29.248 TCP 68 55908 → 4191 [ACK] Seq=12781 Ack=275695 Win=211 Len=0 TSval=3998112866 TSecr=3562806877
6235 444.431061596 10.29.29.248 → 172.20.0.10 DNS 140 Standard query 0xf28f A linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6236 444.431783149 172.20.0.10 → 10.29.29.248 DNS 233 Standard query response 0xf28f No such name A linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6237 444.431979095 10.29.29.248 → 172.20.0.10 DNS 140 Standard query 0xe9e1 AAAA linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local OPT
6238 444.432409448 172.20.0.10 → 10.29.29.248 DNS 233 Standard query response 0xe9e1 No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.linkerd.svc.cluster.local SOA ns.dns.cluster.local OPT
6239 444.432535350 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0xacd5 A linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local OPT
6240 444.433064122 172.20.0.10 → 10.29.29.248 DNS 225 Standard query response 0xacd5 No such name A linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6241 444.433158338 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0x18b1 AAAA linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local OPT
6242 444.433615749 172.20.0.10 → 10.29.29.248 DNS 225 Standard query response 0x18b1 No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.svc.cluster.local SOA ns.dns.cluster.local OPT
6243 444.433707651 10.29.29.248 → 172.20.0.10 DNS 128 Standard query 0xd613 A linkerd-grafana.linkerd.svc.cluster.local.cluster.local OPT
6244 444.434029936 172.20.0.10 → 10.29.29.248 DNS 221 Standard query response 0xd613 No such name A linkerd-grafana.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6245 444.434256930 10.29.29.248 → 172.20.0.10 DNS 128 Standard query 0xe907 AAAA linkerd-grafana.linkerd.svc.cluster.local.cluster.local OPT
6246 444.434515499 172.20.0.10 → 10.29.29.248 DNS 221 Standard query response 0xe907 No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.cluster.local SOA ns.dns.cluster.local OPT
6247 444.434640185 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0xa2ec A linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6248 444.435082804 172.20.0.10 → 10.29.29.248 DNS 246 Standard query response 0xa2ec No such name A linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-89kv0s8od0m.aws.escloud.co.uk OPT
6249 444.435190781 10.29.29.248 → 172.20.0.10 DNS 132 Standard query 0x3ee6 AAAA linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk OPT
6250 444.435595154 172.20.0.10 → 10.29.29.248 DNS 246 Standard query response 0x3ee6 No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.aws.escloud.co.uk SOA win-ev2vegdtb9d.aws.escloud.co.uk OPT
6251 444.435684983 10.29.29.248 → 172.20.0.10 DNS 141 Standard query 0x019c A linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6252 444.436609713 172.20.0.10 → 10.29.29.248 DNS 141 Standard query response 0x019c No such name A linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6253 444.436690038 10.29.29.248 → 172.20.0.10 DNS 141 Standard query 0x200e AAAA linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6254 444.437893278 172.20.0.10 → 10.29.29.248 DNS 141 Standard query response 0x200e No such name AAAA linkerd-grafana.linkerd.svc.cluster.local.eu-west-1.compute.internal OPT
6255 444.437997488 10.29.29.248 → 172.20.0.10 DNS 114 Standard query 0x14de A linkerd-grafana.linkerd.svc.cluster.local OPT
6256 444.438295497 172.20.0.10 → 10.29.29.248 DNS 171 Standard query response 0x14de A linkerd-grafana.linkerd.svc.cluster.local A 172.20.95.193 OPT
6257 444.648646551 172.20.220.44 → 10.29.29.248 TCP 68 [TCP Keep-Alive] 8080 → 46412 [ACK] Seq=0 Ack=2 Win=62 Len=0 TSval=2238191108 TSecr=3573648281

No firewalls installed nor any network policy in that namespace:

kubectl get networkpolicy -n linkerd
No resources found in linkerd namespace.

@jonbbbb thanks for the info, it looks like there is a mix of internal EC2 addresses and pod IPs

6221 443.059246593 172.20.95.193 → 10.29.29.248 HTTP 160 HTTP/1.1 503 Service Unavailable

Can you remind me how you deployed the clusters? Are they EKS clusters or did you use a tool like kubeadm?

In the end I destroyed the cluster and rebuilt it, all working now (the same scripts as used the first time to bring it up)

That’s another option :slight_smile:

Glad to hear that it’s all working now. Sounds like it was definitely an issue in the cluster configuration.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.