Tls not provided by_service_discovery while setting up MC service mesh

I am setting up a MultiCluster service mesh following the steps outlined here

This is what my basic setup looks like. My aim is to link west to east. east is a GKE cluster so the gateway will be a GKE LB.

On both clusters ive installed linkerd using

linkerd install \
  --identity-trust-anchors-file root.crt \
  --identity-issuer-certificate-file issuer.crt \
  --identity-issuer-key-file issuer.key \
  | tee \
    >(kubectl --context=west apply -f -) \
    >(kubectl --context=east apply -f -)

Im using a common trust anchor for both cluster. I then went ahead and installed viz and mc components.

In east, the GW service is a GKE based LB with the IP 34.91.170.56 which is public accepting connections on 4143 and 4191.

I went ahead and linked west to east. the gw status comes up fine

linkerd mc gateways
CLUSTER  ALIVE    NUM_SVC  LATENCY_P50  LATENCY_P95  LATENCY_P99
east     True           0         25ms         30ms         30ms

I went ahead and deployed an nginx service and exported in and immediately i saw nginx-east svc created in west.

I spun up a curl client and injected it using the manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: curl
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: curl
    spec:
      containers:
      - image: curlimages/curl
        name: curl
        command: [ "sh", "-c", "--" ]
        args: [ "while true; do sleep 30; done;" ]

When I curl on the mirrored service, I get

k exec -n default -it curl-55f694956d-kbdjn -c curl -- curl -v http://gke-nginx-east.default.svc.cluster.local:8080
*   Trying 100.64.192.230:8080...
* Connected to gke-nginx-east.default.svc.cluster.local (100.64.192.230) port 8080 (#0)
> GET / HTTP/1.1
> Host: gke-nginx-east.default.svc.cluster.local:8080
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< l5d-proxy-error: client: Connection reset by peer (os error 104)
< connection: close
< content-length: 0
< date: Thu, 03 Mar 2022 18:26:51 GMT
<

The logs of the client’s linkerd-proxy state

[  4839.295328s]  INFO ThreadId(01) outbound:server{orig_dst=34.91.170.56:4143}:rescue{client.addr=100.96.4.176:45992}: linkerd_app_core::errors::respond: Request failed error=connection error: client: Connection reset by peer (os error 104)
[  4839.295389s]  INFO ThreadId(01) outbound:server{orig_dst=34.91.170.56:4143}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set

Doing a viz tap gives

req id=0:0 proxy=out src=100.96.4.176:47096 dst=34.91.170.56:4143 tls=not_provided_by_service_discovery :method=GET :authority=gke-nginx-east.default.svc.cluster.local:8080 :path=/
rsp id=0:0 proxy=out src=100.96.4.176:47096 dst=34.91.170.56:4143 tls=not_provided_by_service_discovery :status=502 latency=22756µs
end id=0:0 proxy=out src=100.96.4.176:47096 dst=34.91.170.56:4143 tls=not_provided_by_service_discovery duration=12µs response-length=0B

And the log on the GW states

[  6058.122277s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated

Although the client is meshed, why do I still see all errors pointing to it not being meshed ?