Linkerd2-proxy unmeshed

#1

I want to run linkerd2-proxy on bare-metal server.
something like this:

bare-metal server                                          kubernetes node 
_____________________________                      ____________________________
|                           |                      |                           |
|   nginx -> linkerd2-proxy |    ---->             |  linkerd2-proxy -> app    |
|___________________________|                      |___________________________|

all control-palne components in kubernetes cluster as usual

List what I do

  1. create entrypoint to identity and destination pods by calico floatingIP
  2. create ServiceAccount for bare-metal server (front-servers) in kubernetes
  3. grab token from this SA and place it on front server
  4. configure all LINKERD2_* variables in .env file on front server (IDENTITY_SVC_ADDR, DESTINATION_SVC_ADDR, IDENTITY_TRUST_ANCHORS, IDENTITY_TOKEN_FILE etc)
  5. start linkerd2-proxy by linkerd2-proxy-run wrapper what generate certs from identity by linkerd2-proxy-identity
  6. configure iptables on front server for access from tap and prometheus to ports 4190 and 4191
  7. configure prometheus to scrap metrics from linkerd2-proxy on front servers
  8. configure header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port in nginx

And proxying works as design. mtls, dst service all good.

response_latency_ms_bucket{
authority="nginx-my-api.vkp.svc.cluster.local",
direction="outbound",
dst_control_plane_ns="pfm-admins",
dst_deployment="nginx-my-api",
dst_namespace="vkp",
dst_pod="nginx-my-api-7d4dcfb9b8-ftr5x",
dst_pod_template_hash="7d4dcfb9b8",
dst_service="nginx-my-api",
dst_serviceaccount="nginx-my-api",
tls="true",
server_id="nginx-my-api.vkp.serviceaccount.identity.pfm-admins.cluster.local",
status_code="200",
le="3000"
}

But in linkerd-web i see this proxy as unmeshed
Screenshot_2020-09-04 Linkerd

in prometheus I set all labels as on linkerd-proxy job

Screenshot_2020-09-04 Prometheus Time Series Collection and Processing Server
Screenshot_2020-09-04 Prometheus Time Series Collection and Processing Server1470×528 69.8 KB

what else needs to be done to make linkerd2-proxy on front server meshed?

In kubernetes we don’t use xvlan or ipip tunels, all pods ip is announced by bird in 10/8 network so front servers have direct access to pods ip

#2

In dashbord I can see my front servers as a pod

Снимок экрана от 2020-09-04 14-55-27
Снимок экрана от 2020-09-04 14-55-271922×1057 166 KB

#3

found in linkerd2-proxy on front server:
outbound: linkerd2_proxy_transport::listen::sys::linux: failed to read SO_ORIGINAL_DST: Os { code: 92, kind: Other, message: "Protocol not available"

#4

identity works:

Screenshot_2020-09-04 Linkerd(1)
Screenshot_2020-09-04 Linkerd(1)1569×303 17.7 KB

But linkerd2-proxy on front servrer still unmeshed

#5

I create virtual-kubelet node with my custom provider, when create StatefulSet for each bare metal node with node ip and now all works!

# kubectl get pods -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP           NODE              NOMINATED NODE   READINESS GATES
vkp-front1-0   2/2     Running   0          3m42s   10.11.57.0   virtual-kubelet   <none>           <none>
vkp-front2-0   2/2     Running   0          2m26s   10.11.57.1   virtual-kubelet   <none>           <none>
vkp-front3-0   2/2     Running   0          110s    10.11.57.2   virtual-kubelet   <none>           <none>
vkp-front4-0   2/2     Running   0          12s     10.11.57.3   virtual-kubelet   <none>           <none>

Снимок экрана от 2020-09-07 18-01-53
Снимок экрана от 2020-09-07 18-01-531922×1057 159 KB

#6

@sbvitok this is a really interesting application of the Linkerd proxy. Since the proxy is designed to run in the same cluster as the control plane, it’s not too surprising that you’ve had to set up infrastructure like this to get it to work.

Pretty amazing hacking there :slight_smile:! Thanks for sharing your process.

Can you share the wrapper script that you created and any additional virtual kubelet and custom provider configurations?

#7 
# cat /usr/bin/linkerd2-proxy-run 
#!/usr/bin/env bash
set -eu
set -a
cd /etc/linkerd2-proxy
source config.env
set +a
set +eu
rm -f $LINKERD2_PROXY_IDENTITY_DIR/* 2>/dev/null
set -eu
/usr/lib/linkerd/linkerd2-proxy-identity -dir "$LINKERD2_PROXY_IDENTITY_DIR" -name "$LINKERD2_PROXY_IDENTITY_LOCAL_NAME"

exec /usr/lib/linkerd/linkerd2-proxy
#8

Thanks @sbvitok ! I hope someone will find this useful if they have the same use-case as you.

#9

@sbvitok I am trying to accomplish similar. I am trying to run linkerd2-proxy on my machine but unable to get it to talk to linkerd-identity service running inside k8s cluster.

Can you clarify what you mean by running 1) running linkerd2-proxy-run wrapper? Where is it located? 2) How do you setup a Service Account for bare metal.

Thanks & appreciate help here.

#10

@hakeemsyd, this looks like the linkerd-run-wrapper script.

My guess is that there is more info in the config.env file that enables this to run by setting the environment variables.

I think that the identity part that might be missing is the linkerd-proxy-identity executable here