Linkerd2-proxy unmeshed

#1

I want to run linkerd2-proxy on bare-metal server.
something like this:

bare-metal server                                          kubernetes node 
_____________________________                      ____________________________
|                           |                      |                           |
|   nginx -> linkerd2-proxy |    ---->             |  linkerd2-proxy -> app    |
|___________________________|                      |___________________________|

all control-palne components in kubernetes cluster as usual

List what I do

  1. create entrypoint to identity and destination pods by calico floatingIP
  2. create ServiceAccount for bare-metal server (front-servers) in kubernetes
  3. grab token from this SA and place it on front server
  4. configure all LINKERD2_* variables in .env file on front server (IDENTITY_SVC_ADDR, DESTINATION_SVC_ADDR, IDENTITY_TRUST_ANCHORS, IDENTITY_TOKEN_FILE etc)
  5. start linkerd2-proxy by linkerd2-proxy-run wrapper what generate certs from identity by linkerd2-proxy-identity
  6. configure iptables on front server for access from tap and prometheus to ports 4190 and 4191
  7. configure prometheus to scrap metrics from linkerd2-proxy on front servers
  8. configure header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port in nginx

And proxying works as design. mtls, dst service all good.

response_latency_ms_bucket{
authority="nginx-my-api.vkp.svc.cluster.local",
direction="outbound",
dst_control_plane_ns="pfm-admins",
dst_deployment="nginx-my-api",
dst_namespace="vkp",
dst_pod="nginx-my-api-7d4dcfb9b8-ftr5x",
dst_pod_template_hash="7d4dcfb9b8",
dst_service="nginx-my-api",
dst_serviceaccount="nginx-my-api",
tls="true",
server_id="nginx-my-api.vkp.serviceaccount.identity.pfm-admins.cluster.local",
status_code="200",
le="3000"
}

But in linkerd-web i see this proxy as unmeshed
Screenshot_2020-09-04 Linkerd

in prometheus I set all labels as on linkerd-proxy job

Screenshot_2020-09-04 Prometheus Time Series Collection and Processing Server
Screenshot_2020-09-04 Prometheus Time Series Collection and Processing Server1470×528 69.8 KB

what else needs to be done to make linkerd2-proxy on front server meshed?

In kubernetes we don’t use xvlan or ipip tunels, all pods ip is announced by bird in 10/8 network so front servers have direct access to pods ip

#2

In dashbord I can see my front servers as a pod

Снимок экрана от 2020-09-04 14-55-27
Снимок экрана от 2020-09-04 14-55-271922×1057 166 KB

#3

found in linkerd2-proxy on front server:
outbound: linkerd2_proxy_transport::listen::sys::linux: failed to read SO_ORIGINAL_DST: Os { code: 92, kind: Other, message: "Protocol not available"

#4

identity works:

Screenshot_2020-09-04 Linkerd(1)
Screenshot_2020-09-04 Linkerd(1)1569×303 17.7 KB

But linkerd2-proxy on front servrer still unmeshed

#5

I create virtual-kubelet node with my custom provider, when create StatefulSet for each bare metal node with node ip and now all works!

# kubectl get pods -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP           NODE              NOMINATED NODE   READINESS GATES
vkp-front1-0   2/2     Running   0          3m42s   10.11.57.0   virtual-kubelet   <none>           <none>
vkp-front2-0   2/2     Running   0          2m26s   10.11.57.1   virtual-kubelet   <none>           <none>
vkp-front3-0   2/2     Running   0          110s    10.11.57.2   virtual-kubelet   <none>           <none>
vkp-front4-0   2/2     Running   0          12s     10.11.57.3   virtual-kubelet   <none>           <none>

Снимок экрана от 2020-09-07 18-01-53
Снимок экрана от 2020-09-07 18-01-531922×1057 159 KB

#6

@sbvitok this is a really interesting application of the Linkerd proxy. Since the proxy is designed to run in the same cluster as the control plane, it’s not too surprising that you’ve had to set up infrastructure like this to get it to work.

Pretty amazing hacking there :slight_smile:! Thanks for sharing your process.

Can you share the wrapper script that you created and any additional virtual kubelet and custom provider configurations?

#7 
# cat /usr/bin/linkerd2-proxy-run 
#!/usr/bin/env bash
set -eu
set -a
cd /etc/linkerd2-proxy
source config.env
set +a
set +eu
rm -f $LINKERD2_PROXY_IDENTITY_DIR/* 2>/dev/null
set -eu
/usr/lib/linkerd/linkerd2-proxy-identity -dir "$LINKERD2_PROXY_IDENTITY_DIR" -name "$LINKERD2_PROXY_IDENTITY_LOCAL_NAME"

exec /usr/lib/linkerd/linkerd2-proxy
#8

Thanks @sbvitok ! I hope someone will find this useful if they have the same use-case as you.