Kubernetes - Help getting started

froch · September 15, 2017, 8:48pm

Hello! Thank you for setting up a Discourse forum.

We have an HA setup of Kubernetes, with an LB running k8s-proxy (available at http://llb-sandbox, for example).

Each of our services is assigned a specific port. For example, the admintool k8s-replicationcontroller exposes port 8086, which is fronted by a k8s-service that also exposes port 8086. Our services currently address each other by sending requests through llb-sandbox, eg. to reach the admintool’s health check endpoint, we issue an HTTP GET to http://llb-sandbox:8086/admin/health.

After deploying linkerd-1.2.0 using https://github.com/linkerd/linkerd-examples/blob/master/k8s-daemonset/k8s/servicemesh.yml, we can use the delegator to validate that linkerd knows where to find our admintool-service.

Where we get stuck is in trying to query the admintool’s health check endpoint through linkerd. From within a k8s-node, for example, if we issue an HTTP GET to 0.0.0.0:4140/svc/admintool-service/admin/health, we don’t reach it. Most likely because we need a dtab rule to further handle port 8086 and the extra path in some way?

A simple nudge in the right direction would be much appreciated. Thanks for your time!

siggy · September 15, 2017, 9:00pm

Hi @froch, thanks for all the detail, it’s helpful.

Your dtab screenshot indicates .../#/io.l5d.k8s.http/default/http/admintool-service. In this case default is the Kubernetes namespace, http is the port name, and admintool-service is the service name. Can you confirm that port 8086 is named http? If it’s not, rename it to http, or modify your linkerd config to match the name of port 8086.

froch · September 15, 2017, 10:21pm

Thanks for your quick reply!

I had misunderstood, I thought the “http” identified the protocol, not the
service port name. In our kubernetes configs we haven’t named the ports,
just the Services.

I’ll try that first thing Monday morning. Thank you! Have a good weekend

froch · September 18, 2017, 5:32pm

Hello again!

I followed your suggestion this morning, and named the port for the Service to http. Additionally, I named the corresponding port on the associated ReplicationController to http as well, in case it had any bearing. Yielding this output of kubectl get -o yaml:

---
apiVersion: v1
kind: Service
metadata:
  ...
spec:
  clusterIP: 10.250.1.25
  ports:
  - name: http
    nodePort: 8086
    port: 8086
    protocol: TCP
    targetPort: 8086
  selector:
    name: admintool-service
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}
---
apiVersion: v1
kind: ReplicationController
metadata:
  ...
spec:
  replicas: 1
  selector:
    name: admintool-service
  template:
    metadata:
      ...
    spec:
      containers:
        image: repo/admintool-service:latest
        imagePullPolicy: IfNotPresent
        name: admintool-service
        ports:
        - containerPort: 8086
          name: http
          protocol: TCP

Using the delegator through the linkerd admin interface to query /svc/admintool-service yields the same result as the pasted image in my original post (the green bars, resolving to the right linkerd instance on NODE_IP:4141).

I can also still query the health check endpoint of my service through our load balancer (using Python’s httpie here to capture additional header information):

root@dcs001-ds1:~# http llb-sandbox:8086/admin/health
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain; charset=UTF-8
Date: Mon, 18 Sep 2017 15:49:12 GMT
Server: spray-can/1.3.3

ok

Now, from a host running a linkerd daemonset-managed pod, I run this query, and obtain this response:

$ http localhost:4140/svc/admintool-service
HTTP/1.1 431 Request Header Fields Too Large
Content-Length: 0
Via: 1.0 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd, 1.1 linkerd
l5d-success-class: 1.0

That last part seems quite odd to me, the HTTP 431 status (with no additional headers specified), with what appears to be a loop in the Via header, through many instances of linkerd.

Furthermore, if I look at the Recent Requests tab through the linkerd-admin:9990, i don’t see any requests listed there. Is the localhost:4140/svc/admintool-service the correct syntax for the request?

EDIT: I’ve performed a single change to the stock servicemesh.yaml config that is found on the linkerd-examples github repo, which is to enable host-level networking since we run flanneld for our overlay network. Come to think of it, this might be causing these kinds of problems; while running linkerd, do we even need overlay networks like flannel, or calico or weave at all?

Any insight as to what I’m doing wrong here is much appreciated! Thank you again for your time.

Alex · September 18, 2017, 6:35pm

Hi @froch!

I believe the problem is in the way that you’re issuing requests. By default, Linkerd routes based on the Host header. For example, a request with “Host: admintool-service” would route with the name /svc/admintool-service. So when you issue a request like

$ http localhost:4140/svc/admintool-service

The host header is set to localhost:4140 which tells Linkerd to route to Linkerd (thus the infinite loop). Instead, I think what you want is to issue requests something like

$ http localhost:4140/admin/health Host:admintool-service

Hope this clears it up.

froch · September 18, 2017, 7:01pm

Yes! That was it. I fetched the endpoint through linkerd with that last example request. Thank you for pointing me in the right direction! Cheers