K8s + CNI + Internal TLS + Egress

I’m trying to work with the Kubernetes examples to setup a daemonset that will route TLS between Nodes (a corporate requirement) but route everything else out to the internet.

I’ve got a setup that works with CNI (Calico) and TLS between nodes and I’ve begun to modify the linkerd-egress.yml to work with CNI. Sadly its not returning any data for external connections (though the dtab indicates the correct IP endpoints). The in cluster routes ARE working but I haven’t added TLS back in yet.

Any help would be appreciated…

Here’s my attempt at CNI + Egress

# runs linkerd in a daemonset, in linker-to-linker mode
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: l5d-config
data:
  config.yaml: |-
    admin:
      port: 9990

    namers:
    - kind: io.l5d.k8s # This namer has the daemonset transformer "built-in"
      prefix: /io.l5d.k8s.ds # We reference this in the outgoing router's dtab
      experimental: true
      host: localhost
      port: 8001
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: default
        port: incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.k8s # The "basic" k8s namer.  We reference this in the incoming router's dtab
      experimental: true
      host: localhost
      port: 8001

    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25

    usage:
      orgId: linkerd-examples-daemonset-egress

    routers:
    - protocol: http
      label: outgoing
      dtab: |
        /ph        => /$/io.buoyant.rinet ; # Lookup the name in DNS
        /srv       => /ph/80 ; # Use port 80 if unspecified
        /srv       => /$/io.buoyant.porthostPfx/ph ; # Attempt to extract the port from the hostname
        /srv       => /#/io.l5d.k8s.ds/default/http ; # Lookup the name in Kubernetes, use the linkerd daemonset pod
        /svc       => /srv ;
        /svc/world => /srv/world-v1 ;
      servers:
      - port: 4140
        ip: 0.0.0.0
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX
      client:
        kind: io.l5d.static
        configs:
        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            commonName: "{service}"

    - protocol: http
      label: incoming
      dtab: |
        /srv        => /#/io.l5d.k8s/default/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      servers:
      - port: 4141
        ip: 0.0.0.0
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: l5d
  name: l5d
spec:
  template:
    metadata:
      labels:
        app: l5d
    spec:
      volumes:
      - name: l5d-config
        configMap:
          name: "l5d-config"
      containers:
      - name: l5d
        image: buoyantio/linkerd:1.1.2
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - /io.buoyant/linkerd/config/config.yaml
        ports:
        - name: outgoing
          containerPort: 4140
          hostPort: 4140
        - name: incoming
          containerPort: 4141
        - name: admin
          containerPort: 9990
        volumeMounts:
        - name: "l5d-config"
          mountPath: "/io.buoyant/linkerd/config"
          readOnly: true

      - name: kubectl
        image: buoyantio/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
# ---
# apiVersion: v1
# kind: Service
# metadata:
#   name: l5d
# spec:
#   selector:
#     app: l5d
#   type: LoadBalancer
#   ports:
#   - name: outgoing
#     port: 4140
#   - name: incoming
#     port: 4141
#   - name: admin
#     port: 9990

At first glance, I don’t see anything in that config that seems wrong. Can you be more specific about what happens when you try to send a request to an external address? Does it hang? Return an error?

@leopoldodonnell this is a common use case, so we should be able to help you get it working!

Sigh - it just hangs - dtab shows what I think is a the correctly bound destination.
My cluster instances are in a private subnet but I can route out.

If I check the dtab binding for /svc/hello I get the following (I think) correct bound path:
192.168.213.110:4141 192.168.213.28:4141 [/%/io.l5d.k8s.daemonset/default/incoming/l5d/#/io.l5d.k8s.ds/default/http/hello] (residual: /)

If I check the dtab binding for /svc/imgur.com (hard to find a site that just accepts http nowadays), it get:
151.101.56.193:80 [/$/io.buoyant.rinet/80/imgur.com] (residual: /)

With the setup bellow I get the following output:

http_proxy=$INGRESS_LB:4140 curl -i http://hello

<h3>Application Blocked!</h3>
<div class="notice">You have attempted to use an application which is in violation of your internet usage policy.</div>
<div class="app-title">Proxy.HTTP</div>
<div class="app-info">Category: Proxy</div>
<div class="app-info">URL: http://world/</div>
<div class="app-info">Client IP: 10.126.29.26</div>
<div class="app-info">Server IP: 52.44.228.134</div>
<div class="app-info">User name: </div>
<div class="app-info">Group name: </div>
<div class="app-info">Policy: 3f58a742-271b-51e5-6d85-df65a9de75c5</div>
<div class="app-info">FortiGate Hostname: us-bomas-fwb01</div>    </div>
</body>
</html>

Trying an outside endpoint…
http_proxy=$INGRESS_LB:4140 curl -i http://imgur.com

HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Cache-Control: max-age=60, stale-while-revalidate=600, stale-if-error=86400, public
Accept-Ranges: bytes
Date: Fri, 21 Jul 2017 15:17:07 GMT
Age: 127
X-Served-By: cache-iad2646-IAD
X-Cache: HIT
X-Cache-Hits: 4
X-Timer: S1500650228.684634,VS0,VE0
Vary: Accept-Encoding
Server: cat factory 1.0
X-Frame-Options: DENY
l5d-success-class: 1.0
Via: 1.1 linkerd
Transfer-Encoding: chunked

^C

# runs linkerd in a daemonset, in linker-to-linker mode
# for CNI setups such as Calico and Weave
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: l5d-config
data:
  config.yaml: |-
    admin:
      port: 9990

    namers:
    # This first namer comes from the default egress example, but has the experimental flag turned
    # on in hopes that it will work with CNI
    - kind: io.l5d.k8s # This namer has the daemonset transformer "built-in"
      prefix: /io.l5d.k8s.ds # We reference this in the outgoing router's dtab
      experimental: true
      host: localhost
      port: 8001
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: default
        port: incoming
        service: l5d
        hostNetwork: true
    # This second name is identical to the namer used in a working TLS CNI example
    - kind: io.l5d.k8s # The "basic" k8s namer.  We reference this in the incoming router's dtab
      experimental: true
      host: localhost
      port: 8001

    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25

    usage:
      orgId: linkerd-examples-daemonset-tls-cni

    routers:
    - protocol: http
      label: outgoing
      # Added egress example entries
      dtab: |
        /ph        => /$/io.buoyant.rinet ; # Lookup the name in DNS
        /srv       => /ph/80 ; # Use port 80 if unspecified
        /srv       => /$/io.buoyant.porthostPfx/ph ; # Attempt to extract the port from the hostname
        /srv        => /#/io.l5d.k8s.ds/default/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      # This is taken from the egress example
      client:
        kind: io.l5d.static
        configs:
        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            commonName: "{service}"
      # This client is from my working TLS CNI example
      # client:
        # tls:
        #   commonName: linkerd
        #   trustCerts:
        #   - /io.buoyant/linkerd/certs/cacertificate.pem
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX
      # interpreter:
      #   kind: default
      #   transformers:
      #   - kind: io.l5d.k8s.daemonset
      #     namespace: default
      #     port: incoming
      #     service: l5d
      #     hostNetwork: true
      servers:
      - port: 4140
        ip: 0.0.0.0

    - protocol: http
      label: incoming
      dtab: |
        /srv        => /#/io.l5d.k8s/default/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true
      servers:
      - port: 4141
        ip: 0.0.0.0
        tls:
          certPath: /io.buoyant/linkerd/certs/certificate.pem
          keyPath: /io.buoyant/linkerd/certs/key.pem
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: l5d
  name: l5d
spec:
  template:
    metadata:
      labels:
        app: l5d
    spec:
      hostNetwork: true
      volumes:
      - name: l5d-config
        configMap:
          name: "l5d-config"
      - name: certificates
        secret:
          secretName: certificates
      containers:
      - name: l5d
        image: buoyantio/linkerd:1.1.2
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - /io.buoyant/linkerd/config/config.yaml
        ports:
        - name: outgoing
          containerPort: 4140
          hostPort: 4140
        - name: incoming
          containerPort: 4141
          hostPort: 4141
        - name: admin
          containerPort: 9990
          hostPort: 9990
        volumeMounts:
        - name: "l5d-config"
          mountPath: "/io.buoyant/linkerd/config"
          readOnly: true
        - name: "certificates"
          mountPath: "/io.buoyant/linkerd/certs"
          readOnly: true

      - name: kubectl
        image: buoyantio/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
# ---
# apiVersion: v1
# kind: Service
# metadata:
#   name: l5d
# spec:
#   selector:
#     app: l5d
#   type: LoadBalancer
#   ports:
#   - name: outgoing
#     port: 4140
#   - name: incoming
#     port: 4141
#   - name: admin
#     port: 9990

That’s quite strange! It looks like you get the response headers but not the body? I’ve tried to reproduce this using the configuration in linkerd/examples/proxy.yaml but it works for me:

$ http_proxy=localhost:4140 curl -si http://imgur.com | head -n 50
HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Cache-Control: max-age=60, stale-while-revalidate=600, stale-if-error=86400, public
Fastly-Debug-Digest: e5ebee0537a9f2c6d062d6bb2c1b785c063368aec041d914c77e2a8e74ed831d
Accept-Ranges: bytes
Date: Fri, 21 Jul 2017 17:42:29 GMT
Age: 63
X-Served-By: cache-iad2127-IAD, cache-lax8632-LAX
X-Cache: HIT, HIT
X-Cache-Hits: 1, 8
X-Timer: S1500658949.482614,VS0,VE0
Vary: Accept-Encoding
Server: cat factory 1.0
X-Frame-Options: DENY
l5d-success-class: 1.0
Via: 1.1 linkerd
Transfer-Encoding: chunked

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
... (body continues) ...

Hey thanks for looking Alex - is your K8s cluster using Calico?

Sadly, as I read the docs I end up with more questions than answers.
Linkerd looks like it solve a lot of issues that we have at Pearson VUE
going forward, but not if we can’t get our base use case to work.

Do you have a better starting point for me where I can get Egress AND Pod
to Pod TLS with Calico?

Leo

I don’t have a K8s cluster with Calico handy to test with, unfortunately, but this should just work. The fact that you are able to get the headers from imgur.com but not the body means that something strange is going on. Are there any proxies other than Linkerd that are involved? If you make that same curl request directly to imgur.com without Linkerd, does it work as expected?

Meanwhile, I’ll see what we can do to investigate on our end.

yes, I can curl imgur.com directly if I log into the hello container and
unset http_proxy.

FWIW I’m going to try the servicemesh.yml that was recently published.

Any luck with the servicemesh.yml config?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.