Multiple K8S namespace support w/ static namer config

Help Kubernetes
#1

I’m currently running linkerd as a daemon set in my K8S cluster for testing w/ great success. Thank you for a great project! I am a bit confused, however, about how I can support multiple namespaces w/out having to run multiple daemon sets. I am wanting linkerd to route requests between my gRPC services, but only w/in the same namespace, so I can maintain separation between my test and production namespaces.

Which of the following config files is the correct approach? I’ve tried the first one, but it borked comms in my current test namespace so I backed it out. I have not tried the second one yet.

This config uses the same namer with separate transformers of the same kind for each namespace, and has router configs for each namespace.

admin:
  ip: 0.0.0.0
  port: 9990
namers:
- kind: io.l5d.k8s
- kind: io.l5d.k8s
  prefix: /io.l5d.k8s.grpc
  transformers:
  - kind: io.l5d.k8s.daemonset
    namespace: linkerd
    port: grpc-incoming-test
    service: l5d
  - kind: io.l5d.k8s.daemonset
    namespace: linkerd
    port: grpc-incoming-prod
    service: l5d
routers:
- label: grpc-outgoing-test
  protocol: h2
  experimental: true
  servers:
  - port: 4340
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  dtab: |
    /hp  => /$/inet ;
    /svc => /$/io.buoyant.hostportPfx/hp ;
    /srv => /#/io.l5d.k8s.grpc/test/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;
  client:
    kind: io.l5d.static
    configs:
    - prefix: "/$/inet/{service}"
      tls:
        commonName: "{service}"
- label: gprc-incoming-test
  protocol: h2
  experimental: true
  servers:
  - port: 4341
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  interpreter:
    kind: default
    transformers:
    - kind: io.l5d.k8s.localnode
  dtab: |
    /srv => /#/io.l5d.k8s/test/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;
- label: grpc-outgoing-prod
  protocol: h2
  experimental: true
  servers:
  - port: 4350
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  dtab: |
    /hp  => /$/inet ;
    /svc => /$/io.buoyant.hostportPfx/hp ;
    /srv => /#/io.l5d.k8s.grpc/prod/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;
  client:
    kind: io.l5d.static
    configs:
    - prefix: "/$/inet/{service}"
      tls:
        commonName: "{service}"
- label: gprc-incoming-prod
  protocol: h2
  experimental: true
  servers:
  - port: 4351
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  interpreter:
    kind: default
    transformers:
    - kind: io.l5d.k8s.localnode
  dtab: |
    /srv => /#/io.l5d.k8s/prod/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;

This config uses a separate namer for each namespace and has router configs for each namespace that use the prefix defined in the namer.

admin:
  ip: 0.0.0.0
  port: 9990
namers:
- kind: io.l5d.k8s
- kind: io.l5d.k8s
  prefix: /io.l5d.k8s.grpc-test
  transformers:
  - kind: io.l5d.k8s.daemonset
    namespace: linkerd
    port: grpc-incoming-test
    service: l5d
- kind: io.l5d.k8s
  prefix: /io.l5d.k8s.grpc-prod
  transformers:
  - kind: io.l5d.k8s.daemonset
    namespace: linkerd
    port: grpc-incoming-prod
    service: l5d
routers:
- label: grpc-outgoing-test
  protocol: h2
  experimental: true
  servers:
  - port: 4340
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  dtab: |
    /hp  => /$/inet ;
    /svc => /$/io.buoyant.hostportPfx/hp ;
    /srv => /#/io.l5d.k8s.grpc-test/test/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;
  client:
    kind: io.l5d.static
    configs:
    - prefix: "/$/inet/{service}"
      tls:
        commonName: "{service}"
- label: gprc-incoming-test
  protocol: h2
  experimental: true
  servers:
  - port: 4341
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  interpreter:
    kind: default
    transformers:
    - kind: io.l5d.k8s.localnode
  dtab: |
    /srv => /#/io.l5d.k8s/test/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;
- label: grpc-outgoing-prod
  protocol: h2
  experimental: true
  servers:
  - port: 4350
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  dtab: |
    /hp  => /$/inet ;
    /svc => /$/io.buoyant.hostportPfx/hp ;
    /srv => /#/io.l5d.k8s.grpc-prod/prod/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;
  client:
    kind: io.l5d.static
    configs:
    - prefix: "/$/inet/{service}"
      tls:
        commonName: "{service}"
- label: gprc-incoming-prod
  protocol: h2
  experimental: true
  servers:
  - port: 4351
    ip: 0.0.0.0
  identifier:
    kind: io.l5d.header.path
    segments: 1
  interpreter:
    kind: default
    transformers:
    - kind: io.l5d.k8s.localnode
  dtab: |
    /srv => /#/io.l5d.k8s/prod/grpc ;
    /svc => /$/io.buoyant.http.domainToPathPfx/srv ;

Please help!

#2

Hey @activeshadow - great question. yeah, you’re on the right track. The idea is to have the services call the router that is configured to route within each namespace.

For a quick example, (sorry I wrote this using our http example before seeing you were using grpc):
linkerd.yml:

# runs linkerd in a daemonset, in linker-to-linker mode
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: l5d-config
data:
  config.yaml: |-
    admin:
      ip: 0.0.0.0
      port: 9990

    namers:
    - kind: io.l5d.k8s
      experimental: true
      host: localhost
      port: 8001

    telemetry:
    - kind: io.l5d.prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25

    usage:
      orgId: linkerd-examples-daemonset

    routers:
    - protocol: http
      label: outgoing-first
      dtab: |
        /srv        => /#/io.l5d.k8s/first/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.daemonset
          namespace: default
          port: incoming-first
          service: l5d
      servers:
      - port: 4140
        ip: 0.0.0.0
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX
    - protocol: http
      label: incoming-first
      dtab: |
        /srv        => /#/io.l5d.k8s/first/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
      servers:
      - port: 4141
        ip: 0.0.0.0

    - protocol: http
      label: outgoing-second
      dtab: |
        /srv        => /#/io.l5d.k8s/second/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.daemonset
          namespace: default
          port: incoming-second
          service: l5d
      servers:
      - port: 4142
        ip: 0.0.0.0
      service:
        responseClassifier:
          kind: io.l5d.http.retryableRead5XX
    - protocol: http
      label: incoming-second
      dtab: |
        /srv        => /#/io.l5d.k8s/second/http;
        /host       => /srv;
        /svc        => /host;
        /host/world => /srv/world-v1;
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
      servers:
      - port: 4143
        ip: 0.0.0.0
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: l5d
  name: l5d
spec:
  template:
    metadata:
      labels:
        app: l5d
    spec:
      volumes:
      - name: l5d-config
        configMap:
          name: "l5d-config"
      containers:
      - name: l5d
        image: buoyantio/linkerd:1.2.0
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        args:
        - /io.buoyant/linkerd/config/config.yaml
        ports:
        - name: outgoing-first
          containerPort: 4140
          hostPort: 4140
        - name: incoming-first
          containerPort: 4141
        - name: outgoing-second
          containerPort: 4142
          hostPort: 4142
        - name: incoming-second
          containerPort: 4143
        - name: admin
          containerPort: 9990
        volumeMounts:
        - name: "l5d-config"
          mountPath: "/io.buoyant/linkerd/config"
          readOnly: true

      - name: kubectl
        image: buoyantio/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
---
apiVersion: v1
kind: Service
metadata:
  name: l5d
spec:
  selector:
    app: l5d
  type: LoadBalancer
  ports:
  - name: outgoing-first
    port: 4140
  - name: incoming-first
    port: 4141
  - name: outgoing-second
    port: 4142
  - name: incoming-second
    port: 4143
  - name: admin
    port: 9990

Quick minikube instructions for the above, using the hello world example from linkerd-examples:

kubectl apply -f linkerd.yml
kubectl create ns first
kubectl create ns second
kubectl apply -f hello-world-legacy.yml -n first
kubectl apply -f hello-world-legacy.yml -n second

Then, you can test it out by going to the dtab playground and seeing what pod ips svc/hello resolves to in each ns. Or test out like:

curl $(minikube ip):<nodeport for outgoing-first>
curl $(minikube ip):<nodeport for outgoing-second>