Unable to connect to app

When my ‘hello world’ app was deployed for the first time, it looked ok. But after the deletion, redeploy, prompt ‘service failure: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/default/http/mydeploy, Dtab.base=[/svc=>/#/io.l5d.k8s], Dtab.local=[]. Remote Info: Not Available’

My English is not good

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: l5d-config
  namespace: linkerd
data:
  config.yaml: |-
    admin:
      port: 9990
    namers:
    - kind: io.l5d.k8s
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.http
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: linkerd
        port: http-incoming
        service: l5d
        hostNetwork: true # Uncomment if using host networking (eg for CNI)
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.h2
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: linkerd
        port: h2-incoming
        service: l5d
        hostNetwork: true # Uncomment if using host networking (eg for CNI)
    - kind: io.l5d.k8s
      prefix: /io.l5d.k8s.grpc
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: linkerd
        port: grpc-incoming
        service: l5d
        hostNetwork: true # Uncomment if using host networking (eg for CNI)
    - kind: io.l5d.rewrite
      prefix: /portNsSvcToK8s
      pattern: "/{port}/{ns}/{svc}"
      name: "/k8s/{ns}/{port}/{svc}"

    telemetry:
    - kind: io.l5d.prometheus # Expose Prometheus style metrics on :9990/admin/metrics/prometheus
    - kind: io.l5d.recentRequests
      sampleRate: 0.25 # Tune this sample rate before going to production
    routers:
    - label: http-outgoing
      protocol: http
      servers:
      - port: 4140
        ip: 0.0.0.0
      # This dtab looks up service names in k8s and falls back to DNS if they're
      # not found (e.g. for external services). It accepts names of the form
      # "service" and "service.namespace", defaulting the namespace to
      # "default". For DNS lookups, it uses port 80 if unspecified. Note that
      # dtab rules are read bottom to top. To see this in action, on the Linkerd
      # administrative dashboard, click on the "dtab" tab, select "http-outgoing"
      # from the dropdown, and enter a service name like "a.b". (Or click on the
      # "requests" tab to see recent traffic through the system and how it was
      # resolved.)
      dtab: |
        /ph  => /$/io.buoyant.rinet ;                     # /ph/80/google.com -> /$/io.buoyant.rinet/80/google.com
        /svc => /ph/80 ;                                  # /svc/google.com -> /ph/80/google.com
        /svc => /$/io.buoyant.porthostPfx/ph ;            # /svc/google.com:80 -> /ph/80/google.com
        /k8s => /#/io.l5d.k8s.http ;                      # /k8s/default/http/foo -> /#/io.l5d.k8s.http/default/http/foo
        /portNsSvc => /#/portNsSvcToK8s ;                 # /portNsSvc/http/default/foo -> /k8s/default/http/foo
        /host => /portNsSvc/http/default ;                # /host/foo -> /portNsSvc/http/default/foo
        /host => /portNsSvc/http ;                        # /host/default/foo -> /portNsSvc/http/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ; # /svc/foo.default -> /host/default/foo
      client:
        kind: io.l5d.static
        configs:
        # Use HTTPS if sending to port 443
        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            commonName: "{service}"

    - label: http-incoming
      protocol: http
      servers:
      - port: 4141
        ip: 0.0.0.0
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true # Uncomment if using host networking (eg for CNI)
      dtab: |
        /k8s => /#/io.l5d.k8s ;                           # /k8s/default/http/foo -> /#/io.l5d.k8s/default/http/foo
        /portNsSvc => /#/portNsSvcToK8s ;                 # /portNsSvc/http/default/foo -> /k8s/default/http/foo
        /host => /portNsSvc/http/default ;                # /host/foo -> /portNsSvc/http/default/foo
        /host => /portNsSvc/http ;                        # /host/default/foo -> /portNsSvc/http/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ; # /svc/foo.default -> /host/default/foo
    - label: h2-outgoing
      protocol: h2
      experimental: true
      servers:
      - port: 4240
        ip: 0.0.0.0
      dtab: |
        /ph  => /$/io.buoyant.rinet ;                       # /ph/80/google.com -> /$/io.buoyant.rinet/80/google.com
        /svc => /ph/80 ;                                    # /svc/google.com -> /ph/80/google.com
        /svc => /$/io.buoyant.porthostPfx/ph ;              # /svc/google.com:80 -> /ph/80/google.com
        /k8s => /#/io.l5d.k8s.h2 ;                          # /k8s/default/h2/foo -> /#/io.l5d.k8s.h2/default/h2/foo
        /portNsSvc => /#/portNsSvcToK8s ;                   # /portNsSvc/h2/default/foo -> /k8s/default/h2/foo
        /host => /portNsSvc/h2/default ;                    # /host/foo -> /portNsSvc/h2/default/foo
        /host => /portNsSvc/h2 ;                            # /host/default/foo -> /portNsSvc/h2/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ;   # /svc/foo.default -> /host/default/foo
      client:
        kind: io.l5d.static
        configs:
        # Use HTTPS if sending to port 443
        - prefix: "/$/io.buoyant.rinet/443/{service}"
          tls:
            commonName: "{service}"

    - label: h2-incoming
      protocol: h2
      experimental: true
      servers:
      - port: 4241
        ip: 0.0.0.0
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true # Uncomment if using host networking (eg for CNI)
      dtab: |
        /k8s => /#/io.l5d.k8s ;                             # /k8s/default/h2/foo -> /#/io.l5d.k8s/default/h2/foo
        /portNsSvc => /#/portNsSvcToK8s ;                   # /portNsSvc/h2/default/foo -> /k8s/default/h2/foo
        /host => /portNsSvc/h2/default ;                    # /host/foo -> /portNsSvc/h2/default/foo
        /host => /portNsSvc/h2 ;                            # /host/default/foo -> /portNsSvc/h2/default/foo
        /svc => /$/io.buoyant.http.domainToPathPfx/host ;   # /svc/foo.default -> /host/default/foo
    - label: grpc-outgoing
      protocol: h2
      experimental: true
      servers:
      - port: 4340
        ip: 0.0.0.0
      identifier:
        kind: io.l5d.header.path
        segments: 1
      dtab: |
        /hp  => /$/inet ;                                # /hp/linkerd.io/8888 -> /$/inet/linkerd.io/8888
        /svc => /$/io.buoyant.hostportPfx/hp ;           # /svc/linkerd.io:8888 -> /hp/linkerd.io/8888
        /srv => /#/io.l5d.k8s.grpc/default/grpc;         # /srv/service/package -> /#/io.l5d.k8s.grpc/default/grpc/service/package
        /svc => /$/io.buoyant.http.domainToPathPfx/srv ; # /svc/package.service -> /srv/service/package
      client:
        kind: io.l5d.static
        configs:
        # Always use TLS when sending to external grpc servers
        - prefix: "/$/inet/{service}"
          tls:
            commonName: "{service}"

    - label: gprc-incoming
      protocol: h2
      experimental: true
      servers:
      - port: 4341
        ip: 0.0.0.0
      identifier:
        kind: io.l5d.header.path
        segments: 1
      interpreter:
        kind: default
        transformers:
        - kind: io.l5d.k8s.localnode
          hostNetwork: true # Uncomment if using host networking (eg for CNI)
      dtab: |
        /srv => /#/io.l5d.k8s/default/grpc ;             # /srv/service/package -> /#/io.l5d.k8s/default/grpc/service/package
        /svc => /$/io.buoyant.http.domainToPathPfx/srv ; # /svc/package.service -> /srv/service/package
    # HTTP Ingress Controller listening on port 80
    - protocol: http
      label: http-ingress
      servers:
        - port: 80
          ip: 0.0.0.0
          clearContext: true
      identifier:
        kind: io.l5d.ingress
      dtab: /svc => /#/io.l5d.k8s

    # HTTP/2 Ingress Controller listening on port 81
    - protocol: h2
      experimental: true
      label: h2-ingress
      servers:
        - port: 81
          ip: 0.0.0.0
          clearContext: true
      identifier:
        kind: io.l5d.ingress
      dtab: /svc => /#/io.l5d.k8s

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: l5d
  name: l5d
  namespace: linkerd
spec:
  template:
    metadata:
      labels:
        app: l5d
    spec:
      hostNetwork: true # Uncomment to use host networking (eg for CNI)
      nodeSelector:
          node-role.kubernetes.io/node: "true"
      volumes:
      - name: l5d-config
        configMap:
          name: "l5d-config"
      containers:
      - name: l5d
        image: hub.guazi-cloud.com/library/linkerd:1.1.3
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - /io.buoyant/linkerd/config/config.yaml
        ports:
        - name: http-outgoing
          containerPort: 4140
          hostPort: 4140
        - name: http-incoming
          containerPort: 4141
        - name: h2-outgoing
          containerPort: 4240
          hostPort: 4240
        - name: h2-incoming
          containerPort: 4241
        - name: grpc-outgoing
          containerPort: 4340
          hostPort: 4340
        - name: grpc-incoming
          containerPort: 4341
        - name: http-ingress
          containerPort: 80
        - name: h2-ingress
          containerPort: 81
        volumeMounts:
        - name: "l5d-config"
          mountPath: "/io.buoyant/linkerd/config"
          readOnly: true

      # Run `kubectl proxy` as a sidecar to give us authenticated access to the
      # Kubernetes API.
      - name: kubectl
        image: hub.guazi-cloud.com/library/kubectl:v1.4.0
        args:
        - "proxy"
        - "-p"
        - "8001"
---
apiVersion: v1
kind: Service
metadata:
  name: l5d
  namespace: linkerd
spec:
  selector:
    app: l5d
  ports:
  - name: http-outgoing
    port: 4140
  - name: http-incoming
    port: 4141
  - name: h2-outgoing
    port: 4240
  - name: h2-incoming
    port: 4241
  - name: grpc-outgoing
    port: 4340
  - name: grpc-incoming
    port: 4341
  - name: http-ingress
    port: 80
  - name: h2-ingress
    port: 81
1 Like

Can you update your question with how you are testing it?

NoBrokersAvailableException will occur if there is no service available make sure that you have service, “mydeploy” available and is up & running.

kubectl get svc

Try the below command to see if your service is working

curl $(kubectl get svc mydeploy -o jsonpath="{.status.loadBalancer.ingress[0].*}"):<port-number>

`

Look into this,
https://twitter.github.io/finagle/docs/com/twitter/finagle/NoBrokersAvailableException.html

# kubectl get ing
NAME              HOSTS                           ADDRESS   PORTS     AGE
etcd-deployment   etcd-test.guazi-test.com                  80        5h
mydeploy          mytest.default.guazi-test.com             80        4h
worldv2           worldv2.guazi-test.com                    80        <invalid>
# kubectl get endpoints
NAME              ENDPOINTS                                                          AGE
etcd-deployment   192.168.10.176:2379,192.168.10.176:2379                            5h
kubernetes        10.216.91.249:6443,10.216.91.250:6443                              5d
l5d               10.216.91.247:80,10.216.91.248:80,10.216.91.247:9990 + 1 more...   2m
mydeploy          192.168.10.177:80                                                  4h
world-v21         192.168.10.188:7778
# kubectl describe ing worldv2
Name:			worldv2
Namespace:		default
Address:
Default backend:	default-http-backend:80 (<none>)
Rules:
  Host				Path	Backends
  ----				----	--------
  worldv2.guazi-test.com
    				 	world-v21:http (<none>)
Annotations:
Events:	<none>

my service is ok. i can visit it by service ip curl 10.10.238.23:7778
but can not visit by curl worldv2.guazi-test.com

the log :

I 0905 08:41:56.434 UTC THREAD30: k8s found rule matching worldv2.guazi-test.com /: IngressPath(Some(worldv2.guazi-test.com),None,default,world-v21,http)
E 0905 08:41:56.436 UTC THREAD30: service failure: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/default/http/world-v21, Dtab.base=[/svc=>/#/io.l5d.k8s], Dtab.local=[]. Remote Info: Not Available
D 0905 09:08:49.221 UTC THREAD28: k8s ns default modified: world-v21
D 0905 09:08:49.229 UTC THREAD28: k8s ns default service world-v21 port http missing
D 0905 09:08:52.412 UTC THREAD28: k8s ns default deleted: world-v21
D 0905 09:08:52.414 UTC THREAD28: k8s ns default initial state: kubernetes, etcd-deployment, l5d, mydeploy
D 0905 09:08:52.414 UTC THREAD28: k8s ns default service world-v21 missing
D 0905 09:08:52.514 UTC THREAD29: k8s ns default deleted service : world-v21
D 0905 09:09:11.089 UTC THREAD10: UsageMessage(Some(50eba37e-df41-4848-bf21-cea698a4fbf1),Some(linkerd-examples-ingress),Some(1.1.3),None,Some(Linux),Some(4.4.0-83-generic),Some(2017-09-05T09:07Z),List(Router(Some(http),Some(),List(io.l5d.ingress),List())),List(io.l5d.k8s),List(Counter(Some(srv_requests),Some(1))),List(Gauge(Some(jvm_mem),Some(7.7273208E7)), Gauge(Some(jvm/gc/msec),Some(7.7304352E7)), Gauge(Some(jvm/uptime),Some(73153.0)), Gauge(Some(jvm/num_cpus),Some(2.0))))
0905 09:09:11.815 a8fedca0bb0ee5d5.a8fedca0bb0ee5d5<:a8fedca0bb0ee5d5] ServerAddr(stats.buoyant.io/104.28.23.233:443)
0905 09:09:11.815 a8fedca0bb0ee5d5.a8fedca0bb0ee5d5<:a8fedca0bb0ee5d5] ClientAddr(/10.216.91.247:27932)
D 0905 09:09:12.222 UTC THREAD33:
D 0905 09:09:19.612 UTC THREAD28: k8s ns default added: world-v21
D 0905 09:09:19.613 UTC THREAD28: k8s ns default initial state: kubernetes, world-v21, etcd-deployment, l5d, mydeploy
D 0905 09:09:19.613 UTC THREAD28: k8s ns default service world-v21 found
D 0905 09:09:19.613 UTC THREAD28: k8s ns default service world-v21 port http missing
I 0905 09:09:19.700 UTC THREAD29: k8s ns default added service: world-v21
D 0905 09:09:21.009 UTC THREAD28: k8s ns default service world-v21 port http found + /
D 0905 09:09:21.008 UTC THREAD28: k8s ns default modified: world-v21
I 0905 09:09:35.713 UTC THREAD37: k8s no suitable rule found in etcd-deployment for request worldv2.guazi-test.com /
I 0905 09:09:35.713 UTC THREAD37: k8s no suitable rule found in mydeploy for request worldv2.guazi-test.com /
0905 09:09:35.715 4153c9547aba514f.4153c9547aba514f<:4153c9547aba514f] Message(namer.success)
I 0905 09:09:35.713 UTC THREAD37: k8s found rule matching worldv2.guazi-test.com /: IngressPath(Some(worldv2.guazi-test.com),None,default,world-v21,http)
0905 09:09:35.719 4153c9547aba514f.4153c9547aba514f<:4153c9547aba514f] BinaryAnnotation(io.buoyant.router.Failure,ClientAcquisition)
E 0905 09:09:35.727 UTC THREAD37: service failure: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/default/http/world-v21, Dtab.base=[/svc=>/#/io.l5d.k8s], Dtab.local=[]. Remote Info: Not Available
D 0905 09:08:49.221 UTC THREAD28: k8s ns default modified: world-v21
D 0905 09:08:49.229 UTC THREAD28: k8s ns default service world-v21 port http missing
D 0905 09:08:52.412 UTC THREAD28: k8s ns default deleted: world-v21
D 0905 09:08:52.414 UTC THREAD28: k8s ns default initial state: kubernetes, etcd-deployment, l5d, mydeploy
D 0905 09:08:52.414 UTC THREAD28: k8s ns default service world-v21 missing
D 0905 09:08:52.514 UTC THREAD29: k8s ns default deleted service : world-v21

Hmm… Can you share me the world-v21 config?

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: world-v2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: world-v2
    spec:
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: dk-v04
      containers:
      - name: service
        image: hub.com/helloworld:0.1.4
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: TARGET_WORLD
          value: earth
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        args:
        - "-addr=:7778"
        ports:
        - name: service
          containerPort: 7778
---
apiVersion: v1
kind: Service
metadata:
  name: world-v21
spec:
  selector:
    app: world-v2
  ports:
  - name: http
    port: 7778
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: worldv2-ingress
  annotations:
    kubernetes.io/ingress.class: "linkerd"
spec:
  rules:
  - host: worldv2.guazi-test.com
    http:
      paths:
      - backend:
          serviceName: world-v21
          servicePort: http

Is that because the /svc/default/http/world-v21 was marked as “dead” when the service was deleting. when I create service “world-v21” , it was still “dead”?

Possibly related to https://github.com/linkerd/linkerd/issues/1612 ?

I think this is actually an instance of https://github.com/linkerd/linkerd/issues/1635

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.