RANDOMLY getting error: No hosts are available for /svc/13.82.188.252:51022, Dtab.base=[/svc=>/#/io.l5d.k8s/linkerd/shopping/shopping-deployment-stable], Dtab.local=[]. Remote Info: Not Available

Hello all, when i am running my jmeter test to this particular shopping service ,out of 2000 requests some 100 to 200 requests were getting this “no host available issue” …Remaining requests are routed to the correct destination.And i get this in a very random manner.Sometimes all the requests are routed successfully without a single error.

In some error it shows info like its trying to route to an ip but that was a deleted pod ip…its failing to route to newly created pod ip. Could someone tell me why this kind of routing is happening and what can the fix?

Hi @shankari

Can you provide some more details please? What version of Linkerd are you running? Can you provide your entire Linkerd config (with confidential information redacted, if necessary)? What command are you running that is producing the above error? Where are you seeing that it is attempting to route to the ip of a deleted pod?

Thanks!

Hello
I am running linkerd 1.3.1…This is my linkerd config:


apiVersion: v1
kind: ConfigMap
metadata:
name: l5d-config
namespace: linkerd
data:
config.yaml: |-
admin:
ip: 0.0.0.0
port: 51029

namers:
- kind: io.l5d.k8s
- kind: io.l5d.k8s
  prefix: /io.l5d.k8s.http
  transformers:
    # The daemonset transformer replaces the address of the target app with
    # the address of the http-incoming router of the Linkerd daemonset pod
    # on the target app's node.
  - kind: io.l5d.k8s.daemonset
    namespace: linkerd
    port: http-incoming
    service: l5d
    hostNetwork: true # Uncomment if using host networking (eg for CNI)
- kind: io.l5d.rewrite
  prefix: /portNsSvcToK8s
  pattern: "/{port}/{ns}/{svc}"
  name: "/k8s/{ns}/{port}/{svc}"

# Telemeters export metrics and tracing data about Linkerd, the services it
# connects to, and the requests it processes.
telemetry:
- kind: io.l5d.prometheus 
- kind: io.l5d.recentRequests
  sampleRate: 0.25
- kind: io.l5d.zipkin
  host: //hidden//
  port: 51021
  sampleRate: 1.0
- kind: io.l5d.tracelog
  sampleRate: 1.0
  level: DEBUG

# Usage is used for anonymized usage reporting.  You can set the orgId to
# identify your organization or set `enabled: false` to disable entirely.
usage:
 orgId: linkerd-examples-servicemesh

routers:
- label: pricing
  protocol: http
  servers:
  - port: 51020
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/pricing/pricing;        
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000
        
- label: scheduling
  protocol: http
  servers:
  - port: 51021
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/scheduling/scheduling;  
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000
        
- label: shopping
  protocol: http
  servers:
  - port: 51022
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/shopping/shopping-deployment-stable;  
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000
        
- label: booking
  protocol: http
  servers:
  - port: 51023
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/booking/booking;        
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000
        
- label: reservation
  protocol: http
  servers:
  - port: 51024
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/reservation/reservation;        
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000
        
- label: payment
  protocol: http
  servers:
  - port: 51025
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/payment/payment;        
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000
        
- label: ticketing
  protocol: http
  servers:
  - port: 51026
    ip: 0.0.0.0
  dtab: |
    /svc => /#/io.l5d.k8s/linkerd/ticketing/ticketing;        
  client:
    loadBalancer:
      kind: ewma
      maxEffort: 10
      decayTimeMs: 15000
    failureAccrual:
      kind: io.l5d.consecutiveFailures
      failures: 2
      backoff:
        kind: constant
        ms: 10000        

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
app: l5d
name: l5d
namespace: linkerd
spec:
template:
metadata:
labels:
app: l5d
spec:
hostNetwork: true # Uncomment to use host networking (eg for CNI)
volumes:
- name: l5d-config
configMap:
name: “l5d-config"
containers:
- name: l5d
image: buoyantio/linkerd:1.3.1
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
args: [”/io.buoyant/linkerd/config/config.yaml"]
ports:
- name: pricing
containerPort: 51020
hostPort: 51020
- name: scheduling
containerPort: 51021
hostPort: 51021
- name: shopping
containerPort: 51022
hostPort: 51022
- name: booking
containerPort: 51023
hostPort: 51023
- name: reservation
containerPort: 51024
hostPort: 51024
- name: payment
containerPort: 51025
hostPort: 51025
- name: ticketing
containerPort: 51026
hostPort: 51026
volumeMounts:
- name: "l5d-config"
mountPath: "/io.buoyant/linkerd/config"
readOnly: true

  # Run `kubectl proxy` as a sidecar to give us authenticated access to the
  # Kubernetes API.
  - name: kubectl
    image: buoyantio/kubectl:v1.4.0
    args:
    - "proxy"
    - "-p"
    - "8001"

apiVersion: v1
kind: Service
metadata:
name: l5d
namespace: linkerd
spec:
selector:
app: l5d
type: LoadBalancer
ports:

  • name: admin
    port: 51029
  • name: pricing
    port: 51020
  • name: scheduling
    port: 51021
  • name: shopping
    port: 51022
  • name: booking
    port: 51023
  • name: reservation
    port: 51024
  • name: payment
    port: 51025
  • name: ticketing
    port: 51026

So here ,when im calling shopping service using JMETER tool,
POST http://13.82.188.252:51022/shop
occasionally i get this error in jmeter response:
No hosts are available for /svc/13.82.188.252:51022, Dtab.base=[/svc=>/#/io.l5d.k8s/linkerd/shopping/shopping-deployment-stable], Dtab.local=[]. Remote Info: Not Available

And regarding that its routing to deleted pod ip,I regularly noticed the pods ips of shopping service…and later when old pods are terminated and new pods are up and in running state…in some of the jmeter response in addition to the above error ,i could see something like this
ChannelException at remote address: /10.244.1.58:51021 from service: #/io.l5d.k8s/linkerd/shopping/shopping-deployment-stable. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: /10.244.1.58:51021, Downstream label: #/io.l5d.k8s/linkerd/shopping/shopping-deployment-stable, Trace Id: f0bcdfed84acc2af.b368e4cb5790238d<:f0bcdfed84acc2af

       In the above error, /10.244.1.58:51021 this is nothing but the deleted pod's ip which i noted down .

Thanks

Ok great, thanks for all the info!

The next thing I would do to figure out what’s going on is to go to the linkerd admin page in your browser: 13.82.188.252:9990, go to the dtab section, select the shopping router, and enter /svc/13.82.188.252:51022 as the path to resolve.

This should give you a graphical representation of how that name is resolved.

Another question is, how many replicas of the shopping-deployment-stable service are there? Do the errors occur while that shopping pod is being rescheduled? If there is a brief window where the shopping pod has been deleted but a new one hasn’t been created, Linkerd will be unable to route to the shopping service for that window of time, which could potentially explain these errors.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.