Admin /delegator.json not updating the tree dynamically after flipping traffic

[kunalpathak14]

Hello,
As per our use case we very frequently switch traffic between 2 DC,
I’m trying to curl /admin/delgator.json to fetch me list of ips linkerd is routing the traffic to,
After we flip dtabs to other DC - curl output is still returning me the old list of ips.

When I restart Linkerd service, it gets me correct IPs.
Am I missing anything here, Any suggesations ?

curl -H “Content-Type: application/json” -X POST -d ‘{“namespace”:“abc”,“dtab”:"",“path”:"/svc"}’ http://localhost:7300/delegator.json

[esbie]

Hi @kunalpathak14!

Could you say more about your setup, environment, config, and linkerd version? IPs found in the delegator api are the same ones used by linkerd to route traffic, so old ips returned by the api mean that linkerd hasn’t observed the changes you’re making.

[kunalpathak14]

I can see the actual traffic is routed to correct IPs after we Flip consul to other DC.
but the delegator.json doest not return correct IPs . I use admin/delegator.json only for validation to check correct set of IPs are returned, as I understand it is only read only output. ??

Below is the my Linkerd Config, running this on VMs

`admin:
port: 7300

namers:

• kind: io.l5d.consul
  useHealthCheck: true
  failFast: true
  host: 127.0.0.1
  port: 8500

telemetry:

• kind: io.l5d.influxdb

routers:

• protocol: http
  interpreter:
  kind: io.l5d.namerd.http
  experimental: true
  dst: /$/inet//4180
  namespace: api-abc

  label: abc
  servers:

  • port: 9025
    ip: 0.0.0.0
    client:
    tls:
    commonName: localhost
    trustCerts:
    • abc.pem

• protocol: http
  dtab: |
  /srv => /#/io.l5d.consul/ttce-stage/uncolored-xyz;
  /host => /srv;
  /svc => /host;

  label: xyz
  servers:

  • port: 9027
    ip: 0.0.0.0
    client:
    tls:
    commonName: localhost
    trustCerts:
    • abc.pem
      `

However when I run below on the my Namerd instance correct IPs is getting returned before and after the consul change for flipping(blue/green),

NAMERD -
curl http://localhost:4180/api/1/bind/abc?path=/svc
OUTPUT -
$ curl http://localhost:4180/api/1/bind/abc?path=/svc
{“type”:“leaf”,“bound”:{“addr”:{“type”:“bound”,“addrs”:[{“ip”:“Correct Blue Ips”,“port”:8005,“meta”:{}},{“ip”:“Correct Blue Ips”,“port”:8005,“meta”:{}}],“meta”:{}},“id”:"/#/io.l5d.consul/ttc-stage/blue-abc",“path”:"/"}}

After Flipping the traffic to other DC

(1:35)$ curl http://localhost:4180/api/1/bind/abc?path=/svc
{“type”:“leaf”,“bound”:{“addr”:{“type”:“bound”,“addrs”:[{“ip”:"< Correct Green Point IPs>",“port”:8005,“meta”:{}},{“ip”:"< Correct Green Point IPs >",“port”:8005,“meta”:{}}],“meta”:{}},“id”:"/#/io.l5d.consul/ttce-stage/green-abc",“path”:"/"}}

[kunalpathak14]

Linkerd version I am currently on is 1.2

[esbie]

Yep you are correct delegator.json output is read only. Nice to see that namerd is picking up the correct IPs.

If traffic is being correctly routed through linkerd to the green ips, and it’s only the delegator that’s not working, it sounds like an admin ui/delegator bug. Does the namerd delegator endpoint reflect the changes?

[kunalpathak14]

yep, delegator.json giving correct ips for Namerd when I change dtab for flipping.

[esbie]

okay, and is linkerd routing the traffic correctly? Even though the delegator.json ips are wrong?

[kunalpathak14]

that is correct, linkerd is routing correctly.