responseClassifier at per service level

zshaik · August 21, 2017, 1:37pm

With linkerd configured to route to external endpoints, my goal is to disable retryable config globally and apply at per service level.

I am not sure if “responseClassifier” can be defined as config element under service. I tried with the config below and looks like only default config (responseClassifier- nonRetryable) is being applied.

Config:

  service:
    kind: io.l5d.static
    configs:
    - prefix: "/$/io.buoyant.rinet/4222/*"
      retries:
        budget:
          minRetriesPerSec: 5
          percentCanRetry: 0.5
          ttlSecs: 15
      responseClassifier:
        kind: io.l5d.http.retryableRead5XX

The above config doesn’t apply to the specified prefix,

  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/requests" : 1,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/budget" : 100.0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/budget_exhausted" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/cannot_retry" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/not_open" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/request_limit" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/requeues" : 0,

klingerf · August 21, 2017, 6:36pm

Hey @zshaik, I haven’t tested this, but am pretty sure the “prefix” field is just a string prefix match – it doesn’t accept wildcards. Can you try /$/io.buoyant.rinet/4222/ instead?

zshaik · August 22, 2017, 11:27am

Hi Kevin, @klingerf thanks for looking into this, removing the wild card didn’t help.
Can you check the configuration below, also I have tested on internal service, “service1”

namers:

  namers:
    - kind: io.l5d.k8s # This namer has the daemonset transformer "built-in"
      prefix: /io.l5d.k8s.ds # We reference this in the outgoing router's dtab
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: default
        port: incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.k8s # The "basic" k8s namer.  We reference this in the incoming router's dtab

- kind: io.l5d.rewrite
  prefix: /portNsSvcToK8s
  pattern: "/{port}/{ns}/{svc}"
  name: "/k8s/{ns}/{port}/{svc}"

tested prefix

- prefix: /svc
- prefix: /#/io.l5d.k8s/default/http/service1
- prefix: /$/io.buoyant.rinet/
- prefix: /$/io.buoyant.rinet/4222/
- prefix: "/$/io.buoyant.rinet/4222/"

linkerd-1.1.3.yml (5.6 KB)
service1.yml (157 Bytes)
helloworld01.yml (584 Bytes)

Alex · August 28, 2017, 5:18pm

Hi @zshaik!

How are you testing that the retryableRead5XX classifier is not being applied to the /$/io.buoyant.rinet/4222/foo.bar.com client?

A good way to debug this is to look at the client registry :9990/admin/registry.json. For each client, the registry will list all params set for that client. Under the /$/io.buoyant.rinet/4222/foo.bar.com client you should be able to see which response classifier it is using.

zshaik · August 29, 2017, 7:10pm

Hey @Alex, Thanks. I was looking into the config.json. The registry.json is not showing any of the static configs(prefix/retries) that’s being setup for retryable configuration

Right now I am testing this on internal service, service3 which returns 500 Err

config

  service:
    kind: io.l5d.static
    configs:
    - prefix: /#/io.l5d.k8s/default/http/service3
      retries:
        budget:
          minRetriesPerSec: 5
          percentCanRetry: 0.5
          ttlSecs: 15
        backoff:
          kind: jittered
          minMs: 10
          maxMs: 10000
      responseClassifier:
        kind: io.l5d.http.retryableRead5XX

Also Why doesn’t this have a “service” : { key?

Alex · August 29, 2017, 8:54pm

Can you share the contents of registry.json?

zshaik · August 30, 2017, 1:10am

Sure, registry.json (57.0 KB)

Alex · August 31, 2017, 6:04pm

Looking at the registry that you sent I only see two clients:

/$/inet/localhost/8001
/$/io.buoyant.rinet/8080/configsvc.us-west-2.net

Are you sure that you’re sending traffic to /#/io.l5d.k8s/default/http/service3? It’s possible that this client has been expired if it has been idle for too long. Are you able to see that client in the registry shortly after sending traffic to it?

zshaik · August 31, 2017, 6:43pm

Ops! My bad, that vanished before I could copy.

Here it is, I copied client and path for “service3” from registry.json

Client:

"client" : {
  "http" : {
    "#/io.l5d.k8s/default/http/service3" : {
      "/#/io.l5d.k8s/default/http/service3" : {
        "ServiceTimeout" : {
          "label" : "#/io.l5d.k8s/default/http/service3",
          "timer" : "DefaultTimer(Netty4Timer)",
          "timeout" : "Duration.Top"
        },
        "ServiceCreationStats" : {
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
        },
        "EndpointTracing" : {
          "addr" : "Failed(IllegalArgumentException(\"failing\"))",
          "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
        },
        "AddrMetadataExtraction" : {
          "dest" : "Path(#,io.l5d.k8s,default,http,service3)",
          "va" : "com.twitter.util.Var$$anon$2@3cace365"
        },
        "PrepFactory" : {
          "enabled" : "false"
        },
        "WireTracingFilter" : {
          "label" : "#/io.l5d.k8s/default/http/service3",
          "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
        },
        "RewriteHostHeader" : {
          "metadata" : "Map()"
        },
        "ExceptionSource" : {
          "label" : "#/io.l5d.k8s/default/http/service3"
        },
        "ClientTracingFilter" : {
          "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a",
          "label" : "#/io.l5d.k8s/default/http/service3"
        },
        "Endpoint" : {
          "transporter" : "com.twitter.finagle.netty4.http.exp$$$Lambda$434/1780399359@328d044f",
          "listener" : "com.twitter.finagle.netty4.http.exp$$$Lambda$437/1802736936@10f7c76",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "serverTransport" : "com.twitter.finagle.Http$$$Lambda$433/653345773@599e4d41",
          "clientTransport" : "com.twitter.finagle.Http$$$Lambda$432/36627152@56da7487",
          "addr" : "Failed(IllegalArgumentException(\"failing\"))",
          "implName" : "Netty4"
        },
        "PerDstPathStatsFilter" : {
          "categorizer" : "DefaultCategorizer",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "unit" : "MILLISECONDS"
        },
        "NackAdmissionFilter" : {
          "nackRateThreshold" : "0.5",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "window" : "2.minutes"
        },
        "EndpointRecorder" : {
          "baseDtab" : "() => com.twitter.finagle.Dtab.base",
          "label" : "#/io.l5d.k8s/default/http/service3",
          "dest" : "Path(#,io.l5d.k8s,default,http,service3)"
        },
        "Binding" : {
          "dest" : "Path(#,io.l5d.k8s,default,http,service3)",
          "baseDtab" : "() => com.twitter.finagle.Dtab.base",
          "label" : "#/io.l5d.k8s/default/http/service3",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "timer" : "DefaultTimer(Netty4Timer)"
        },
        "FactoryStats" : {
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
        },
        "StatsFilter" : {
          "categorizer" : "DefaultCategorizer",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "unit" : "MILLISECONDS"
        },
        "RetryBudget" : {
          "minRetriesPerSec" : "None",
          "ttlSecs" : "None",
          "percentCanRetry" : "None"
        },
        "HttpLogger" : {
          "loggerStack" : "Leaf(role = endpoint, description = endpoint)"
        },
        "PendingRequestLimit" : {
          "limit" : "None",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
        },
        "ExceptionRemoteInfo" : {
          "addr" : "Failed(IllegalArgumentException(\"failing\"))"
        },
        "Retries" : {
          "requeueBackoffs" : "Stream(0.seconds, ?)",
          "timer" : "com.twitter.util.JavaTimer(HighResTimer)",
          "retryBudget" : "TokenRetryBudget(deposit=1000, withdraw=5000, balance=100)",

Path:

   >  "path" : {
>       "http" : {
>         "/svc/service3" : {
>           "/svc/service3" : {
>             "ResponseClassifier.Setter" : {
>               "responseClassifier" : "NonRetryableChunked[HeaderRetryable[ServerErrorsAsFailures orElse DefaultResponseClassifier]]"
>             },
>             "DstTracing.Path" : {
>               "localDtab" : "Dtab()",
>               "label" : "outgoing",
>               "baseDtab" : "Dtab(/ph=>/$/io.buoyant.rinet;/svc=>/ph/80;/svc=>/$/io.buoyant.porthostPfx/ph;/k8s=>/#/io.l5d.k8s;/portNsSvc=>/#/portNsSvcToK8s;/host=>/portNsSvc/http/default;/host=>/portNsSvc/http;/svc=>/$/io.buoyant.http.domainToPathPfx/host)",
>               "path" : "Path(svc,service3)"
>             },
>             "RequestStats" : {
>               "responseClassifier" : "NonRetryableChunked[HeaderRetryable[ServerErrorsAsFailures orElse DefaultResponseClassifier]]",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@4a14f98c",
>               "unit" : "MILLISECONDS",
>               "categorizer" : "DefaultCategorizer"
>             },
>             "Total Timeout" : {
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "timeout" : "Duration.Top"
>             },
>             "RetryBudget" : {
>               "ttlSecs" : "None",
>               "percentCanRetry" : "None",
>               "minRetriesPerSec" : "None"
>             },
>             "Dst.Path.Setter" : {
>               "localDtab" : "Dtab()",
>               "path" : "Path(svc,service3)",
>               "baseDtab" : "Dtab(/ph=>/$/io.buoyant.rinet;/svc=>/ph/80;/svc=>/$/io.buoyant.porthostPfx/ph;/k8s=>/#/io.l5d.k8s;/portNsSvc=>/#/portNsSvcToK8s;/host=>/portNsSvc/http/default;/host=>/portNsSvc/http;/svc=>/$/io.buoyant.http.domainToPathPfx/host)"
>             },
>             "ClassifiedRetries" : {
>               "backoff" : "Stream(0.seconds, ?)",
>               "retryBudget" : "TokenRetryBudget(deposit=1000, withdraw=5000, balance=100)",
>               "requeueBackoffs" : "Stream(0.seconds, ?)",
>               "discard" : "io.buoyant.router.Http$Router$$$Lambda$454/544628437@620f4601",
>               "responseClassifier" : "NonRetryableChunked[HeaderRetryable[ServerErrorsAsFailures orElse DefaultResponseClassifier]]",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@4a14f98c",
>               "timer" : "com.twitter.util.JavaTimer(HighResTimer)"
>             }
>           }
>         },
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "ClassifiedTracing" : {
>               "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
>             },
>             "Monitoring" : {
>               "monitor" : "NullMonitor"
>             },
>             "Expiration" : {
>               "idleTime" : "Duration.Top",
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "lifeTime" : "Duration.Top",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "DtabStats" : {
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "RequestTimeout" : {
>               "timeout" : "Duration.Top",
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "howlong" : "0.seconds"
>             },
>             "TraceInitializerFilter" : {
>               "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
>             },
>             "LoadBalancer" : {
>               "monitor" : "NullMonitor",
>               "reporter" : "LoadedReporterFactory()",
>               "label" : "unknown",
>               "loadBalancerFactory" : "P2CPeakEwma",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "va" : "com.twitter.util.Var$$anon$2@3cace365",
>               "ordering" : "Address.OctetOrdering",
>               "hostStatsReceiver" : "MetricsStatsReceiver",
>               "whenNoNodesOpen" : "PickOne",
>               "log" : "java.util.logging.Logger@a0db585"
>             },
>             "StatusCodeStatsFilter" : {
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "protoTracing" : {
>               "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
>             },
>             "FailureAccrual" : {
>               "label" : "#/io.l5d.k8s/default/http/service3",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "addr" : "Failed(IllegalArgumentException(\"failing\"))",
>               "failureAccrualPolicy" : "io.buoyant.linkerd.failureAccrual.SuccessRateConfig$$Lambda$1327/1902241772@319e9bd6",
>               "log" : "java.util.logging.Logger@a0db585",
>               "timer" : "DefaultTimer(Netty4Timer)"
>             },
>             "Pool" : {
>               "idleTime" : "Duration.Top",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "maxWaiters" : "2147483647",
>               "low" : "0",
>               "bufferSize" : "0",
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "high" : "2147483647"
>             },
>             "FailFast" : {
>               "enabled" : "false",
>               "label" : "#/io.l5d.k8s/default/http/service3",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "addr" : "Failed(IllegalArgumentException(\"failing\"))",
>               "log" : "java.util.logging.Logger@a0db585",
>               "timer" : "DefaultTimer(Netty4Timer)"
>             },
>             "StatsScoping" : {
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "scoper" : "Unscoped",
>               "metadata" : "Map()"
>             },
>             "LatencyCompensation" : {
>               "metadata" : "Map()",
>               "compensator" : "NoCompensation"
>             },
>             "FactoryToService" : {
>               "enabled" : "false"
>             }
>           }
>         },

Alex · August 31, 2017, 6:50pm

Aha! Sorry for not noticing sooner, but the “service” config prefix matches against the service name: /svc/service3.

This means you want a config like:

  service:
    kind: io.l5d.static
    configs:
    - prefix: "/svc/service3"
      ...

zshaik · August 31, 2017, 8:04pm

Awesome! Okay, so because its the “service” config we cannot prefix just like how we did in the “client” and resolves just like the dtab?

Alex · August 31, 2017, 9:11pm

Prefixes in the service configs match against the service name (eg /svc/service3). Prefixes in the client configs match against the client name (eg /#/io.l5d.k8s/foo/bar/bas).