responseClassifier at per service level

With linkerd configured to route to external endpoints, my goal is to disable retryable config globally and apply at per service level.

I am not sure if “responseClassifier” can be defined as config element under service. I tried with the config below and looks like only default config (responseClassifier- nonRetryable) is being applied.

Config:

  service:
    kind: io.l5d.static
    configs:
    - prefix: "/$/io.buoyant.rinet/4222/*"
      retries:
        budget:
          minRetriesPerSec: 5
          percentCanRetry: 0.5
          ttlSecs: 15
      responseClassifier:
        kind: io.l5d.http.retryableRead5XX

The above config doesn’t apply to the specified prefix,

  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/requests" : 1,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/budget" : 100.0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/budget_exhausted" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/cannot_retry" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/not_open" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/request_limit" : 0,
  "rt/outgoing/client/$/io.buoyant.rinet/4222/foo.bar.com/retries/requeues" : 0,

Hey @zshaik, I haven’t tested this, but am pretty sure the “prefix” field is just a string prefix match – it doesn’t accept wildcards. Can you try /$/io.buoyant.rinet/4222/ instead?

Hi Kevin, @klingerf thanks for looking into this, removing the wild card didn’t help.
Can you check the configuration below, also I have tested on internal service, “service1”

namers:

  namers:
    - kind: io.l5d.k8s # This namer has the daemonset transformer "built-in"
      prefix: /io.l5d.k8s.ds # We reference this in the outgoing router's dtab
      transformers:
      - kind: io.l5d.k8s.daemonset
        namespace: default
        port: incoming
        service: l5d
        hostNetwork: true
    - kind: io.l5d.k8s # The "basic" k8s namer.  We reference this in the incoming router's dtab

- kind: io.l5d.rewrite
  prefix: /portNsSvcToK8s
  pattern: "/{port}/{ns}/{svc}"
  name: "/k8s/{ns}/{port}/{svc}"

tested prefix

- prefix: /svc
- prefix: /#/io.l5d.k8s/default/http/service1
- prefix: /$/io.buoyant.rinet/
- prefix: /$/io.buoyant.rinet/4222/
- prefix: "/$/io.buoyant.rinet/4222/"

linkerd-1.1.3.yml (5.6 KB)
service1.yml (157 Bytes)
helloworld01.yml (584 Bytes)

Hi @zshaik!

How are you testing that the retryableRead5XX classifier is not being applied to the /$/io.buoyant.rinet/4222/foo.bar.com client?

A good way to debug this is to look at the client registry :9990/admin/registry.json. For each client, the registry will list all params set for that client. Under the /$/io.buoyant.rinet/4222/foo.bar.com client you should be able to see which response classifier it is using.

1 Like

Hey @Alex, Thanks. I was looking into the config.json. The registry.json is not showing any of the static configs(prefix/retries) that’s being setup for retryable configuration :frowning:

Right now I am testing this on internal service, service3 which returns 500 Err

config

  service:
    kind: io.l5d.static
    configs:
    - prefix: /#/io.l5d.k8s/default/http/service3
      retries:
        budget:
          minRetriesPerSec: 5
          percentCanRetry: 0.5
          ttlSecs: 15
        backoff:
          kind: jittered
          minMs: 10
          maxMs: 10000
      responseClassifier:
        kind: io.l5d.http.retryableRead5XX 

Also Why doesn’t this have a “service” : { key?

Can you share the contents of registry.json?

Sure, registry.json (57.0 KB)

Looking at the registry that you sent I only see two clients:

  • /$/inet/localhost/8001
  • /$/io.buoyant.rinet/8080/configsvc.us-west-2.net

Are you sure that you’re sending traffic to /#/io.l5d.k8s/default/http/service3? It’s possible that this client has been expired if it has been idle for too long. Are you able to see that client in the registry shortly after sending traffic to it?

Ops! My bad, that vanished before I could copy. :turtle:

Here it is, I copied client and path for “service3” from registry.json

Client:

"client" : {
  "http" : {
    "#/io.l5d.k8s/default/http/service3" : {
      "/#/io.l5d.k8s/default/http/service3" : {
        "ServiceTimeout" : {
          "label" : "#/io.l5d.k8s/default/http/service3",
          "timer" : "DefaultTimer(Netty4Timer)",
          "timeout" : "Duration.Top"
        },
        "ServiceCreationStats" : {
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
        },
        "EndpointTracing" : {
          "addr" : "Failed(IllegalArgumentException(\"failing\"))",
          "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
        },
        "AddrMetadataExtraction" : {
          "dest" : "Path(#,io.l5d.k8s,default,http,service3)",
          "va" : "com.twitter.util.Var$$anon$2@3cace365"
        },
        "PrepFactory" : {
          "enabled" : "false"
        },
        "WireTracingFilter" : {
          "label" : "#/io.l5d.k8s/default/http/service3",
          "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
        },
        "RewriteHostHeader" : {
          "metadata" : "Map()"
        },
        "ExceptionSource" : {
          "label" : "#/io.l5d.k8s/default/http/service3"
        },
        "ClientTracingFilter" : {
          "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a",
          "label" : "#/io.l5d.k8s/default/http/service3"
        },
        "Endpoint" : {
          "transporter" : "com.twitter.finagle.netty4.http.exp$$$Lambda$434/1780399359@328d044f",
          "listener" : "com.twitter.finagle.netty4.http.exp$$$Lambda$437/1802736936@10f7c76",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "serverTransport" : "com.twitter.finagle.Http$$$Lambda$433/653345773@599e4d41",
          "clientTransport" : "com.twitter.finagle.Http$$$Lambda$432/36627152@56da7487",
          "addr" : "Failed(IllegalArgumentException(\"failing\"))",
          "implName" : "Netty4"
        },
        "PerDstPathStatsFilter" : {
          "categorizer" : "DefaultCategorizer",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "unit" : "MILLISECONDS"
        },
        "NackAdmissionFilter" : {
          "nackRateThreshold" : "0.5",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "window" : "2.minutes"
        },
        "EndpointRecorder" : {
          "baseDtab" : "() => com.twitter.finagle.Dtab.base",
          "label" : "#/io.l5d.k8s/default/http/service3",
          "dest" : "Path(#,io.l5d.k8s,default,http,service3)"
        },
        "Binding" : {
          "dest" : "Path(#,io.l5d.k8s,default,http,service3)",
          "baseDtab" : "() => com.twitter.finagle.Dtab.base",
          "label" : "#/io.l5d.k8s/default/http/service3",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "timer" : "DefaultTimer(Netty4Timer)"
        },
        "FactoryStats" : {
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
        },
        "StatsFilter" : {
          "categorizer" : "DefaultCategorizer",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
          "unit" : "MILLISECONDS"
        },
        "RetryBudget" : {
          "minRetriesPerSec" : "None",
          "ttlSecs" : "None",
          "percentCanRetry" : "None"
        },
        "HttpLogger" : {
          "loggerStack" : "Leaf(role = endpoint, description = endpoint)"
        },
        "PendingRequestLimit" : {
          "limit" : "None",
          "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
        },
        "ExceptionRemoteInfo" : {
          "addr" : "Failed(IllegalArgumentException(\"failing\"))"
        },
        "Retries" : {
          "requeueBackoffs" : "Stream(0.seconds, ?)",
          "timer" : "com.twitter.util.JavaTimer(HighResTimer)",
          "retryBudget" : "TokenRetryBudget(deposit=1000, withdraw=5000, balance=100)",

Path:

   >  "path" : {
>       "http" : {
>         "/svc/service3" : {
>           "/svc/service3" : {
>             "ResponseClassifier.Setter" : {
>               "responseClassifier" : "NonRetryableChunked[HeaderRetryable[ServerErrorsAsFailures orElse DefaultResponseClassifier]]"
>             },
>             "DstTracing.Path" : {
>               "localDtab" : "Dtab()",
>               "label" : "outgoing",
>               "baseDtab" : "Dtab(/ph=>/$/io.buoyant.rinet;/svc=>/ph/80;/svc=>/$/io.buoyant.porthostPfx/ph;/k8s=>/#/io.l5d.k8s;/portNsSvc=>/#/portNsSvcToK8s;/host=>/portNsSvc/http/default;/host=>/portNsSvc/http;/svc=>/$/io.buoyant.http.domainToPathPfx/host)",
>               "path" : "Path(svc,service3)"
>             },
>             "RequestStats" : {
>               "responseClassifier" : "NonRetryableChunked[HeaderRetryable[ServerErrorsAsFailures orElse DefaultResponseClassifier]]",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@4a14f98c",
>               "unit" : "MILLISECONDS",
>               "categorizer" : "DefaultCategorizer"
>             },
>             "Total Timeout" : {
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "timeout" : "Duration.Top"
>             },
>             "RetryBudget" : {
>               "ttlSecs" : "None",
>               "percentCanRetry" : "None",
>               "minRetriesPerSec" : "None"
>             },
>             "Dst.Path.Setter" : {
>               "localDtab" : "Dtab()",
>               "path" : "Path(svc,service3)",
>               "baseDtab" : "Dtab(/ph=>/$/io.buoyant.rinet;/svc=>/ph/80;/svc=>/$/io.buoyant.porthostPfx/ph;/k8s=>/#/io.l5d.k8s;/portNsSvc=>/#/portNsSvcToK8s;/host=>/portNsSvc/http/default;/host=>/portNsSvc/http;/svc=>/$/io.buoyant.http.domainToPathPfx/host)"
>             },
>             "ClassifiedRetries" : {
>               "backoff" : "Stream(0.seconds, ?)",
>               "retryBudget" : "TokenRetryBudget(deposit=1000, withdraw=5000, balance=100)",
>               "requeueBackoffs" : "Stream(0.seconds, ?)",
>               "discard" : "io.buoyant.router.Http$Router$$$Lambda$454/544628437@620f4601",
>               "responseClassifier" : "NonRetryableChunked[HeaderRetryable[ServerErrorsAsFailures orElse DefaultResponseClassifier]]",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@4a14f98c",
>               "timer" : "com.twitter.util.JavaTimer(HighResTimer)"
>             }
>           }
>         },
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "ClassifiedTracing" : {
>               "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
>             },
>             "Monitoring" : {
>               "monitor" : "NullMonitor"
>             },
>             "Expiration" : {
>               "idleTime" : "Duration.Top",
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "lifeTime" : "Duration.Top",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "DtabStats" : {
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "RequestTimeout" : {
>               "timeout" : "Duration.Top",
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "howlong" : "0.seconds"
>             },
>             "TraceInitializerFilter" : {
>               "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
>             },
>             "LoadBalancer" : {
>               "monitor" : "NullMonitor",
>               "reporter" : "LoadedReporterFactory()",
>               "label" : "unknown",
>               "loadBalancerFactory" : "P2CPeakEwma",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "va" : "com.twitter.util.Var$$anon$2@3cace365",
>               "ordering" : "Address.OctetOrdering",
>               "hostStatsReceiver" : "MetricsStatsReceiver",
>               "whenNoNodesOpen" : "PickOne",
>               "log" : "java.util.logging.Logger@a0db585"
>             },
>             "StatusCodeStatsFilter" : {
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d"
>             },
>             "protoTracing" : {
>               "tracer" : "io.buoyant.telemetry.recentRequests.RecentRequetsTracer@5a3031a"
>             },
>             "FailureAccrual" : {
>               "label" : "#/io.l5d.k8s/default/http/service3",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "addr" : "Failed(IllegalArgumentException(\"failing\"))",
>               "failureAccrualPolicy" : "io.buoyant.linkerd.failureAccrual.SuccessRateConfig$$Lambda$1327/1902241772@319e9bd6",
>               "log" : "java.util.logging.Logger@a0db585",
>               "timer" : "DefaultTimer(Netty4Timer)"
>             },
>             "Pool" : {
>               "idleTime" : "Duration.Top",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "maxWaiters" : "2147483647",
>               "low" : "0",
>               "bufferSize" : "0",
>               "timer" : "DefaultTimer(Netty4Timer)",
>               "high" : "2147483647"
>             },
>             "FailFast" : {
>               "enabled" : "false",
>               "label" : "#/io.l5d.k8s/default/http/service3",
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "addr" : "Failed(IllegalArgumentException(\"failing\"))",
>               "log" : "java.util.logging.Logger@a0db585",
>               "timer" : "DefaultTimer(Netty4Timer)"
>             },
>             "StatsScoping" : {
>               "statsReceiver" : "io.buoyant.telemetry.MetricsTreeStatsReceiver@60e7c41d",
>               "scoper" : "Unscoped",
>               "metadata" : "Map()"
>             },
>             "LatencyCompensation" : {
>               "metadata" : "Map()",
>               "compensator" : "NoCompensation"
>             },
>             "FactoryToService" : {
>               "enabled" : "false"
>             }
>           }
>         },

Aha! Sorry for not noticing sooner, but the “service” config prefix matches against the service name: /svc/service3.

This means you want a config like:

  service:
    kind: io.l5d.static
    configs:
    - prefix: "/svc/service3"
      ...
1 Like

Awesome! Okay, so because its the “service” config we cannot prefix just like how we did in the “client” and resolves just like the dtab?

Prefixes in the service configs match against the service name (eg /svc/service3). Prefixes in the client configs match against the client name (eg /#/io.l5d.k8s/foo/bar/bas).

1 Like