Per-request overrides using HTTP headers

Help
#1

(This issue originally appeared as https://github.com/linkerd/linkerd/issues/1311. The user asked about modifying behavior based on HTTP headers.)

DTabs are the primary routing mechanism in linkerd. Using the l5d-dtab HTTP Header, you can modify routing rules on a per-request basis.

A few examples:

  • Test a new version of the web service, web-v2

    curl -H 'l5d-dtab: /svc/web => /svc/web-v2' "localhost:5000"

  • Force 10% of requests to the web service to fail

    curl -H 'l5d-dtab: /svc/web => 9 * /svc/web-v1 & 1 * /$/nil' "localhost:5000"

  • Inject 100ms latency for 10% of requests to the web service

    # `web-slow` is a modified deployment of `web` which injects latency
curl -H 'l5d-dtab: /svc/web => 9 * /svc/web & 1 * /svc/web-slow' "localhost:5000"

A more complete writeup of this mechanism is on the linkerd blog:
https://blog.buoyant.io/2016/05/04/real-world-microservices-when-services-stop-playing-well-and-start-getting-real/

#2

The most general way to do fault or latency injection would be to have a separate proxy that does this. You could then use per-request dtab overrides to route requests through the fault proxy. Decoupling the fault proxy from linkerd itself means that you can iterate, modify, and restart the fault proxy without needing to restart linkerd.

This is a pretty powerful and very cool pattern and we’d like to write a blog post demonstrating it. Stay tuned.

#3

I’ve written a little library(hopefully will we os but not that difficult to build from scratch) that uses linkerd and toxiproxy for resiliency tests, so yeah separating fault injecter from linkerd makes it way simpler to manage both linkerd and the fault injector.

2 Likes
#4

This is very cool, nice work!