Http proxy routing problem

Hello,

I am trying to test a simple setup which does http proxyifying.

A server A with linkerd should route http calls to a server B.
An endpoint https://localhost:7000/foo should route to https://foo.local
Everything should be in https.

The server B uses nginx to expose the routes.

I use the following configuration:

admin:
  port: 9990

routers:
- protocol: http
  label: http-proxy
  service:
    responseClassifier:
      kind: io.l5d.http.retryableIdempotent5XX
  identifier:
    kind: io.l5d.path
    segments: 1
    consume: true
  dtab: |
    /svc/foo => /$/inet/foo.local/443;
    /svc/bar => /$/inet/bar.local/443;
    /http/*/* => /svc;
  client:
    tls:
      disableValidation: true
  servers:
  - ip: 0.0.0.0
    port: 7000

telemetry:
- kind: io.l5d.prometheus
- kind: io.l5d.zipkin
  sampleRate: 0.02
- kind: io.l5d.recentRequests
  sampleRate: 1.0
  capacity: 10
- kind: io.l5d.tracelog
  sampleRate: 1.0
  level: TRACE

It is working when I use different ports in the configuration. But when I use the port 443… the routing is not working and I don’t go any further info in the logs. Every call is routed to the firsts service configured in nginx.

I can’t see the real url resolved and called from linkerd. I started the linkerd instance with -log.level=TRACE.

A sample logs trace:

D 0807 13:23:11.959 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.959 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(srv/finagle.version,6.45.0)
D 0807 13:23:11.959 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.959 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] ServiceName(0.0.0.0/7000)
D 0807 13:23:11.961 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.961 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] ServerRecv()
D 0807 13:23:11.962 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.962 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] LocalAddr(/172.17.0.3:7000)
D 0807 13:23:11.963 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.962 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] ServerAddr(/172.17.0.3:7000)
D 0807 13:23:11.964 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.963 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] ClientAddr(/172.17.0.1:39686)
D 0807 13:23:11.965 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.964 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(router.label,http-proxy)
D 0807 13:23:11.967 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.966 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] Rpc(http-proxy /svc/foo)
D 0807 13:23:11.967 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.967 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(dtab.base,/svc/foo=>/$/inet/foo.local/443;/svc/bar=>/$/inet/bar.local/443)
D 0807 13:23:11.967 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.967 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(dtab.local,)
D 0807 13:23:11.968 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.968 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(service,/svc/foo)
D 0807 13:23:11.969 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.969 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] Message(namer.success)
D 0807 13:23:11.969 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.969 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(client,/$/inet/foo.local/443)
D 0807 13:23:11.969 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.969 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] BinaryAnnotation(residual,/)
D 0807 13:23:11.970 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.970 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] ServiceName($/inet/foo.local/443)
D 0807 13:23:11.970 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.970 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(clnt/finagle.version,6.45.0)
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.970 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] ClientSend()
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.971 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] Rpc(GET /api/users/5988603f6522050005841910)
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.971 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.uri,/api/users/5988603f6522050005841910)
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.971 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.req.method,GET)
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.971 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.req.host,gateway.com)
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.971 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.req.version,HTTP/1.1)
D 0807 13:23:11.971 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.971 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.req.content-length,0)
D 0807 13:23:11.972 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.972 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] ServerAddr(foo.local/172.16.1.54:443)
D 0807 13:23:11.972 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.972 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] WireSend
D 0807 13:23:11.977 UTC THREAD32 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.976 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] ClientAddr(/172.17.0.3:56538)
D 0807 13:23:11.985 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.985 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] WireRecv
D 0807 13:23:11.986 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.986 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.rsp.status,200)
D 0807 13:23:11.986 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.986 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.rsp.version,HTTP/1.1)
D 0807 13:23:11.987 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.986 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.rsp.content-length,152)
D 0807 13:23:11.987 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.987 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(http.rsp.content-type,text/html; charset=utf-8)
D 0807 13:23:11.987 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.987 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] BinaryAnnotation(l5d.success,1.0)
D 0807 13:23:11.987 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.987 8c9d9da9ae845aaf.01c7e3045f47956a<:8c9d9da9ae845aaf] ClientRecv()
D 0807 13:23:11.988 UTC THREAD25 TraceId:8c9d9da9ae845aaf: 0807 13:23:11.988 8c9d9da9ae845aaf.8c9d9da9ae845aaf<:8c9d9da9ae845aaf] ServerSend()
D 0807 13:23:12.000 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.000 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] ServiceName(0.0.0.0/7000)
D 0807 13:23:12.000 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.000 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] ServerRecv()
D 0807 13:23:12.000 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.000 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] LocalAddr(/172.17.0.3:7000)
D 0807 13:23:12.001 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.001 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] ServerAddr(/172.17.0.3:7000)
D 0807 13:23:12.001 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.001 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] ClientAddr(/172.17.0.1:39692)
D 0807 13:23:12.001 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.001 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(router.label,http-proxy)
D 0807 13:23:12.000 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.000 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(srv/finagle.version,6.45.0)
D 0807 13:23:12.002 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.002 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] Rpc(http-proxy /svc/bar)
D 0807 13:23:12.002 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.002 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(dtab.base,/svc/foo=>/$/inet/foo.local/443;/svc/bar=>/$/inet/bar.local/443)
D 0807 13:23:12.002 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.002 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(dtab.local,)
D 0807 13:23:12.002 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.002 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(service,/svc/bar)
D 0807 13:23:12.003 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.003 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] Message(namer.success)
D 0807 13:23:12.003 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.003 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(client,/$/inet/bar.local/443)
D 0807 13:23:12.003 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.003 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] BinaryAnnotation(residual,/)
D 0807 13:23:12.005 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.005 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] ServiceName($/inet/bar.local/443)
D 0807 13:23:12.005 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.005 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(clnt/finagle.version,6.45.0)
D 0807 13:23:12.006 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.006 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] ClientSend()
D 0807 13:23:12.006 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.006 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] Rpc(POST /api/push)
D 0807 13:23:12.006 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.006 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.uri,/api/push)
D 0807 13:23:12.006 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.006 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.req.method,POST)
D 0807 13:23:12.006 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.006 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.req.host,gateway.com)
D 0807 13:23:12.007 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.006 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.req.version,HTTP/1.1)
D 0807 13:23:12.007 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.007 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.req.content-length,569)
D 0807 13:23:12.007 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.007 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.req.content-type,application/json)
D 0807 13:23:12.007 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.007 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] ServerAddr(bar.local/172.16.1.54:443)
D 0807 13:23:12.008 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.007 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] WireSend
D 0807 13:23:12.012 UTC THREAD21 TraceId:c05c7ea27a8bea03: 0807 13:23:12.012 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] ClientAddr(/172.17.0.3:56546)
D 0807 13:23:12.018 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.018 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] WireRecv
D 0807 13:23:12.019 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.019 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.rsp.status,400)
D 0807 13:23:12.019 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.019 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.rsp.version,HTTP/1.1)
D 0807 13:23:12.019 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.019 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.rsp.content-length,54)
D 0807 13:23:12.019 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.019 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(http.rsp.content-type,application/json; charset=utf-8)
D 0807 13:23:12.020 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.020 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] BinaryAnnotation(l5d.success,1.0)
D 0807 13:23:12.020 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.020 c05c7ea27a8bea03.e0fecc23006e2d02<:c05c7ea27a8bea03] ClientRecv()
D 0807 13:23:12.021 UTC THREAD31 TraceId:c05c7ea27a8bea03: 0807 13:23:12.021 c05c7ea27a8bea03.c05c7ea27a8bea03<:c05c7ea27a8bea03] ServerSend()

Any idea to have the real url resolved by linkerd?

Thanks in advance.

I recommend looking at the Requests tab in the linkerd admin dashboard. That will show you a log of recent requests and the destinations they were routed to. Looking at your trace logs, it looks like the most recent request was sent to bar.local:

ServerAddr(bar.local/172.16.1.54:443)

Indeed, the requests tab show the right destination but the request is not correctly routed :confused:
What’s the way to see the request sent as TRACE?

I’m not sure what you’re asking for, you’ve got trace logging on and it’s showing the destination as

ServerAddr(bar.local/172.16.1.54:443)

Indeed, the destination seems ok but in fact it is not.

When I do a curl command from the linkerd server, it works:

curl http://foo.local/api/1
curl http://bar.local/api/1

But when I do

curl http://linkerd/foo/api/1
curl http://linkerd/bar/api/2

The routing is not ok with the configuration from the first post.
It routes somewhere where I can not debug, atm.

At the end, it seems like linkerd resolves foo.local to an ip.
Is there a way to configure linkerd to use the dns configured the dtab and disable the dns lookup ?

The issue comes from the fact that the two services are hosted on the same machine…

A DNS name must always be resolved to an IP address before a connection can be established.

What it sounds like maybe you’re asking for is for linkerd to route both /svc/foo and /svc/bar to the same address (ip and port) but to set the Host header differently: (foo.local for /svc/foo and bar.local for /svc/bar). Currently, Linkerd does not support this kind of header modification. Instead, it will simply route the request to the destination address without modifying the Host header in any way.

I understand the resolve part but would it be possible that linkerd uses the dns declared in the config instead of the ip?

Something like that…

foo.local:443 => 172.16.1.54:443

Linkerd resolves the ip, but in the end, it would do a http://foo.loca:443.

Linkerd does not currently support modifying the Host header like this.

Ok, thanks for all your answers.

I will do it an another way.

I think that we can close the topic then.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.