Proxying Kafka on DC/OS with Linkerd

We’re using DC/OS and leveraging Linkerd as our edge and mesh LB. This post is meant as a dissemination of info about how we proxy our brokers for Kafka to the extra-cluster services that need them. If you have any suggestions or a similar stack / issue this is a good place to discuss it!

We run the Kafka-beta package with 3 brokers, Linkerd configured with path proxy and using marathon as the namer. We had two options in talking to Kafka from outside the cluster:

  1. Query /service/kafka/v1/connection and parse address key for the slaves. This option allows us to use default Kafka clients but anything that is outside of the private subnet isn’t able to access the slaves running the brokers.

Pros: no changes to the Kafka client used by connecting services.

Cons: we have to parse the address strings from /service/kafka/v1/connection and then make requests to the slave directly; we can not use this for services outside the private subnet where the slaves live. This latter bit might be solved by running the brokers on public slaves, but that may break some peoples security model.

  1. Use kafka REST proxy ( DC/OS package and query our linkerd framework which is proxied by an ELB at <linkerd-elb>/kafka-proxy (kafka-proxy is the marathon service name of our kafka-rest proxy framework). This means we have to use REST as our client interface with Kafka.

Pros: easily talk to brokers via a simple layer 7 interface. Can proxy the linkerd framework by a public ELB if desired.

Cons: have to reconfigure the app layer with clients using the REST interface over default Kafka client.

1 Like

Great writeup @malnick! Thanks!