Linkerd 2.7.0 - Low throughput

linkerd-version: stable-2.7.0 - followed the getting-started guide

I’m currently performance testing Linkerd and Istio in GKE cluster. I use Jmeter to generate a constant throughput to see how both service meshes preforms. I’m currently having a problem to reach high throughput when using Linkerd.

Without Linkerd generating ruffly 500 req/sec before the VM cpu hits the ceiling
With Linkerd the throughput get stuck at 90 req/sec with a VM cpu load ruffly at 40 %.

Can’t find the problem why the throughput is low, do I need to configure Linkerd to use more cpu?
All tips are welcome!
Thanks!

@ErikD, this is great! Thanks for taking the time to run performance testing…

Another community member has been doing some benchmarking and you can find details of that here: Benches for http webservers in different languages

Can you share more detail about your methodology? It will be really helpful to understand the resource profile of the VM that you are using. Do you have details about the services for which Linkerd is proxying requests?

What kind of requests is JMeter making? You can use the grafana dashboards included with Linkerd to get more detail about the resource profiles of the pods where Linkerd is injected. Those might point us in the right direction to explore this behavior.

Finally, try the test with Linkerd 2.7.1, which is the latest version and includes performance improvements for the proxy.

I look forward to hearing the results.

Sorry my bad, where using stable-2.7.1.

Methodology:
Creating cluster, deploying application (namespace default) and Prometheus with node exporter (namespace monitoring) ,deploying Linkerd with side-cars , preheating the service for 2-3min, start benchmark. Jmeter is set to generate a constant throughput of 300 req/sek with 100 user. Jmeter have a script to send different http request, emulating users browsing the store, adding items to their shopping carts and then discarding the carts. No front-end load balancer is used, the website is reached by connecting through nodeport to the web UI service pod.

Set up:
One cluster with five n1-standard-4 (4 vCPU, 15 GB RAM). Where one is the a benchmark node, only running a benchmark pod with Jmeter (no Linkerd side-car). 4 nodes is used to host the micro service test application called Teastore, it’s emulates the basic web store.

Result - 10min test, jmeter set to 300 req/sek:
Throughput did not reach higher then 86 req/sec. It’s seems like the proxy pod for webui is stuck at 0.98 vcpu. Is there a limit for Linkerd proxy set by default?

@ErikD are you running Linkerd in HA mode?

The resources are set only in HA mode and are defined in the helm values-ha.yaml file.

Did use CLI

linkerd install | kubectl apply -f -

by follow the Getting Started guide .

Have tried helm with both values.yaml and values-ha.yaml file with same result, think it was a bit slower with values-ha.yaml. Have also tried CLI with flag --proxy-cpu-limit 4000m.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.