Load Testing with Vegeta

The Big Jump

We had one of those good news/bad news situations at work recently: we learned that we’d be signing a major media site that will end up bringing us something like a 50% increase in traffic (awesome news!) and that they’d start sending us traffic as soon as this week (scary!).

In an ideal world, we’d throttle the traffic from this new site and slowly ramp up, allowing us to see how our system performs under the additional load. We use AWS autoscaling for our api servers, so we have some flexibility, but there are some underlying aspects that don’t scale — we have a fixed number of MySQL instances, for example, and our log rolling has always been a pain point so there’s no guarantees whether it will scale gracefully.

The issue is that the traffic comes from a 3rd party JavaScript library, and there’s not really a graceful way to make a throttling decision on the client side (at least not one that gives us an easy ability to change the throttling rate). The best we could do is implement a request filter that quickly looks up the throttling rate of the site the request is from (there’s some caching involved, but I’m already on a bit of a tangent) and then ‘short circuits’ the request so to speak and sends back a jsonp response that disables our JavaScript.

Ultimately, this means that even in the best case, we’re going to see a 50% jump in initial calls, even if those calls immediately return a disable response. So that means we needed a load testing solution!

The Usual Suspects

Usually, when one thinks of load testing Java applications, Apache’s JMeter comes to mind. There are some other good options as well — wrk seems to be well regarded, for example.

The problem is, most of these solutions involve thread-based workloads. With the previously mentioned wrk, for example, you specify a number of threads and a number of connections and let nature take its course.

For our particular use case, though, we know our requests won’t involve much processing — mainly just checking an in-memory loading cache, maybe a call to memcached and on the rarest of occaisions an actual db call. Our rate of traffic follows fairly steady patterns, and so what’s more important for our use case is testing a specific rate of requests.

Surprisingly, I came across very few options for specifying a certain rate/sec of requests, but there was one that offered this very feature: a golang based tool called Vegeta.

The Super Sayan

And that, friends, is the extent of my DBZ knowledge… back to the topic at hand, however, Vegeta is a powerful but easy to use tool that gives you lots of customization options…

Install

Setting up Vegeta is incredibly simple, which is a huge plus. You can use brew, or if you needed to install on Ubuntu, like we use for our AWS servers, you can either download binaries or the source code from the Vegeta Github Releases page.

Setup

Vegeta is a simple command line tool with very little overhead, but that being said, you’ll want to go ahead and do some preparations for your load test. While you COULD technically setup everything from the command line, for more complicated requests, we can point the Vegeta CLI at various configuration files.

You’ll want to decide…

1- What endpoints to call. We can provide a text file with verb and endpoint, for example.

The second request illustrates how to add additional headers to a single requests (headers added to all requests can be added via a CLI option).

2- The last request above shows how to specify the body of a single request, but we can also specify a file to use for the body of all requests (or at least the ones that aren’t overriden).

3- You’ll want to decide on the load, the number of requests/sec, the duration of the request and how many cores to use for the load testing.

As per the Vegeta repo, here’s the syntax of a vegeta request:

In our case a simple request might look like this…

You can see the vegeta github page for more complicated options like dealing with SSL certs and the like, but in many cases this is enough for a simple load test.

Metrics

While you’ll probably want to rely on some sort of profiling on the actual machines under test, Vegeta also offers some simple metrics from its load testing. There are several options including text and json, but I used Vegeta’s “plot” option, which produces an HTML graph with zoom capabilities.

You can simply pipe the output of your test to Vegeta’s ‘report’ command with a specified output, like so…

This is what my graph looks like (zoomed in to cover about 0.5 seconds of a 30 second test at 500/requests per second):

An Example of a Plot produced by Vegeta

Conclusion

As you can see, Vegeta is as lightweight as it comes for a load testing utility. You can do most of your work on the command line, but still point to files for convenience. When you need to do a quick and dirty test at a set rate of requests, you don’t get much easier than Vegeta.

And as for our new customer, the above graph was performing a load test with ~50% ‘throttled’ requests, and you can see that about half our requests were returning at around or even under 1 ms, which correspond to the throttled requests. This means the throttle feature seems to be a success, and although our call is normally pretty quick, the even faster speed means request handler threads are freed up more quickly and we have a little more wiggle room in one of several potential bottlenecks.

Resources:

https://dzone.com/articles/how-load-test-tune-performance : A comparison of different load testing tools.

https://github.com/tsenart/vegeta : Vegeta load testing tool

https://github.com/wg/wrk : Wrk load testing tool

Leave a Reply

Your email address will not be published. Required fields are marked *