Performance Testing 101 - 5 min intro & example
When developing and deploying web services, apps or sites the following questions come up: "How will it perform?", "How many concurrent users will it support?", "If I tweak this setting, will it be faster?", "Do these new features effect performance?". The list could go on and on and on. Performance questions are common, solid answers are not.
Performance testing can take many different shapes, from dead-simple one-liners to complex setups, tests, tear-downs and analysis. While this article focuses on quick, easy and straightforward testing, future articles will address more advanced topics.
There are some great easy tools to get first ball-park answers to performance questions getting at the number of concurrent users as well as how the response time changes as load increases. Here I'll give a short intro to ApacheBench.
Let us begin with setting up some basic terminology, first let's refer to our machine under test as the host, this can be any kind of http-accessible server you have. Second we will want an agent machine to drive our tests from.
When performance testing, it is key to limit the number of possible variables which could distort our results. Ideally your agent is a separate dedicated machine and as close as possible (network distance wise) to you host system in order to minimize the amount of networking you test. This is especially relevant when you are testing applications hosted in a shared environment (ie cloud). The performance impact of noisy neighbors can be surprising, but that is a topic we will explore in detail in the future.
ApacheBench
ApacheBench is a command line tool (ab) which allows for simple load driving against HTTP hosts. It's great at producing large numbers of REST requests, capable of producing thousands of requests per second. Generally I find ApacheBench most useful for getting a rough idea of how many requests an application can handle. It's extremely simple to use and therefore a great tool while debugging configurations.
To install on Debian/Ubuntu:
sudo apt-get install apache2-utils
To Install on RHEL/Centos/Fedora
sudo yum install httpd-tools
Usage:
Usage: ab [options] [http[s]://]hostname[:port]/path
Options are:
-n requests Number of requests to perform
-c concurrency Number of multiple requests to make at a time
-t timelimit Seconds to max. to spend on benchmarking
This implies -n 50000
-s timeout Seconds to max. wait for each response
Default is 30 seconds
-b windowsize Size of TCP send/receive buffer, in bytes
-B address Address to bind to when making outgoing connections
-p postfile File containing data to POST. Remember also to set -T
-u putfile File containing data to PUT. Remember also to set -T
-T content-type Content-type header to use for POST/PUT data, eg.
'application/x-www-form-urlencoded'
Default is 'text/plain'
-v verbosity How much troubleshooting info to print
-w Print out results in HTML tables
-i Use HEAD instead of GET
-x attributes String to insert as table attributes
-y attributes String to insert as tr attributes
-z attributes String to insert as td or th attributes
-C attribute Add cookie, eg. 'Apache=1234'. (repeatable)
-H attribute Add Arbitrary header line, eg. 'Accept-Encoding: gzip'
Inserted after all normal header lines. (repeatable)
-A attribute Add Basic WWW Authentication, the attributes
are a colon separated username and password.
-P attribute Add Basic Proxy Authentication, the attributes
are a colon separated username and password.
-X proxy:port Proxyserver and port number to use
-V Print version number and exit
-k Use HTTP KeepAlive feature
-d Do not show percentiles served table.
-S Do not show confidence estimators and warnings.
-q Do not show progress when doing more than 150 requests
-l Accept variable document length (use this for dynamic pages)
-g filename Output collected data to gnuplot format file.
-e filename Output CSV file with percentages served
-r Don't exit on socket receive errors.
-m method Method name
-h Display usage information (this message)
-Z ciphersuite Specify SSL/TLS cipher suite (See openssl ciphers)
-f protocol Specify SSL/TLS protocol
(TLS1, TLS1.1, TLS1.2 or ALL)
Example usage:
ab -c 1 -n 1000 http://example.com/
Putting ApacheBench to use:
The above section should be plenty to get you started, but lets look at a quick example of testing caching. Below I've setup a simple flask server example, which on request calculated a random Fibonacci number between 1 and 30.
#!/usr/bin/env python
#
# To start this server, you must have python and flask installed
# Start server: python testserver-fib.py
#
# To install flask use the pip line below:
# pip install Flask
# or visit: http://flask.pocoo.org/docs/0.12/installation/
from flask import Flask
import random
app = Flask(__name__)
# snagged from: http://stackoverflow.com/a/499245
def F(n):
if n == 0: return 0
elif n == 1: return 1
else: return F(n-1)+F(n-2)
@app.route('/')
def hello_world():
r = random.randint(1,30)
fib = F(r)
# ApacheBench expects constant output
return 'fib({0:02}):{0:06}'.format(r,fib)
if __name__ == "__main__":
app.run(debug=True)
Now let's see how we do performance wise. We set the concurrency to 1 using -c 1 and specify the number of requests to 500 by setting -n 500. Note that we are using the simple flask dev-server, which is single threaded.
m@test:~$ ab -c 1 -n 500 http://127.0.0.1:5000/
-- snip --
Concurrency Level: 1
Time taken for tests: 17.821 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 85000 bytes
HTML transferred: 7000 bytes
Requests per second: 28.06 [#/sec] (mean)
Time per request: 35.642 [ms] (mean)
Time per request: 35.642 [ms] (mean, across all concurrent requests)
Transfer rate: 4.66 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 35 88.2 1 471
Waiting: 0 35 88.2 1 471
Total: 1 36 88.2 1 471
Percentage of the requests served within a certain time (ms)
50% 1
66% 5
75% 16
80% 27
90% 107
95% 277
98% 447
99% 461
100% 471 (longest request)
In the above example, we see the average request time was 34ms, the median was 1ms and we had 28rps (requests per second). What happens if instead of a single connection we have 10 concurrent connections (setting -c 10)?
m@test:~$ ab -c 1 -n 500 http://127.0.0.1:5000/
-- snip --
Concurrency Level: 10
Time taken for tests: 18.579 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 85000 bytes
HTML transferred: 7000 bytes
Requests per second: 26.91 [#/sec] (mean)
Time per request: 371.583 [ms] (mean)
Time per request: 37.158 [ms] (mean, across all concurrent requests)
Transfer rate: 4.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 3 360 294.4 294 1470
Waiting: 2 360 294.4 294 1470
Total: 3 360 294.4 294 1470
Percentage of the requests served within a certain time (ms)
50% 294
66% 443
75% 529
80% 579
90% 746
95% 1018
98% 1136
99% 1160
100% 1470 (longest request)
While the RPS remains very similar to before at ~27rps, our response times have gone through the roof (mean of 371ms and median of 294ms)! Here we have a situation, where multiple parallel connections get serialized and processed one at a time, while the overall rate remains unchanged, the quality of service delivered to each client degrades by a factor roughly similar to the number of concurrent connections.
Let's see if we can do better. Since we repeatedly calculate the same 30 Fibonacci numbers, let's add some caching into the mix. Generally, if you have long-running requests that will always return an unchanging value, it is a good idea to cache these. With the caching in place, the first few requests will still have the same slow response time, but all of the following requests will benefit from the cache and therefore be as fast as our cache lookup. See the modified code below:
#!/usr/bin/env python
#
# * To start this server, you must have python and flask installed
# * Copy this into a file named testserver-fib-cached.py
# * Start server: python testserver-fib-cached.py
#
# * To install flask use the pip line below:
# `pip install Flask`
# or visit: http://flask.pocoo.org/docs/0.12/installation/
from flask import Flask
import random
app = Flask(__name__)
cache = {}
# snagged from: http://stackoverflow.com/a/499245
def F(n):
if n == 0: return 0
elif n == 1: return 1
else: return F(n-1)+F(n-2)
@app.route('/')
def hello_world():
r = random.randint(1,30)
if r in cache:
print('hit')
# ApacheBench expects constant output
return 'Cache Hit! fib({0:02}):{0:06}'.format(r,cache[r])
else:
fib = F(r)
cache[r] = fib
print('miss')
# ApacheBench expects constant output
return 'Cache Miss! fib({0:02}):{0:06}'.format(r,fib)
if __name__ == "__main__":
app.run(debug=True)
Now let's run our single connection, 500 request benchmark again:
-- snip --
Concurrency Level: 1
Time taken for tests: 1.680 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 91000 bytes
HTML transferred: 13000 bytes
Requests per second: 297.55 [#/sec] (mean)
Time per request: 3.361 [ms] (mean)
Time per request: 3.361 [ms] (mean, across all concurrent requests)
Transfer rate: 52.89 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 3 27.7 1 497
Waiting: 0 3 27.7 1 497
Total: 1 3 27.7 1 497
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 7
99% 85
100% 497 (longest request)
The results are quite impressive: the mean is down to 3.3ms, the median down to 1ms and the request rate is at 297rps! That is 10x faster. Once the cache is initialized and our benchmark no longer includes the cache seeding time, we get even higher performance, which at this point is likely limited only by the cache-lookups. My local testing gets me up to 1100rps with median and mean both less than1ms. While this is a simple example for demonstration, it is important to note that part of what we are seeing is a misleading flaw in how most load driving tools generate requests and record latencies, this is known as the coordinated-omission problem, but that is a topic for another day.
This concludes our short introduction into performance testing, but soon to follow will be more articles addressing more complex setups, benchmarking methods and types, metrics to be evaluating and considerations for repeatability.