Unfortunately in August 2017, CrashPlan announced they will discontinue their consumer offering in order to refocus on small-businesses, which I'm sure is a much more profitable market. The announcement email has been sitting starred in my inbox since that day and I've been dreading to find the next solutions. I remembered, last time I looked around, CrashPlan was more or less the only self-hosted choice that worked across all OSs and offered incremental encrypted backups. More recently I came across the new Duplicati, which has seen a major revamp since the last time I had looked at it, now supporting all sorts of storage backends, encryption, incremental backups and a variety of retention policies as well as a slick new web-based UI.

Below I will describe how to setup duplicati to use a central linux server as the backend to store all of your files.

Setup and Install Duplicati

On the client machine, which will be backed up, go to duplicati.com/download and grab the latest installer for your operating system and follow the default installation. Below are the directions I used on my Ubuntu Laptop

user@box:~$ wget https://updates.duplicati.com/beta/duplicati_2.0.3.3-1_all.deb
user@box:~$ sudo dpkg -i duplicati_2.0.3.3-1_all.deb

then start duplicati and continue in the web interface at localhost:8200/ngax/index.html.

Adding a Backup location

I usually like to keep all of my data on my own servers so here we will configure backing up via SSH, making sure to setup ssh-keys and a separate backup user on my disk box.

1. Getting Started Select "+ Add Backup", then "Configure New Backup", then "next"
   

2. Encryption A great feature of duplicati is the encryption of backups, to make good use of it, generate a password and store it in your password manager, I prefer KeePass and LastPass.
   

3. Destination Next we select where the backups will be stored, this could be an external drive attached to your computer or a remote machine via SSH, FTP, WebDav or even S3-like storage. For the purpose of this tutorial we will go with SSH.

• Add SSH Key When using SSH, I recommend using an SSH key for security, especially if you use the default SSH port. Under Advanced Settings select ssh-keyfile, if your key has a password, enter this into the Password field above.

• Test your connection Now hit Test Connection to make sure everything works properly. You'll be asked whether you accept the Host-key, select Yes, then go Next

4. Configure Source Here we select the files we wish yo backup. Generally on a linux machine I backup ~/ and exclude some of the dot-files.

5. Backup Schedule Generally I don't need backup every day, so go with Weekly backup.

6. General Settings These final settings are important.

• Volume Size controls the size of chunks that will be pushed to your backup system and will therefore control how many file will land on your backup server. I've found 50MB to work well for me.
• blocksize controls the size of the blocks which will be hashed, i set this to 1MB, cutting down on the size of the database duplicati has to maintain.
• Backup Retention Generally it is most useful to thin-out older backups in order to save space. I set this to Smart Backup Retention.

Play with these settings to find what is most optimal for your setup, it will depend on your upload speed, the backend server as well as the compute capability of your client device.

All Done!
Now, a word of wisdom, after you get this setup, make sure you can successfully go through a full backup-restore cycle to work any kinks you may encounter.

After the upgrade my system did not automatically start eth0 with its assigned static IP address, only the loopback device was configured. Thankfully DigitalOcean provides easy console access. So i started some debugging:

m@box:~$ ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

m@box:~$ cat ifconfig-a.log 
eth0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 04:01:01:8a:1b:01  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Turns out the root cause was no ifup or ifdown, which were both scripts required by /etc/init.d/networking to bring up networking

m@box:~$ ifup eth0
-bash: ifup: command not found
m@box:~$ ifdown eth0
-bash: ifdown: command not found

Excerpt from /etc/init.d/networking

m@box:~$ grep -in "ifup\|ifdown" /etc/init.d/networking 
3:# Provides:          networking ifupdown
17:[ -x /sbin/ifup ] || exit 0
18:[ -x /sbin/ifdown ] || exit 0
101:ifup_hotplug () {
115:		ifup $ifaces "$@" || true
141:	if ifup -a $exclusions $verbose && ifup_hotplug $exclusions $verbose
157:	if ifdown -a --exclude=lo $verbose; then
172:	ifdown -a --exclude=lo $verbose || true
173:	if ifup --exclude=lo $state $verbose ; then
188:	ifdown -a --exclude=lo $verbose || true
191:	if ifup -a --exclude=lo $exclusions $verbose && ifup_hotplug $exclusions $verbose

Temp fix to get back onto the internet to actually install networkd/netplan:

m@box:~$ sudo ifconfig eth0 up
# xx.xx.xx.xx/24 is your static IP
m@box:~$ sudo ip addr add xx.xx.xx.xx/24 dev eth0
# xx.xx.xx.1 is your gateway
m@box:~$ sudo ip route add default via xx.xx.xx.1 dev eth0

long-term fix, install netplan, configure it to manage networkd:

m@box:~$ sudo apt-get install netplan.io

We configure netplan(/etc/netplan/01-netcfg.yaml):

network:
  version: 2
  renderer: networkd
  ethernets:
   eth0:
    dhcp4: no
    dhcp6: no
    addresses: [xx.xx.xx.xx/24]
    gateway4: xx.xx.xx.1
    nameservers:
     addresses: [8.8.8.8, 8.8.4.4]

next we apply the new changes and make sure it gives no errors:

m@box:~$ sudo netplan --debug apply
** (generate:1883): DEBUG: 19:19:36.239: Processing input file //etc/netplan/01-netcfg.yaml..
** (generate:1883): DEBUG: 19:19:36.240: starting new processing pass
** (generate:1883): DEBUG: 19:19:36.240: eth0: setting default backend to 1
** (generate:1883): DEBUG: 19:19:36.244: Generating output files..
** (generate:1883): DEBUG: 19:19:36.244: NetworkManager: definition eth0 is not for us (backend 1)
DEBUG:netplan generated networkd configuration exists, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:device eth0 operstate is up, not replugging
DEBUG:netplan triggering .link rules for eth0
DEBUG:device lo operstate is unknown, not replugging
DEBUG:netplan triggering .link rules for lo

now the last step is to reboot and make sure the new configuration persists.

And viola, after a reboot everything stayed in place:

m@box:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet xx.xx.xx.xx  netmask 255.255.255.0  broadcast xx.xx.xx.255
        inet6 xx::xx:xx:xx:xx  prefixlen 64  scopeid 0x20<link>
        ether 04:01:xx:xx:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 654  bytes 69420 (69.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 517  bytes 108478 (108.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

First, let's start with a simple set of performance test results, featuring 1000 samples of a web service endpoint.

Our favorite summary statistics have been overlaid, showing us the mean, median and +/- standard deviation. At first glance there is nothing interesting about these results. We see some variability, perhaps due to network jitter or load on the system, but otherwise a pretty consistent result. Can you identify any interesting features? Given these summary statistics, could you determine a change in behaviour?
In the above example, the mean sits at 26.6, the median at 26.01 and we have a standard deviation of around 3. Given that the median is slightly lower than the mean suggests we may have a positively skewed distribution. Which is a common feature of latency distributions, since there are at least a few packets that always hit snags such as errors or taking the scenic network path.
If we just look at the summary statistics, we do not get the full picture. Figure 2 shows two distributions with an identical mean, median and standard deviation, but as you can see these are very different in shape.
Could you identify which distribution corresponds to the samples from figure 1?

Intuitively and most commonly with latencies, the distribution tends to look more like figure 2(a), but in our case the actual distribution is as in figure 2(b). Multi-modal distributions often indicate some sort of caching at work, the lower mode representing a cache hit and the higher mode a cache miss. Understanding changes in the relationship between cache hits and misses is very important, as a rise in cache misses could indicate a serious problem.
Given only the medians, means and standard deviations, it would be impossible to determine any difference, therefore performance changes such as these would never surface. There is no easy solution here besides adding more advanced metrics. One such metric to consider is the Kolmogorov–Smirnov test, which computes the difference between two Cumulative Density Functions.

Another gotcha are results, which when evaluated based on their summary statistics and their distribution (see figure 3), look to be completely normal, no bimodal tendencies, a slight right skew, but nothing that stands out.

These are the trickiest, the ones that don't ring any alarm bells, are the ones that will bite you once you go into production. The critical missing piece is the time-domain information of the original results, which by definition cannot be captured by summary statistics or distributions.
Collecting time-domain information usually is not a problem, but on high throughput tests, it may become prohibitively expensive, both memory- and storage-wise. Instead you may be tempted to purely rely on streaming statistics, perhaps using a snazzy sliding-window histogram to do reservoir sampling or something like the t-digest. These are fantastic approaches and I am absolutely in favor of using these, but if you do not keep at least some interval-based snapshots of the streaming statistics, you may end up discarding valuable information.
Let's return to our example from figure 3, when viewed as a time-series, see figure 4, it is clear that we have significant trend!

Trends cannot be characterized using summary statistics and add extra complexity to performance comparisons, therefore should be avoided whenever possible.
To ensure that that you trends do not secretly distort your statistics, compute a robust linear regression metric (I've had good luck with the RANSAC algorithm) to quantify the trend in terms of slope and y-intercept. Given these metrics it becomes easy to develop a sanity check to determine whether any drastic trend changes have occurred.

Summary statistics can be valuable first indicators about performance, but can easily lead to false conclusions if not combined with other metrics. It is especially important to retain time-domain information to be able to detect trends which might otherwise be hidden. Stay tuned for future posts which will deep dive on how to accurately detect the slightest performance changes.

Configuring the cache:

First we need to enable mod_cache, mod_cache_disk and mod_expires:

sudo a2enmod cache
sudo a2enmod cache_disk
sudo a2enmod expires

Then edit your virtual host file, usually /etc/apache2/sites-enabled/default.conf (may differ based on setup)

<VirtualHost *:80>
     # Domain name and Alias
     ServerName example.com
     ServerAlias www.example.com

     # Configure Reverse proxy for Ghost
     ProxyPreserveHost on
     ProxyPass / http://localhost:1234/
     ProxyPassReverse / http://localhost:1234/

     CacheQuickHandler off
     CacheLock on
     CacheLockPath /tmp/mod_cache-lock
     CacheLockMaxAge 5
     CacheIgnoreHeaders Set-Cookie

     <Location />
        # Enable disk cache, set defaults
        CacheEnable disk
        CacheHeader on
        CacheDefaultExpire 600
        CacheMaxExpire 86400
        FileETag All

        # Set cache-control headers for all request
        # which do not have them by default
        # must enable: mod_expires
        ExpiresActive on
        ExpiresDefault "access plus 15 minutes"
    </Location>

    # While this is not needed since Ghost automatically
    # passes back 'Cache-Control: no-cache, private'
    # It makes me feel better to explicitly state it again.
    <Location /ghost>
        # Don't cache the ghost admin interface
        SetEnv no-cache
    </Location>

</VirtualHost>

Finally we need to restart apache and then we are done!

sudo service apache2 restart

Now it's time to check whether your cache is working by inspecting the headers.

:~$ curl -i -X GET http://example.com | less
HTTP/1.1 200 OK
Date: Sun, 21 Apr 2017 05:15:13 GMT
Server: Apache
Cache-Control: public, max-age=0, max-age=900
Expires: Sun, 21 Apr 2017 05:30:12 GMT
Age: 832
X-Cache: HIT from example.com
...

As long as you see an X-Cache and a Cache-Control header it is all working. Now lets see what kind of performance improvement we have achieved.

Performance Testing:

To quantify the improvement, I broke out ApacheBench and did a couple of quick tests from a neighboring machine.
First test without caching enabled, yielding approximately 25 Requests per second with a median response time of ~4 seconds!:

Concurrency Level:      100
Time taken for tests:   40.943 seconds
Complete requests:      1000
Requests per second:    24.42 [#/sec] (mean)
Time per request:       4094.286 [ms] (mean)
Time per request:       40.943 [ms] (mean, across all concurrent requests)
Transfer rate:          282.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.7      1      14
Processing:  1052 3936 670.4   3933    5252
Waiting:     1052 3936 670.4   3933    5252
Total:       1056 3938 669.7   3934    5252

Percentage of the requests served within a certain time (ms)
  50%   3934
  66%   4119
  75%   4261
  80%   4406
  90%   4554
  95%   5067
  98%   5173
  99%   5211
 100%   5252 (longest request)

After enabling the caching, we get ~2700 Requests per second with a median response time of 31 milliseconds!. That is 100x more requests served per second!

Concurrency Level:      100
Time taken for tests:   18.487 seconds
Complete requests:      50000
Failed requests:        0
Total transferred:      597400000 bytes
HTML transferred:       578850000 bytes
Requests per second:    2704.57 [#/sec] (mean)
Time per request:       36.974 [ms] (mean)
Time per request:       0.370 [ms] (mean, across all concurrent requests)
Transfer rate:          31556.82 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    8  52.1      4    1019
Processing:     1   29  19.1     26     693
Waiting:        1   28  16.9     26     659
Total:          2   37  55.1     31    1158

Percentage of the requests served within a certain time (ms)
  50%     31
  66%     34
  75%     37
  80%     40
  90%     46
  95%     51
  98%     67
  99%     91
 100%   1158 (longest request)

Performance testing can take many different shapes, from dead-simple one-liners to complex setups, tests, tear-downs and analysis. While this article focuses on quick, easy and straightforward testing, future articles will address more advanced topics.

There are some great easy tools to get first ball-park answers to performance questions getting at the number of concurrent users as well as how the response time changes as load increases. Here I'll give a short intro to ApacheBench.

Let us begin with setting up some basic terminology, first let's refer to our machine under test as the host, this can be any kind of http-accessible server you have. Second we will want an agent machine to drive our tests from.
When performance testing, it is key to limit the number of possible variables which could distort our results. Ideally your agent is a separate dedicated machine and as close as possible (network distance wise) to you host system in order to minimize the amount of networking you test. This is especially relevant when you are testing applications hosted in a shared environment (ie cloud). The performance impact of noisy neighbors can be surprising, but that is a topic we will explore in detail in the future.

ApacheBench

ApacheBench is a command line tool (ab) which allows for simple load driving against HTTP hosts. It's great at producing large numbers of REST requests, capable of producing thousands of requests per second. Generally I find ApacheBench most useful for getting a rough idea of how many requests an application can handle. It's extremely simple to use and therefore a great tool while debugging configurations.

To install on Debian/Ubuntu:

sudo apt-get install apache2-utils

To Install on RHEL/Centos/Fedora

sudo yum install httpd-tools

Usage:

Usage: ab [options] [http[s]://]hostname[:port]/path
Options are:
    -n requests     Number of requests to perform
    -c concurrency  Number of multiple requests to make at a time
    -t timelimit    Seconds to max. to spend on benchmarking
                    This implies -n 50000
    -s timeout      Seconds to max. wait for each response
                    Default is 30 seconds
    -b windowsize   Size of TCP send/receive buffer, in bytes
    -B address      Address to bind to when making outgoing connections
    -p postfile     File containing data to POST. Remember also to set -T
    -u putfile      File containing data to PUT. Remember also to set -T
    -T content-type Content-type header to use for POST/PUT data, eg.
                    'application/x-www-form-urlencoded'
                    Default is 'text/plain'
    -v verbosity    How much troubleshooting info to print
    -w              Print out results in HTML tables
    -i              Use HEAD instead of GET
    -x attributes   String to insert as table attributes
    -y attributes   String to insert as tr attributes
    -z attributes   String to insert as td or th attributes
    -C attribute    Add cookie, eg. 'Apache=1234'. (repeatable)
    -H attribute    Add Arbitrary header line, eg. 'Accept-Encoding: gzip'
                    Inserted after all normal header lines. (repeatable)
    -A attribute    Add Basic WWW Authentication, the attributes
                    are a colon separated username and password.
    -P attribute    Add Basic Proxy Authentication, the attributes
                    are a colon separated username and password.
    -X proxy:port   Proxyserver and port number to use
    -V              Print version number and exit
    -k              Use HTTP KeepAlive feature
    -d              Do not show percentiles served table.
    -S              Do not show confidence estimators and warnings.
    -q              Do not show progress when doing more than 150 requests
    -l              Accept variable document length (use this for dynamic pages)
    -g filename     Output collected data to gnuplot format file.
    -e filename     Output CSV file with percentages served
    -r              Don't exit on socket receive errors.
    -m method       Method name
    -h              Display usage information (this message)
    -Z ciphersuite  Specify SSL/TLS cipher suite (See openssl ciphers)
    -f protocol     Specify SSL/TLS protocol
                    (TLS1, TLS1.1, TLS1.2 or ALL)

Example usage:

 ab -c 1 -n 1000 http://example.com/

Putting ApacheBench to use:

The above section should be plenty to get you started, but lets look at a quick example of testing caching. Below I've setup a simple flask server example, which on request calculated a random Fibonacci number between 1 and 30.

#!/usr/bin/env python
#
# To start this server, you must have python and flask installed
# Start server: python testserver-fib.py
#
# To install flask use the pip line below:
# pip install Flask
# or visit: http://flask.pocoo.org/docs/0.12/installation/

from flask import Flask
import random
app = Flask(__name__)

# snagged from: http://stackoverflow.com/a/499245
def F(n):
    if n == 0: return 0
    elif n == 1: return 1
    else: return F(n-1)+F(n-2)

@app.route('/')
def hello_world():
    r = random.randint(1,30)
    fib = F(r)
    # ApacheBench expects constant output
    return 'fib({0:02}):{0:06}'.format(r,fib)

if __name__ == "__main__":
    app.run(debug=True)

Now let's see how we do performance wise. We set the concurrency to 1 using -c 1 and specify the number of requests to 500 by setting -n 500. Note that we are using the simple flask dev-server, which is single threaded.

m@test:~$ ab -c 1 -n 500 http://127.0.0.1:5000/
-- snip --
Concurrency Level:      1
Time taken for tests:   17.821 seconds
Complete requests:      500
Failed requests:        0
Total transferred:      85000 bytes
HTML transferred:       7000 bytes
Requests per second:    28.06 [#/sec] (mean)
Time per request:       35.642 [ms] (mean)
Time per request:       35.642 [ms] (mean, across all concurrent requests)
Transfer rate:          4.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1   35  88.2      1     471
Waiting:        0   35  88.2      1     471
Total:          1   36  88.2      1     471

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      5
  75%     16
  80%     27
  90%    107
  95%    277
  98%    447
  99%    461
 100%    471 (longest request)

In the above example, we see the average request time was 34ms, the median was 1ms and we had 28rps (requests per second). What happens if instead of a single connection we have 10 concurrent connections (setting -c 10)?

m@test:~$ ab -c 1 -n 500 http://127.0.0.1:5000/
-- snip --
Concurrency Level:      10
Time taken for tests:   18.579 seconds
Complete requests:      500
Failed requests:        0
Total transferred:      85000 bytes
HTML transferred:       7000 bytes
Requests per second:    26.91 [#/sec] (mean)
Time per request:       371.583 [ms] (mean)
Time per request:       37.158 [ms] (mean, across all concurrent requests)
Transfer rate:          4.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     3  360 294.4    294    1470
Waiting:        2  360 294.4    294    1470
Total:          3  360 294.4    294    1470

Percentage of the requests served within a certain time (ms)
  50%    294
  66%    443
  75%    529
  80%    579
  90%    746
  95%   1018
  98%   1136
  99%   1160
 100%   1470 (longest request)

While the RPS remains very similar to before at ~27rps, our response times have gone through the roof (mean of 371ms and median of 294ms)! Here we have a situation, where multiple parallel connections get serialized and processed one at a time, while the overall rate remains unchanged, the quality of service delivered to each client degrades by a factor roughly similar to the number of concurrent connections.

Let's see if we can do better. Since we repeatedly calculate the same 30 Fibonacci numbers, let's add some caching into the mix. Generally, if you have long-running requests that will always return an unchanging value, it is a good idea to cache these. With the caching in place, the first few requests will still have the same slow response time, but all of the following requests will benefit from the cache and therefore be as fast as our cache lookup. See the modified code below:

#!/usr/bin/env python
#
# * To start this server, you must have python and flask installed
# * Copy this into a file named testserver-fib-cached.py
# * Start server: python testserver-fib-cached.py
#
# * To install flask use the pip line below:
#      `pip install Flask`
#   or visit: http://flask.pocoo.org/docs/0.12/installation/
from flask import Flask
import random
app = Flask(__name__)

cache = {}

# snagged from: http://stackoverflow.com/a/499245
def F(n):
    if n == 0: return 0
    elif n == 1: return 1
    else: return F(n-1)+F(n-2)

@app.route('/')
def hello_world():
    r = random.randint(1,30)
    if r in cache:
        print('hit')
        # ApacheBench expects constant output
        return 'Cache Hit!  fib({0:02}):{0:06}'.format(r,cache[r])
    else:
        fib = F(r)
        cache[r] = fib
        print('miss')
        # ApacheBench expects constant output
        return 'Cache Miss! fib({0:02}):{0:06}'.format(r,fib)

if __name__ == "__main__":
    app.run(debug=True)

Now let's run our single connection, 500 request benchmark again:

-- snip --
Concurrency Level:      1
Time taken for tests:   1.680 seconds
Complete requests:      500
Failed requests:        0
Total transferred:      91000 bytes
HTML transferred:       13000 bytes
Requests per second:    297.55 [#/sec] (mean)
Time per request:       3.361 [ms] (mean)
Time per request:       3.361 [ms] (mean, across all concurrent requests)
Transfer rate:          52.89 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    3  27.7      1     497
Waiting:        0    3  27.7      1     497
Total:          1    3  27.7      1     497

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      7
  99%     85
 100%    497 (longest request)

The results are quite impressive: the mean is down to 3.3ms, the median down to 1ms and the request rate is at 297rps! That is 10x faster. Once the cache is initialized and our benchmark no longer includes the cache seeding time, we get even higher performance, which at this point is likely limited only by the cache-lookups. My local testing gets me up to 1100rps with median and mean both less than1ms. While this is a simple example for demonstration, it is important to note that part of what we are seeing is a misleading flaw in how most load driving tools generate requests and record latencies, this is known as the coordinated-omission problem, but that is a topic for another day.

This concludes our short introduction into performance testing, but soon to follow will be more articles addressing more complex setups, benchmarking methods and types, metrics to be evaluating and considerations for repeatability.

After sifting through many different themes I think I've arrived at one which simple, clean and un-obtrusive. I have completed the content migration and ultimately ended up merging my main website and my blog together into one easily manageable platform. Along the way I did some performance tuning using ApacheBench to see how Ghost does under load and ultimately enabled Apache's mod_cache_disk to prevent node.js from pegging the CPU on my humble VPS.

I look forward to having a clean and easy to use platform at my disposal to document some of my thoughts, tips and tricks in a way that it may be useful to others.

#!/usr/bin/env python
import os
import datetime
import psutil
 
# This script will dump the memory state and write it to mem.log
# if more than 2880 entries are in the log file, it will start removing 
# old ones in order to keep the log size down.
#
# To use this script first install psutil: sudo pip install psutil
# then run: crontab -e
# add this line to the bottom(adjust the path to where the script is): 
# * * * * * cd /home/madmaze/trash; /usr/bin/python /home/madmaze/trash/memoryLogger.py
#
# This will write add a new entry to mem.log every minute and keep 48hrs of records
 
def getUsage():
    m=psutil.virtual_memory()
    s=psutil.swap_memory()
    memkeys=["total", "available", "percent", "used", "free", "active", "inactive", "buffers", "cached"]
    swapkeys=["total", "used", "free", "perc","sin","sout"]
    info={}
    # MEM
    for n,k in enumerate(m):
        if memkeys[n] not in ["percent","active","inactive"]:
            info["mem_"+memkeys[n]]=str(k/1024/1024)+" MB"
 
    # SWAP
    for n,k in enumerate(s):
        if n < 3:
            info["swap_"+memkeys[n]]=str(k/1024/1024)+" MB"
    
    return info
 
 
logfile="mem.log"
f_in=open(logfile,"r")
lines=f_in.readlines()
f_in.close()
 
totalLen=len(lines)
maxlen=2*24*60	# 2days * 24hr * 60m
 
if totalLen > maxlen+100:
    # Crop file to length
    f_out=open(logfile,"w")
    for n,l in enumerate(lines):
        if (totalLen-n)<maxlen:
            f_out.write(l)
else:
    # just append
    f_out=open(logfile,"a")
 
# Actually append the next line of data
f_out.write(str(datetime.datetime.now())+" "+str(getUsage())+"\n")
f_out.close()

Github Gist: https://gist.github.com/madmaze/560cbcf0392aab824820

Judging by the recorded memory usage pattern, it seems that at 3AM EST a series of database intensive cron jobs kicked off at the same time causing mysql's memory foot print to grow until it crashed. Long story short, I spaced the cron jobs out such that each has plenty of time to complete before the next one begins.

I went to consult the Conky Doc (http://conky.sourceforge.net/variables.html) at first I came across:

${if_up <iface>} ${endif}

Which switches conky config blocks based on which interfaces are up.
This is great, but still always displays eth0 and wlan0 as long as the interfaces are enabled, even if they are not connected to anything.
Next I found:

${if_existing /sys/class/net/<iface>/operstate up} ${endif}

This will automatically switch blocks of your config only when an interface's "operstate" is "up". Surprisingly this is different from the ${if_up <iface>} statement and at least for me this produces the expected behavior.

I ended up using a combination of the ${if_up} and ${if_existing} mainly because for temporary interfaces such as usb0 and tun0 the ${if_up} statement works fine and looks cleaner.

Here is my current config:

Network ${hr}
${if_existing /sys/class/net/eth0/operstate up}
eth0
Down ${downspeed eth0} k/s ${alignr}Up ${upspeed eth0} k/s
${downspeedgraph eth0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph eth0 25,100 dddddd ffffff 18}
Total ${totaldown eth0} ${alignr}Total ${totalup eth0}
${endif}${if_existing /sys/class/net/wlan0/operstate up}
wlan0
Down ${downspeed wlan0} k/s ${alignr}Up ${upspeed wlan0} k/s
${downspeedgraph wlan0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph wlan0 25,100 dddddd ffffff 18}
Total ${totaldown wlan0} ${alignr}Total ${totalup wlan0}
${endif}${if_up usb0}
usb0
Down ${downspeed usb0} k/s ${alignr}Up ${upspeed usb0} k/s
${downspeedgraph usb0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph usb0 25,100 dddddd ffffff 18}
Total ${totaldown usb0} ${alignr}Total ${totalup usb0}
${endif}${if_up tun0}
tun0
Down ${downspeed tun0} k/s ${alignr}Up ${upspeed tun0} k/s
${downspeedgraph tun0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph tun0 25,100 dddddd ffffff 18}
Total ${totaldown tun0} ${alignr}Total ${totalup tun0}
${endif}

enjoy.

Outline:

• Setting up a backup server (perhaps)
• Advanced authentication with SSH keys
• Task scheduling with Cron
• Various backup methods (scp, rsync and rdiff-backup)

• Setting up a backup server (perhaps)

  • Ubuntu LTS (12.04/14.04)
    • preferably not VPS, especially not with who you host your other stuff.
    • doesn't need to be powerful, just have a spinning disk or two.
  • Perhaps set up RAID? mdadm is great
    • What is RAID?
  • could be headless

• Advanced authentication with SSH keys

  • What are SSH keys?
    • bit size, Algorithm RSA/DSM
    • Password vs Password-less keys
  • Setting up Keys:
    • {lost? there is a great reference in these github docs}
    • Basics, generate a key: $> ssh-keygen
      • this will use the default options and generate a key
    • More advanced: $> ssh-keygen -t rsa -b 2048 -f ./test -C "this is a test"
      • -t <type> specify the algorithm: rsa1, dsa, ecdsa, rsa
      • -b <bits> this specifies the complexity of key, or the "key size", bigger is better usually.
      • -f <filename> name of the key
      • -C "<comment here>" give it a comment to recognize it.

• Task scheduling with Cron

  • What is cron?
    • cron is a "task scheduler"
    • need to do something every day at 3pm??, this is your tool
  • crontab -e
    • <some activation pattern> <some command here>
    • * * * * * /bin/bash /root/some.script - this will run every minute of every hour of every day of every month
    • @reboot /bin/bash /root/some.script - this will run after every reboot
    • @yearly /bin/bash /root/some.script - this will run every year
    • { need more reference, this page is great }

• Various backup methods (scp, rsync and rdiff-backup)

  • scp - "secure" copy, copies over an ssh connection
    • scp -r <from> <to>
  • rsync - copies only the differences between local and remote copy
    • rsync -av <from> <to>
  • rdiff-backup - only copies the diffs between local and remote copy, stores all versions of diffs.
    • rdiff-backup <from> <to>

Screen has some basic, yet important to remember keyboard shortcuts. The most important one is CTRL+a then tap d. This will disconnect your current session.

To start a new screen, just type screen.

madmaze@spork:~& screen

Your terminal will go blank signaling that you have entered a new terminal session inside of a new screen. Lets start a command like top.

madmaze@spork:~$ 			< you are now inside of a "screen" >
madmaze@spork:~$ top

Then to detach from this screen hit CTRL & a, then tap d.

< hit CTRL+a then d to detach >

[detached from 20041.pts-4.sp0rk]
madmaze@spork:~$

Now to see which screens are running, run screen -ls

madmaze@spork:~$ screen -ls
There is a screen on:
	20041.pts-4.sp0rk	(02/22/2014 09:44:35 PM)	(Detached)
1 Socket in /var/run/screen/S-madmaze.

That means you currently have one screen open and you are detached from this screen. To reconnect, run screen -dr 20041

madmaze@spork:~$ screen -dr 20041

.. and now you should be back to top which had been running in the background while you were detached.

s):  9.8 us,  5.3 sy,  0.0 ni, 84.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   3105556 total,  2364832 used,   740724 free,    36460 buffers
KiB Swap:  2572284 total,   362828 used,  2209456 free,   532236 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 2189 root      20   0 98.6m  23m  11m S   8.0  0.8 492:18.82 Xorg
27255 madmaze   20   0  538m 246m  27m S   6.3  8.1 117:39.77 chrome
 2824 madmaze   20   0  265m  12m 5820 S   3.3  0.4   6:31.73 guake
 8479 madmaze   20   0  518m 103m  20m S   3.0  3.4  63:45.22 chrome
 2893 madmaze   20   0 30724 3236  548 S   1.0  0.1  47:08.06 compton
 9733 madmaze   20   0  318m 115m  26m S   1.0  3.8  12:56.18 chrome  

Easy as that. Also when reconnecting, you don't have to type out the full name of the screen, only enough to differentiate it from other ones. In the case that you just have one screen running, screen -dr without a name will do the trick.

• Lets start by soldering on microphone and speaker jacks:
  

• Clip the two separate jacks into place, you might need to adjust the pins a bit to make them fit in the holes. Then grab your trusty soldering iron and solder up those connections
  

• Next lets solder on the reset button:
  

• Solder on the 2 jumpers next to the mic jack:
  

• Now lets solder 6 Ceramic 47 pf Capacitors marked C5-C10:
  
  

• Locate R1-R4(mine were labeled R3-R6), these should be 10K Ohm Resistors(Brown-Black-Orange):
  
  

• Next find the BS170 MOSFET and thread the leads into the holes associated with T1:
  
  

• Attaching the Arduino headers, for this you need to be careful to attach them straight and square, otherwise you'll have trouble attaching it to your arduino.

• I started with the pins on the side of the reset button, I used some extra header to join the two header sections together and put it into my panavise. First I soldered in only one pin of each header:
  

• Then, since on the other side it is non standard spacing and therefore the trick of joining the two heads together with a third doesn't work. Instead I used that third header as a crossbrace from the one side to the other. Yet again I only soldered in one pin on each header:
  
  
  

• Now before we solder in the rest of the pins, lets double check that everything is square by plugging it onto an actual arduino:
  

• If everything checks out, go and solder in all the pins. If not, now you have a chance of reheating and reseating the headers:
  

• Before we continue on to the other easy parts, lets solder on the surface mount headers for the sim900 and sim908 modules.

• For this I applied some flux paste onto the pads, then aligned one of the headers and soldered on one end pin, making sure it was straight:
  

• Then follow by carefully soldering all the other pads. Make sure its clean and none of the solder bridges two pads.

• Next lets solder on the rest of the through-hole headers, again make sure that these are straight as this is where the regular size SIM900 breakout will mate:
  

• Now lets tackle the unlabled capacitor just above the SIM908, according to the data sheet it should be marked CRTC. Clip it in place, making sure the negative lead, has a tiny minus sign stamped on it, faces towards the center of the board:
  

• Then continue by adding more capacitors, this time C1, C3 and C12. These should all be 100nF, probably labled something like "104":
  

• Next locate and solder on C2, C4 and C11. C2 is a 470uF Capacitor. C4 and C11 are both 220uF capacitors. Make sure to get the polarity right, the longer lead has to go through the positive marked hole:
  

• Add the 1N4007 diode to D1:
  

• Finally lets put on the big MOSFET and heatsink, first approximate the bends the need to be done to the MOSFET and then thread the screw through the MOSFET and the heatsink and screw it down. Then solder the connections:
  

• Almost done, to finish everything off, lets add headers for the Battery plug(its a surface mount, pain in the ass. Also optional depending on whether you're plan on using a battery or not), the Vext and the CHRG headers. I failed at attaching the Battery plug, so I guess I'll be running on external power =)
  

Getting Started...

Installing pyGASP:

Easiest way to install pyGASP is using pip or a similar tool.

$> sudo pip install pygasp

Documentation:

The official documentation can be found here: http://pythonhosted.org/PyGASP/
and here: https://pypi.python.org/pypi/PyGASP

The docstring generated documentation is not too bad and certainly gives you the basics. The README on the other hand is a out of date. Following are my notes on evaluating it.

pywt vs pyGASP; Benchmarking the 2D wavelet transform:

I am only going to compare dwt2 between these two packages. I, perhaps wrongly, assume other comparisons would yield similar results.

import numpy as np
import scipy.misc
import pylab
from datetime import datetime as dt

import pywt
import pygasp.dwt.dwt as pygaspDWT
import pygasp.dwt.dwtCuda as pygaspDWTgpu

def show(data):
    pylab.jet()
    pylab.imshow(data)
    pylab.colorbar()
    pylab.show()
	pylab.clf()

# Lets get an image to play with.
img = scipy.misc.lena().astype(np.float32)

# pywt
s = dt.now()
res_pywt = pywt.dwt2(img, "haar", "zpd")
print "pywt took:", dt.now()-s

# pygasp CPU version
s = dt.now()
res_gasp = pygaspDWT.dwt2(img, "haar", "zpd")
print "pygaspCPU took:", dt.now()-s

# pygasp GPU version
s = dt.now()
res_gaspGPU = pygaspDWTgpu.dwt2(img, "haar", "zpd")
print "pygaspGPU took:", dt.now()-s

# if you want to view the results
#show(res_pywt[0])
#show(res_gasp[0])
#show(res_gaspGPU[0])

Now that we have a basic comparison, lets grab a larger image and try it again:

$> wget http://i.imgur.com/CjJL2wG.jpg -O largeTest.jpg

And add the following:

# add this at the top
from PIL import Image
.
.
.
# replace the lena image with this:
imgObj = Image.open("largeTest.jpg")
img = np.array(imgObj)

Sadly the results with the larger image are quite disappointing. I had hoped that the pyGASP GPU code would be at least as fast as the CPU-based pywt.

$> python pygaspTest.py
pywt took: 0:00:01.412802
pygaspCPU took: 0:01:34.589889
pygaspGPU took: 0:00:06.963826

Even though the pyGASP GPU version is 13.5 times faster than its own CPU equivalent, the pywt CPU version is another ~5 times faster!! Perhaps it is not yet prime time for this library, but it might be a starting point to get a truely GPU accelerated version going. These tests were performed on a Nvidia Tesla K20c. For now I will have to venture on to find another faster solution, but I might come back to this and work on optimizing the it to suit my needs. Sadly there is no public code repo available.

Edit: Looks like pyGASP is related to this paper

Installing HP Array admin tools

Download bootstrap from HP

test@ml350:~$ wget http://downloads.linux.hp.com/SDR/downloads/bootstrap.sh

We need to fix HP's bootstrap script to point to the current HP download url. Then we add HP's repo. Since HP no longer supports debian/ubuntu, we must override the distro version detection, therefore we specify -r oneiric, 11.10 being the latest supported version. Then we add the repo's GPG key.

test@ml350:~$ sed -i 's/blofly.usa/downloads.linux/g' ./bootstrap.sh
test@ml350:~$ chmod +x bootstrap.sh
test@ml350:~$ sudo ./bootstrap.sh -r oneiric ProLiantSupportPack
test@ml350:~$ wget -qO- http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/GPG-KEY-ProLiantSupportPack |  sudo apt-key add -

Finally we can update and install hpacu-cli and hp-health

test@ml350:~$ sudo apt-get update
test@ml350:~$ sudo apt-get install hpacucli hp-health

Fixing "Error: No controllers detected."

Now if you are like me and running a semi modern system(Kernel 3.0+), you will get the following error message:

test@ml350:~$ sudo hpacucli ctrl all show config
Error: No controllers detected.

To fix this you can either compile and run uname26 or backport a newer version of hpacucli from RHEL/SUSE to Debian/Ubuntu. For further reference, check here.

Option #1 (Option 2 is recommended):

Compile uname26 to pretend we are in an older kernel

test@ml350:~$ mkdir uname26
test@ml350:~$ cd uname26
test@ml350:~/uname26$ wget http://mirror.linux.org.au/linux/kernel/people/ak/uname26/Makefile
test@ml350:~/uname26$ wget http://mirror.linux.org.au/linux/kernel/people/ak/uname26/uname26.c
test@ml350:~/uname26$ make
test@ml350:~/uname26$ ./uname26

Now you can run it with uname26:

test@ml350:~/uname26$ sudo ./uname26 hpacucli ctrl all show config

Smart Array E200i in Slot 0 (Embedded)    (sn: QXXXXXXXX)

 array A (SAS, Unused Space: 0 MB)

  logicaldrive 1 (341.7 GB, RAID 5, OK)

  physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
  physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
  physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
  physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
  physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 72 GB, OK)
  physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 72 GB, OK)

Option #2:

Backport the newer version of hpacucli from SUSE, since debian and ubuntu are no longer supported.

We use alien to convert the SUSE RPM into a DEB and then install it.

test@ml350:~$ wget http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/SuSE/11.2/x86_64/9.10/hpacucli-9.10-22.0.x86_64.rpm
test@ml350:~$ sudo apt-get install alien
test@ml350:~$ sudo alien hpacucli-9.10-22.0.x86_64.rpm
test@ml350:~$ sudo dpkg -i hpacucli_9.10-23_amd64.deb

And now it works without using uname26:

test@ml350:~$ sudo hpacucli ctrl all show config

Smart Array E200i in Slot 0 (Embedded)    (sn: QXXXXXXXX)

 array A (SAS, Unused Space: 0 MB)

  logicaldrive 1 (341.7 GB, RAID 5, OK)

  physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
  physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
  physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
  physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
  physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 72 GB, OK)
  physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 72 GB, OK)

Cheers & enjoy =).

If your are like me, you may often find yourself messing with VPSes and other servers. All too often these are provisioned with very little RAM, 512MB or less.
When your applications/processes run out of RAM, most of them become relatively unhappy and begin to behave erradically and finally crash.
In many cases you only need more RAM temporarily while you run some memory intensive command or perhaps you just want a safty net. SWAP space offers you this flexibillity by tapping into your hard drive's space. Imagine it as emergency RAM which lives on your hard drive in the form of a file or partition. In the event that more RAM is need than available, the OS will automatically tap that file instead of crashing your programs. The only down side is that SWAP is many times slower than your physical RAM and will therefore affect your overall performance.

Creating a SWAP file

Log into your machine over ssh or locally, then we will create a 512mb file full of zeros using dd and finally we will make it only read-writable by root.

sudo dd if=/dev/zero of=/swapfile bs=1024 count=524288
sudo chown root:root /swapfile
sudo chmod 600 /swapfile

now we will turn this file into a swapfile format using mkswap:

sudo mkswap /swapfile

Swapping the swapfile on:

sudo swapon /swapfile

To check whether it worked or not check the output of free -m for the line specifying swap:

derp@box:~$ free -m
         total       used       free     shared    buffers     cached
Mem:           491        484          6          0         21        169
-/+ buffers/cache:        293        198
Swap:          511          0        511

In the above example it looks like everything went well. Now we have to make it permanent, otherwise at every reboot you would need to run the swapon command again. This is where our friend fstab comes in.

sudo nano /etc/fstab

Add the following line at the bottom:

/swapfile       swap    swap    defaults        0       0

This will mount the swap file at boot with default options.

Cheers and enjoy your swapping.

Also sidenote: VPSes and other servers with fast disks will do much better when swapping. I can recommend Digital Ocean, all their VPSes are backed by SSD, which makes running in swap very quick.