<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[linux - Matthias Lee - Musings on Software and Performance Engineering]]></title><description><![CDATA[Matthias Lee is a Software Performance Engineer, Technical Lead and Computer Science PhD. Currently a Principal Performance Engineer at Appian.]]></description><link>https://matthiaslee.com/</link><image><url>https://matthiaslee.com/favicon.png</url><title>linux - Matthias Lee - Musings on Software and Performance Engineering</title><link>https://matthiaslee.com/</link></image><generator>Ghost 2.14</generator><lastBuildDate>Thu, 20 Nov 2025 18:19:58 GMT</lastBuildDate><atom:link href="https://matthiaslee.com/tag/linux-2/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Duplicati: Self-hosted CrashPlan Alternative]]></title><description><![CDATA[<p>For the past few years I have been using CrashPlan to do incremental encrypted backups for all of my family's various computers. CrashPlan offered <em>free</em> peer-to-peer backups, which allowed me to just specify one of my ZFS disk boxes as the target, then <em>set it and forget it</em>. This has</p>]]></description><link>https://matthiaslee.com/self-hosted-crash-plan-alternative/</link><guid isPermaLink="false">5b4a06a626f9be0001ff6a5d</guid><category><![CDATA[duplicati]]></category><category><![CDATA[backup]]></category><category><![CDATA[linux]]></category><category><![CDATA[crashplan]]></category><dc:creator><![CDATA[Matthias A. Lee]]></dc:creator><pubDate>Thu, 26 Jul 2018 01:27:55 GMT</pubDate><content:encoded><![CDATA[<p>For the past few years I have been using CrashPlan to do incremental encrypted backups for all of my family's various computers. CrashPlan offered <em>free</em> peer-to-peer backups, which allowed me to just specify one of my ZFS disk boxes as the target, then <em>set it and forget it</em>. This has been working spectacularly well, especially considering the wide variety of operating systems I needed to &quot;support&quot;: Linux (Desktop/Server), Windows and Mac.</p>
<p><img src="https://matthiaslee.com/content/images/2018/07/crashplan.jpg" alt="CrashPlan Client/Server User Interface"></p>
<p>Unfortunately in August 2017, CrashPlan announced they will discontinue their consumer offering in order to refocus on small-businesses, which I'm sure is a much more profitable market. The announcement email has been sitting <em>starred</em> in my inbox since that day and I've been dreading to find the next solutions. I remembered, last time I looked around, CrashPlan was more or less the only self-hosted choice that worked across all OSs and offered incremental encrypted backups. More recently I came across the new <a href="https://www.duplicati.com/">Duplicati</a>, which has seen a major revamp since the last time I had looked at it, now supporting all sorts of storage backends, encryption, incremental backups and a variety of retention policies  as well as a slick new web-based UI.</p>
<p><img src="https://matthiaslee.com/content/images/2018/07/duplicati.jpg" alt="Duplicati Web Interface"></p>
<p>Below I will describe how to setup duplicati to use a central linux server as the backend to store all of your files.</p>
<h2 id="setupandinstallduplicati">Setup and Install Duplicati</h2>
<p>On the client machine, which will be backed up, go to <a href="https://www.duplicati.com/download">duplicati.com/download</a> and grab the latest installer for your operating system and follow the default installation. Below are the directions I used on my Ubuntu Laptop</p>
<pre><code>user@box:~$ wget https://updates.duplicati.com/beta/duplicati_2.0.3.3-1_all.deb
user@box:~$ sudo dpkg -i duplicati_2.0.3.3-1_all.deb
</code></pre>
<p>then start <code>duplicati</code> and continue in the web interface at <a href="http://localhost:8200/ngax/index.html#/">localhost:8200/ngax/index.html</a>.</p>
<h2 id="addingabackuplocation">Adding a Backup location</h2>
<p>I usually like to keep all of my data on my own servers so here we will configure backing up via SSH, making sure to setup ssh-keys and a separate backup user on my disk box.</p>
<ol>
<li>
<p><strong>Getting Started</strong> Select &quot;+ Add Backup&quot;, then &quot;Configure New Backup&quot;, then &quot;next&quot;<br>
<img src="https://matthiaslee.com/content/images/2018/07/duplicati_start_backup.png" alt="Duplicati Getting Started"></p>
</li>
<li>
<p><strong>Encryption</strong> A great feature of duplicati is the encryption of backups, to make good use of it, generate a password and store it in your <em>password manager</em>, I prefer KeePass and LastPass.<br>
<img src="https://matthiaslee.com/content/images/2018/07/homedir.png" alt="Setup Encryption"></p>
</li>
<li>
<p><strong>Destination</strong> Next we select where the backups will be stored, this could be an external drive attached to your computer or a remote machine via SSH, FTP, WebDav or even S3-like storage. For the purpose of this tutorial we will go with SSH.</p>
</li>
</ol>
<p><img src="https://matthiaslee.com/content/images/2018/07/backup-settings.png" alt="Set your Backup Destination"></p>
<ul>
<li><strong>Add SSH Key</strong> When using SSH, I recommend using an SSH key for security, especially if you use the default SSH port. Under Advanced Settings select <strong>ssh-keyfile</strong>, if your key has a password, enter this into the <strong>Password</strong> field above.</li>
</ul>
<p><img src="https://matthiaslee.com/content/images/2018/07/test-connection.png" alt="Specify SSH Key"></p>
<ul>
<li><strong>Test your connection</strong> Now hit <strong>Test Connection</strong> to make sure everything works properly. You'll be asked whether you accept the Host-key, select <strong>Yes</strong>, then go <strong>Next</strong></li>
</ul>
<p><img src="https://matthiaslee.com/content/images/2018/07/add-host-key.png" alt="Add SSH Host-key"></p>
<ol start="4">
<li><strong>Configure Source</strong> Here we select the files we wish yo backup. Generally on a linux machine I backup <code>~/</code> and exclude some of the <em>dot-files</em>.</li>
</ol>
<p><img src="https://matthiaslee.com/content/images/2018/07/source-files.png" alt="Selecting Files to be backed up"></p>
<ol start="5">
<li><strong>Backup Schedule</strong> Generally I don't need backup every day, so go with Weekly backup.</li>
</ol>
<p><img src="https://matthiaslee.com/content/images/2018/07/schedule-7day.png" alt="Duplicati Backup Schedule"></p>
<ol start="6">
<li><strong>General Settings</strong> These final settings are important.</li>
</ol>
<ul>
<li><strong>Volume Size</strong> controls the size of chunks that will be pushed to your backup system and will therefore control how many file will land on your backup server. I've found 50MB to work well for me.</li>
<li><strong>blocksize</strong> controls the size of the blocks which will be hashed, i set this to 1MB, cutting down on the size of the database duplicati has to maintain.</li>
<li><strong>Backup Retention</strong> Generally it is most useful to thin-out older backups in order to save space. I set this to <strong>Smart Backup Retention</strong>.</li>
</ul>
<p>Play with these settings to find what is most optimal for your setup, it will depend on your upload speed, the backend server as well as the compute capability of your client device.</p>
<p><img src="https://matthiaslee.com/content/images/2018/07/size-options.png" alt="Genral settings controling Volume Size, Blocksize and Retention."></p>
<p><strong>All Done!</strong><br>
Now, a word of wisdom, after you get this setup, make sure you can successfully go through a full backup-restore cycle to work any kinks you may encounter.</p>
<p><img src="https://matthiaslee.com/content/images/2018/07/action-shot.png" alt="Duplicati in Action!"></p>
]]></content:encoded></item><item><title><![CDATA[No network access after Ubuntu 14.04->16.04->18.04 upgrade]]></title><description><![CDATA[<p>Recently I was tending to my fleet of personal servers and I ran into a problem which essentially took my system offline and with it a number of websites I host.</p>
<p>After the upgrade my system did not automatically start eth0 with its assigned static IP address, only the loopback</p>]]></description><link>https://matthiaslee.com/no-network-access-after-ubuntu-14-04-16-04-18-04-upgrade/</link><guid isPermaLink="false">5b3e5ba8ede43600016b7830</guid><category><![CDATA[linux]]></category><category><![CDATA[networking]]></category><category><![CDATA[netplan]]></category><category><![CDATA[networkd]]></category><dc:creator><![CDATA[Matthias A. Lee]]></dc:creator><pubDate>Thu, 05 Jul 2018 19:29:42 GMT</pubDate><content:encoded><![CDATA[<p>Recently I was tending to my fleet of personal servers and I ran into a problem which essentially took my system offline and with it a number of websites I host.</p>
<p>After the upgrade my system did not automatically start eth0 with its assigned static IP address, only the loopback device was configured. Thankfully DigitalOcean provides easy console access. So i started some debugging:</p>
<pre><code>m@box:~$ ifconfig
lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

m@box:~$ cat ifconfig-a.log 
eth0: flags=4098&lt;BROADCAST,MULTICAST&gt;  mtu 1500
        ether 04:01:01:8a:1b:01  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
</code></pre>
<p>Turns out the root cause was no <code>ifup</code> or <code>ifdown</code>, which were both scripts required by <code>/etc/init.d/networking</code> to bring up networking</p>
<pre><code>m@box:~$ ifup eth0
-bash: ifup: command not found
m@box:~$ ifdown eth0
-bash: ifdown: command not found
</code></pre>
<p>Excerpt from <code>/etc/init.d/networking</code></p>
<pre><code>m@box:~$ grep -in &quot;ifup\|ifdown&quot; /etc/init.d/networking 
3:# Provides:          networking ifupdown
17:[ -x /sbin/ifup ] || exit 0
18:[ -x /sbin/ifdown ] || exit 0
101:ifup_hotplug () {
115:		ifup $ifaces &quot;$@&quot; || true
141:	if ifup -a $exclusions $verbose &amp;&amp; ifup_hotplug $exclusions $verbose
157:	if ifdown -a --exclude=lo $verbose; then
172:	ifdown -a --exclude=lo $verbose || true
173:	if ifup --exclude=lo $state $verbose ; then
188:	ifdown -a --exclude=lo $verbose || true
191:	if ifup -a --exclude=lo $exclusions $verbose &amp;&amp; ifup_hotplug $exclusions $verbose
</code></pre>
<h3 id="tempfixtogetbackontotheinternettoactuallyinstallnetworkdnetplan">Temp fix to get back onto the internet to actually install networkd/netplan:</h3>
<pre><code>m@box:~$ sudo ifconfig eth0 up
# xx.xx.xx.xx/24 is your static IP
m@box:~$ sudo ip addr add xx.xx.xx.xx/24 dev eth0
# xx.xx.xx.1 is your gateway
m@box:~$ sudo ip route add default via xx.xx.xx.1 dev eth0
</code></pre>
<h3 id="longtermfixinstallnetplanconfigureittomanagenetworkd">long-term fix, install netplan, configure it to manage networkd:</h3>
<pre><code>m@box:~$ sudo apt-get install netplan.io
</code></pre>
<p>We configure netplan(<code>/etc/netplan/01-netcfg.yaml</code>):</p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
   eth0:
    dhcp4: no
    dhcp6: no
    addresses: [xx.xx.xx.xx/24]
    gateway4: xx.xx.xx.1
    nameservers:
     addresses: [8.8.8.8, 8.8.4.4]
</code></pre>
<p>next we apply the new changes and make sure it gives no errors:</p>
<pre><code>m@box:~$ sudo netplan --debug apply
** (generate:1883): DEBUG: 19:19:36.239: Processing input file //etc/netplan/01-netcfg.yaml..
** (generate:1883): DEBUG: 19:19:36.240: starting new processing pass
** (generate:1883): DEBUG: 19:19:36.240: eth0: setting default backend to 1
** (generate:1883): DEBUG: 19:19:36.244: Generating output files..
** (generate:1883): DEBUG: 19:19:36.244: NetworkManager: definition eth0 is not for us (backend 1)
DEBUG:netplan generated networkd configuration exists, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:device eth0 operstate is up, not replugging
DEBUG:netplan triggering .link rules for eth0
DEBUG:device lo operstate is unknown, not replugging
DEBUG:netplan triggering .link rules for lo
</code></pre>
<p>now the last step is to reboot and make sure the new configuration persists.</p>
<p>And viola, after a reboot everything stayed in place:</p>
<pre><code>m@box:~$ ifconfig
eth0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet xx.xx.xx.xx  netmask 255.255.255.0  broadcast xx.xx.xx.255
        inet6 xx::xx:xx:xx:xx  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 04:01:xx:xx:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 654  bytes 69420 (69.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 517  bytes 108478 (108.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
</code></pre>
]]></content:encoded></item><item><title><![CDATA[Dynamically changing conky network interface]]></title><description><![CDATA[<p>I use conky on many of my machines, most desktop machines will have one or two network interfaces which I want to always monitor, but my laptop I often switch between many different network interfaces, (eth0, wlan0, usb0 and tun0), and I really like Conky's downspeedgraph and upspeedgraph. The issue</p>]]></description><link>https://matthiaslee.com/dynamically-changing-conky-network-interface/</link><guid isPermaLink="false">5a0a60d882e47c00018dabe3</guid><category><![CDATA[conky]]></category><category><![CDATA[network interface]]></category><category><![CDATA[dynamic]]></category><category><![CDATA[linux]]></category><dc:creator><![CDATA[Matthias A. Lee]]></dc:creator><pubDate>Sun, 03 Aug 2014 17:28:42 GMT</pubDate><content:encoded><![CDATA[<p>I use conky on many of my machines, most desktop machines will have one or two network interfaces which I want to always monitor, but my laptop I often switch between many different network interfaces, (eth0, wlan0, usb0 and tun0), and I really like Conky's downspeedgraph and upspeedgraph. The issue is I dont want to be displaying all interfaces at all times. I only want to display the interfaces I am currently using.</p>
<p>I went to consult the Conky Doc (<a href="http://conky.sourceforge.net/variables.html">http://conky.sourceforge.net/variables.html</a>) at first I came across:</p>
<pre><code>${if_up &lt;iface&gt;} ${endif}
</code></pre>
<p>Which switches conky config blocks based on which interfaces are up.<br>
This is great, but still always displays eth0 and wlan0 as long as the interfaces are enabled, even if they are not connected to anything.<br>
Next I found:</p>
<pre><code>${if_existing /sys/class/net/&lt;iface&gt;/operstate up} ${endif}
</code></pre>
<p>This will automatically switch blocks of your config only when an interface's &quot;operstate&quot; is &quot;up&quot;. Surprisingly this is different from the <code>${if_up &lt;iface&gt;}</code> statement and at least for me this produces the expected behavior.</p>
<p>I ended up using a combination of the <code>${if_up}</code> and <code>${if_existing}</code> mainly because for temporary interfaces such as usb0 and tun0 the  <code>${if_up}</code> statement works fine and looks cleaner.</p>
<h4 id="hereismycurrentconfig">Here is my current config:</h4>
<pre><code>Network ${hr}
${if_existing /sys/class/net/eth0/operstate up}
eth0
Down ${downspeed eth0} k/s ${alignr}Up ${upspeed eth0} k/s
${downspeedgraph eth0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph eth0 25,100 dddddd ffffff 18}
Total ${totaldown eth0} ${alignr}Total ${totalup eth0}
${endif}${if_existing /sys/class/net/wlan0/operstate up}
wlan0
Down ${downspeed wlan0} k/s ${alignr}Up ${upspeed wlan0} k/s
${downspeedgraph wlan0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph wlan0 25,100 dddddd ffffff 18}
Total ${totaldown wlan0} ${alignr}Total ${totalup wlan0}
${endif}${if_up usb0}
usb0
Down ${downspeed usb0} k/s ${alignr}Up ${upspeed usb0} k/s
${downspeedgraph usb0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph usb0 25,100 dddddd ffffff 18}
Total ${totaldown usb0} ${alignr}Total ${totalup usb0}
${endif}${if_up tun0}
tun0
Down ${downspeed tun0} k/s ${alignr}Up ${upspeed tun0} k/s
${downspeedgraph tun0 25,100 dddddd ffffff 150} ${alignr}${upspeedgraph tun0 25,100 dddddd ffffff 18}
Total ${totaldown tun0} ${alignr}Total ${totalup tun0}
${endif}
</code></pre>
<p>enjoy.</p>
]]></content:encoded></item></channel></rss>