Blog

Der Marktmacher by Michael Ridpath

2023-09-15

I really liked the book and read the 450 pages in less than 3 days. It’s entertaining but also somewhat detailed (for a lay person) when it talks about investment banking.

Das Buch stellt ein paar mal grundlegende Investment Banking Konzepte klar und leicht verständlich dar.

Zum Beispiel, Seite 155: Leerverkauf

“Da stimmt doch was nicht”, sagte Dave. “Das ist eine Vier-Milliarde-Dollar-Emission, und wir wissen, dass davon drei Milliarden bei Kunden sind, die ganz bestimmt nicht verkaufen. Bleibt noch eine Milliarde, von der wir den Löwenanteil haben. Also woher hat Bloomfield Weiss seine Bonds?”

“Die gehen short”, sagte Pedro. “Wenn die auf so vielen Anleihen säßen, wüsste ich das doch.”

“Also leihen sie sich diese”, sagte Ricardo. “Von wem, frage ich mich?”

Alle schwiegen. Bloomfield Weiss überschwemmte den Markt mit Bonds, die sie nicht hatten. Pedro vermutete, dass sie Leerverkäufe vornahmen, das heißt, sich die Bonds von einem freundlich gesinnten Inhaber ausliehen, um sie zu verkaufen. Wenn dieser Inhaber seine Bonds zurückforderte, musste Bloomfield Weiss sie natürlich auf dem Markt kaufen. Bloomberg Weiss setzte darauf, dass der Kurs bis dahin so weit gefallen sein würde, dass die Firma einen satten Profit einstreichen konnte. Bis dahin, so hoffte Bloomfield Weiss, würde Dekker Ward aus dem Markt gedrängt sein.

Persönliche Reflektion von Nick nach einer Geschäftsnacht, wissenschon. Mit einem Essen in einem noblem Restaurant, danach zu Hause bei Eduardo mit Frauen und weißem Pulver. Nick saß auf einer Bank und blickte auf London, wo auch seine Arbeitsstelle Canary Wharf lag.

Seite 198

Da wirbelten nun die Dekker-Ward-Leute ganz besonders hektisch, alle um den Nachweis bemüht, dass die vergangene Nacht ihre Arbeitskraft nicht beeinträchtigt hatte. Sie würden viel sagende Blicke austauschen, ihren Kunden vorlügen, was Mexiko doch für ein wunderbares Land sei, die Aufträge notieren und die Profite einstreichen.

Ich ließ die letzten Wochen Revue passieren. Den Favela-Deal, die Geldwäsche, Daves Entlassung, meinen Vertrauensbruch gegenüber Wojtek, das erbärmliche Geschehen der letzten Nacht. Jeden einzelnen dieser Vorfälle hätte ich verkraften können, aber zusammen machten sie mich elend.

Ich war nicht der richtige Mann für diesen Job. Eine Zeit lang konnte ich mir das Gegenteil einreden, aber nicht sehr lange. Oder ich musste mich verändern, wie sich Jamie verändert hatte. So verändern, dass ich leichten Herzens lügen konnte, ignorieren, was ignoriert werden musste, tun, was getan werden musste. Wenn mein Gewissen das nicht verkraften würde, dann musste ich eben auch mein Gewissen verändern.

Oder kündigen.

War das Feigheit vor dem Fein? Konnte ich die wirkliche Welt nicht ertragen? Die Geschäftswelt?

Das glaubte ich nicht, ehrlich nicht. Gewiss, der Angriff am Strand von Ipanema hatte mich erschreckt. Aber ich war mit sicher, dass das keinerlei Einfluss auf mein Urteilsvermögen hatte. Ich musste mich mit den Tatsachen abfinden und einsehen, dass es ein Fehler gewesen war, bei Dekker Ward anzufangen. Es war ein totaler Misserfolg. Zwar gebe ich Fehler nur höchst ungern zu, mein Stolz wehrt sich dagegen, aber in diesem Fall blieb mir schlicht und ergreifend keine andere Wahl.

Ricardo würde sagen, ein guter Trader weiß, wann er seine Verluste realisieren muss. Der Zeitpunkt war gekommen.

Eine höchst amüsante Stelle zu Mergers & Acquisitions, als Nick aus Rache Bloomfield Weiss mit Informationen, die er von seinem Freund Jamie—der noch bei Dekker Ward arbeitet—erfahren hat, dazu animiert, doch Dekker Ward zu übernehmen, weil sie momentan nicht besonders gut dastehen.

Seite 327

Und diese Burschen waren kaum weniger gefährlich als die Mafia. Bloomfield Weiss’ Aggressivität im Bereich der Unternehmensübernahmen wie auf anderen Gebieten war legendär. Diese beiden waren höchstpersönlich an der Zerschlagung von Dutzenden Unternehmen in aller Welt beteiligt gewesen. Die offizielle Bezeichnung lautet M&A, Mergers and Acquisitions, also Fusionen und Übernahmen, aber weniger offizielle Bezeichnungen vermitteln einen besseren Eindruck vom tatsächlichen Geschehen: Personalabbau, Shareholder value, oder Wertezuwachs für Aktionäre, Abstoßen von Randaktivitäten, Geld aus dem Unternehmen quetschen. Und dann gibt es noch eine Reihe von Wortprägungen, die sich mit einem zweiten Aspekt des Prozesses beschäftigen: goldener Fallschirm, das heißt, die großzügige Abfindung für leitende Angestellte nach der feindlichen Übernahme von deren Unternehmen, Anreizsystem und vor allem das kleine Wörtchen Provision.

Wrong Assumptions with irssi

2022-09-13

Today I connected back to my irc bouncer znc with the terminal irc client irssi. I had the configuration with my self-signed certificate and certificate pinning already figured out in the past, but that configuration file was on a different computer. And I’m currently travelling, so I didn’t have access to it and, more importantly, I did not save the configuration to my dotfiles or anywhere else where I could easily look it up. However, I remember that I had some issues getting it to work, but in the end I got there.

So I had to do it from scratch, but I thought it would be easy because I’ve already done it. So how hard could it be?

Well, turns out, it took me about an hour again because I had a wrong assumption about one configuration option in irssi.

A configuration block in irssi for a network called libera-znc may look like the following:

{
    address = "mybouncer.server.net";
    chatnet = "libera-znc";
    port = "1234";
    passowrd = "znc-user/znc-network-name:znc-user-password";
    use_tls = "yes";
    tls_verify = "no";
    tls_pinned_cert = "12:34:56:78:90:59:0F:28:0F:48:56:FF:51:F9:D4:8C:0B:94:BC:BF:2F:D1:0C:25:D4:E9:AA:BB:CC:DD:EE:FF";
    autoconnect = "yes";
}

Most options should be self-explanatory. So, what was my wrong assumption?

My znc server uses a self-signed certificate, so I can’t use the option tls_verify, because the certificate is self-signed and therefore cannot be verified by any CA out there, e.g. Let’s Encrypt. Hence, I removed the option tls_verify from the configuration thinking that this would equal to tls_verify = "no".

However, I want to make sure that I always connect to my server (and to prevent MITM attacks), so I generated the sha256 fingerprint of the certificate the znc service is using with OpenSSL:

openssl x509 -fingerprint -sha256 -noout -in znc.pem

After verifying a thousand times that my generated certificate fingerprint is actually correct, I took a step back and thought about it again. I tried to remember what the problem was and I did remember that it was something stupid and simple. After staring at the configuration for a while, I noticed in another network configuration the tls_verify = "yes"; option again. That’s when it hit me. I put tls_verify = "no"; back into my network configuration block and it just worked.

So, to remember this and to write it down somewhere, I created this short blog post.

I also looked through the irssi documentation and could not find the documented default value of the option tls_verify. However, when looking through the changelog, I found the following bullet point:

-tls_verify is now enabled by default (#1170, an#18, #1309, an#23, #1343, #1351) This may cause an ugly display of notls_verify in the output of /SERVER LIST, even on plain-text connection, on old configs. Kindly remove the “tls_verify = “no”;” entries from your config file manually.

Der Schneesturm - Sorokin

2022-08-12

Es herrscht ein Schneesturm. Der Dokter will unbedingt zu einem Dorf, um dort Leute zu impfen. Allerdings gibt es keine Pferde mehr in dem Ort und seine Pferde sind total ausgepowered. Kosma, der normalerweise Brot ausliefert, nimmt sich der Aufgabe an und sie ziehen mit den 50 Minipferden los. die Kufe seines Schlittens bricht, als sie auf eine seltsame Pyramide fahren. Der Kutscher will umkehren um den Schaden zu reparieren, aber der Doktor möchte unbedingt weiter. Sie durchleben einige Strapazen, wie den Berg und die Mulde, wobei sie den Pferdis helfen müssen. An einem Punkt treffen sie Leute mit Hochtechnologie. Dort probiert der Doktor eine neue Droge. Diese ist in Pyramidenform, wodurch im klar wird, auf was sie da gefahren sind und welcher Wert dort im Schnee liegen muss. Die Droge versetzt den Doktor in Hochstimmung und sie fahren mitten im Schneesturm weiter ohne abzuwarten. Dann fahren sie einem im Eis eingefrorenen Riesen ins Nasenloch mit der gleichen bereits gebrochenen Kufe. Die Nase wird abgehackt um die Kufe wieder rauszubekommen. Die ist allerdings kaputt und somit macht sich der Kutscher Kosma ans Werk eines behelfsmäßige Kufe zu basteln. Währenddessen ist es ziemlich kalt geworden, was Kosma durch die Arbeit nicht wirklich merkt. Der Doktor allerdings ist bis auf die Knochen durchgefroren, sodass Kosma erst noch ein Feuer macht, damit sich der Doktor aufwärmen kann. Irgendwann tauchen auch Wölfe auf, wodurch die “Pferdis” in eine Schreckstarre verfallen, die auch anhält, nachdem der Doktor die Wölfe mit der Pistole vertrieben hat (das könnte auch vor dem Nasenloch und dem Riesen gewesen sein). Ich glaube, sie hatten sich wieder festgefahren, als der Doktor extrem wütend ist und seine Koffer packt und geradeaus losstapft um zu seinem Ziel zu kommen. Das geht allerdings schief und er kommt gerade noch so wieder züruck zum Kutscher, der sich mit seinen Wärme spendenden Pferdis verkrochen hat, um bis zum Morgengrauen zu warten, damit der Weg sichtbar wird. Der Doktor verzieht sich mit Kosma. Allerdings platzt der Behälter in dem sie liegen und Kosma zieht es nun am linken Schulterblatt. Der ist aber zu müde, um sich nochmal zu bewegen und der Doktor nimmt sehr viel Platz weg, sodass er so liegen bleibt. Am nächsten morgen werden sie von Chinesen gefunden. Der Doktor spürt seine Beine nicht und kann kaum reden. Kosma ist allerdings erfroren.

Benchmarking on the Helios4

2021-11-08

If you don’t know about the Helios4, you can read about it here.

First of all, during benchmarking a lot of things can go wrong (I really recommend watching these 5 mintues of a lightning talk about benchmarking). Hence, take the results with a grain of salt. While I’m not doing any fancy calculations here, I am still not an expert. During the tests the Helios was always idle, or, at least, I did not do anything else. For instance, it could have happened that a cronjob was run during a benchmark, but I don’t think that this should have a high impact. Further, I was running the tests (with bonnie) multiple times.

The Helios4 has support for four drives but I use it currently with 2x4TB drives.

At the end of this post, you’ll find a table summarizing the results.

Running OMV with EXT4 on LVM on RAID 1

For the first few tests, I actually installed OpenMediaVault (Helios4 Wiki) as I never used it before and wanted to try it out. I literally followed the instructions on the Official Kobol wiki which meant

setting up my RAID 1 (mirror)
installing the LVM plugin and creating the volumes
creating the EXT4 filesystem

I installed and activated SMB as well and after I was able to connect to it finally (permission things…), I used dd to push some data over the network to get a feeling for the speed.

It’s important to note that I did not change any configration or tried to make any optimizations. This was run with the default settings.

SMB Performance

As you can see in the command, I wanted to copy 10 GB originally, but aborted at 3.1 GB, which took 160s at a rate of about 20 MB/s. Using the same command to read the written data again resulted in about the same speed.

w=20MB/s, r=20MB/s

╭─max@host /run/user/1000/gvfs/smb-share:server=helios4,share=data/test
╰─$ dd if=/dev/zero of=test.img bs=1M count=10000 status=progress
3283091456 bytes (3,3 GB, 3,1 GiB) copied, 160 s, 20,5 MB/s^C
3132+0 records in
3132+0 records out
3284140032 bytes (3,3 GB, 3,1 GiB) copied, 160,051 s, 20,5 MB/s

╭─max@host /run/user/1000/gvfs/smb-share:server=helios4,share=data/test
╰─$ dd if=test.img of=/dev/null bs=1M status=progress
3275751424 bytes (3,3 GB, 3,1 GiB) copied, 166 s, 19,7 MB/s
3132+0 records in
3132+0 records out
3284140032 bytes (3,3 GB, 3,1 GiB) copied, 166,45 s, 19,7 MB/s

Copying a 15 GB video file from my computer onto the Helios4 took almost exactly 13 minutes, which is pretty much the rate shown by dd: 15.000 MB / (13*60s) = 19.2 MB/s.

I was a little confused about this speed. The Helios4 has a gigabit ethernet port, my computer and all other involved components as well. Theoretically, this means a throughput of about 125 MB/s. I did not expect to reach this speed in practice, but only 20 MB/s seemed to be a bit low?

SSHFS Performance

Next, I tried SSHFS. You don’t need anything special, the only requirement is the ssh daemon and ssh is running on all my servers, anyway.

Again, I did not change any configurations or made any optimizations. This was run with the default settings when you mount a folder with SSHFS.

The usage of sshfs is sshfs [user@]host:[dir] mountpoint [options]. The directoy on the Helios4 is mounted to ~/test locally, which you can achieve with

sshfs helios4:/srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test ~/test/

I used the same dd command but with count=1000 so that 1 GB is transferred because I did not want to wait for 10 GB.

w=29MB/s, r=31MB/s

╭─max@host ~/test
╰─$ dd if=/dev/zero of=test.img bs=1M count=1000 status=progress
1044381696 bytes (1,0 GB, 996 MiB) copied, 35 s, 29,6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB, 1000 MiB) copied, 35,5097 s, 29,5 MB/s
╭─max@host ~/test
╰─$ dd if=test.img of=/dev/null bs=1M status=progress
1023410176 bytes (1,0 GB, 976 MiB) copied, 33 s, 31,0 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1,0 GB, 1000 MiB) copied, 33,8286 s, 31,0 MB/s

I transmitted the same 15 GB video file as above as well, which took about 8 minutes: 15.000 MB / (8*60s) = 31 MB/s. That’s definitely better than 19.2 MB/s.

So roughly, SSHFS seems to be about 10 MB/s faster than SMB. Interesting. This actually spiked my interest to actually do some benchmarking on the Helios4 itself. Up until now, I only wanted to get a feel for transmission speeds, which I could expect.

Test on the Helios4 locally

I connected to the Helios4 with ssh and run basically the same dd test as before, this time with a 10 GB file (count=10000). This shows the writing and reading speeds to the disk if no network is involved.

w=158MB/s, r=212Mb/s

╭─max@helios4 /srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test
╰─$ dd if=/dev/zero of=test.img bs=1M count=10000 status=progress
5071962112 bytes (5.1 GB, 4.7 GiB) copied, 32 s, 158 MB/s^C
4944+0 records in
4944+0 records out
5184159744 bytes (5.2 GB, 4.8 GiB) copied, 32.7079 s, 158 MB/s

╭─max@helios4 /srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test
╰─$ dd if=test.img of=/dev/null bs=1M status=progress
5084545024 bytes (5.1 GB, 4.7 GiB) copied, 24 s, 212 MB/s
4944+0 records in
4944+0 records out
5184159744 bytes (5.2 GB, 4.8 GiB) copied, 24.5122 s, 211 MB/s

Great! Writing happens with about 150 MB/s and reading at about 210 MB/s. This shows that the disk I/O is definitely not the limiting factor. Now, let’s do a test with a proper tool: bonnie++

Test with Bonnie++

With bonnie++ (website) you can test the performance of your filesystem and hard drives. The manpage tells you everything you need to know, but let’s look at the used options quickly:

-d sets the directory for the test
-c 1 the level of concurrency
-s 4024 the size of the file(s) for IO performance measures in megabytes. 4024 is twice the size the RAM of the Helios4 (2G)
-n 1 the number of files for the file creation test (measured in multiples of 1024 files)
-f specified without a parameter, this skips the per-char IO tests.
-b no write buffering, so fsync() is called after every write.

I really like the result:

w=146MB/s, rw=78MB/s, r=177MB/s

/usr/sbin/bonnie++ -d /srv/dev-disk-by-uuid-dfb36876-37ed-4860-9a86-ecf608a3d986/data/test/perform  -c 1 -s 4024 -n 1 -f -b
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M            146m  48 78.1m  34            177m  45 147.1   5
Latency                         185ms     383ms               104ms     435ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 1672561147   0 +++++ +++ -1046580777   0 -1081091648   0 +++++ +++ -898267616   0
Latency               177ms     276us   90292us   59943us      24us     103ms

EXT4 on RAID 1

Next, I removed OpenMediaVault and built my own setup. First, I created an EXT4 filesytem directly on the RAID 1 to compare the result to the previous test. The difference is that no LVM is used anymore. Both tests were run with no encryption in use.

The RAID 1 can be accessed with the device /dev/md0.

The following command creates the EXT4 filesystem. By default, a block size of 4096 is used. This is not relevant now, but it will be later when a LUKS container is used for encryption.

sudo mkfs.ext4 /dev/md0

Mount and run the test.

sudo mount /dev/mapper/cryptroot /mnt/md0
/usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b

Result: w=160MB/s, rw=96MB/s, r=160MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M            160m  54 96.6m  39            160m  37 145.0   4
Latency                         170ms     287ms               194ms     471ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -569212263   0 +++++ +++ 1545298178   0 1958692144   0 +++++ +++ 1912692157   0
Latency               199ms     264us     116ms     119ms      40us   91260us

Second run: w=164MB/s, rw=95MB/s, r=167MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M            164m  53 95.9m  38            167m  39 146.9   4
Latency                         176ms     517ms             95391us     400ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -222359118   0 +++++ +++ 1549587127   0 -1213966720   0 +++++ +++ -2058643209   0
Latency               194ms     268us     100ms     105ms      17us   89368us

Interesting! While reading seems to be a bit slower, the writing and rw value are slightly higher than on the setup with OVM.

EXT4 on LUKS on RAID 1

Instead of creating the EXT4 filesystem directly on the RAID 1 device, we will use a LUKS container to provide encryption and create an EXT4 filesystem on top of it.

It’s important to note that these results cannot be compared to the previous speed measurements because no encryption was used there. As you will see, the additional computing power for the encryption takes its performance toll.

First, let’s briefly look at the two 4 TB drives. Both, /dev/sda and /dev/sdb report a physical sector size of 4096 bytes and a logical size of 512 bytes. Hence, it should be ensured that the LUKS container and the filesystem use a block (sector) size of 4096 to use the drive efficiently.

helios4:~:% sudo hdparm -I /dev/sda | grep 'Sector size'
Logical  Sector size:                   512 bytes
Physical Sector size:                  4096 bytes
helios4:~:% sudo hdparm -I /dev/sdb | grep 'Sector size'
Logical  Sector size:                   512 bytes
Physical Sector size:                  4096 bytes

Unaligned LUKS Sector Size and EXT4 Block Size

The LUKS container was created with default options:

sudo cryptsetup luksFormat /dev/md0

And by default, LUKS uses a sector size of 512 bytes. Information about a LUKS container can be read with the cryptsetup command as follows.

helios4:md0:% sudo cryptsetup status /dev/mapper/cryptroot
/dev/mapper/cryptroot is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/md0
  sector size:  512
  offset:  32768 sectors
  size:    7813740160 sectors
  mode:    read/write

The LUKS container can be opened with sudo cryptsetup open /dev/md0 cryptroot.

Thereafter, the EXT4 filesystem was created with default options, too:

sudo mkfs.ext4 /dev/mapper/cryptroot

The default block size is 4096 as can be seen with command dumpe2fs:

helios4:md0:% sudo dumpe2fs /dev/mapper/cryptroot | grep 'Block size'
dumpe2fs 1.44.5 (15-Dec-2018)
Block size:               4096

The sector size of the LUKS container and the block size of the filesystem are not identical. This is not recommended and will likely result in a performance loss. But I wanted to know how big the difference between aligned and unaligned sector/blocksize actually is.

A friendly reminder about the speed with an unencrypted EXT4 filesystem on the RAID 1:

w=164MB/s, rw=95MB/s, r=167MB/s

Bonnie++ is run with /usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b.

Result: w64MB/s, rw=34MB/s, r=53MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           64.1m  20 34.1m  13           53.8m  12 150.5   5
Latency                        1146ms    1184ms             73276us     361ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -1499258492   0 +++++ +++ 1469852849   0 162904477   0 +++++ +++ -1983072567   0
Latency               193ms     281us     171ms     167ms      20us     210ms

Second run: w=64MB/s, rw=30MB/s, r=53MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           64.2m  20 30.6m  12           53.1m  12 147.4   4
Latency                        1044ms     922ms               155ms     312ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 693104966   0 +++++ +++ -1357348910   0 1029287914   0 +++++ +++ 1121994707   0
Latency               230ms     270us     108ms     162ms      39us     142ms

I did a few more runs but they all yielded the same results. As you can see, the speed drops significantly.

Aligned LUKS Sector Size and EXT4 Block Size

The following setup uses the correct sector size of 4096 for the LUKS container, which slightly improves the benchmark results.

The commands to create the setup:

sudo cryptsetup luksFormat --sector-size 4096 /dev/md0
sudo cryptsetup open /dev/md0 cryptroot

Now, the sector size is 4096:

helios4:~:% sudo cryptsetup status /dev/mapper/cryptroot
/dev/mapper/cryptroot is active.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/md0
  sector size:  4096
  offset:  32768 sectors
  size:    7813740160 sectors
  mode:    read/write

Create the EXT4 filesystem.

sudo mkfs.ext4 /dev/mapper/cryptroot
sudo dumpe2fs /dev/mapper/cryptroot | grep 'Block size'

The block size of EXT4 is again 4096 (as the last time).

sudo mount /dev/mapper/cryptroot /mnt/md0
/usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b

Running the test with bonnie++ returns the following result, which is indeed a few megabytes better than previously. In my opinion, the encryption is worth it and I’ll happily take the performance trade-off.

w=73MB/s, rw=37MB/s, r=59MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           73.3m  23 37.4m  14           59.4m  14 143.5   4
Latency                         728ms     748ms             97800us     387ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 1961043105   0 +++++ +++ -1124638686   0 -110386393   0 +++++ +++ 1338972754   0
Latency               244ms     288us     148ms     210ms      25us     186ms

Second go: w=73MB/s, rw=32MB/s, r=59MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           73.4m  23 32.2m  13           59.5m  14 152.3   3
Latency                         694ms     618ms             78881us     397ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 289626538   0 +++++ +++ -1096656038   0 1782974649   0 +++++ +++ 665618027   0
Latency               173ms     260us     110ms     170ms      24us     102ms

Encryption Offloaded to the CESA Unit

Then I remembered that the Helios4 has a CESA unit (Cryptographic Engines and Security Accelerator unit), which can be used to offload encryption. I thought that using the CESA unit will improve the benchmark results further, but the results were quite disappointing.

While creating the LUKS container, the cipher aec-cbc-essiv:sha256 must be specified to use the unit. This can be achieved with the following command. We keep the correct sector size, too.

sudo cryptsetup -c aes-cbc-essiv:sha256 luksFormat --sector-size 4096 /dev/md0

Open the container.

sudo cryptsetup open /dev/md0 cryptroot

Verify cipher and sector size.

helios4:~:% sudo cryptsetup status /dev/mapper/cryptroot
/dev/mapper/cryptroot is active.
  type:    LUKS2
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  key location: keyring
  device:  /dev/md0
  sector size:  4096
  offset:  32768 sectors
  size:    7813740160 sectors
  mode:    read/write

Create filesystem.

sudo mkfs.ext4 /dev/mapper/cryptroot

Check block size.

helios4:~:% sudo dumpe2fs /dev/mapper/cryptroot | grep 'Block size'
dumpe2fs 1.44.5 (15-Dec-2018)
Block size:               4096

Let’s mount the partition and start the test!

sudo mount /dev/mapper/cryptroot /mnt/md0
/usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b

w=48MB/s, rw=29MB/s, r=60MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           48.8m  16 29.9m  12           60.7m  14 145.8   5
Latency                        1150ms    1852ms             58338us     347ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -2068442525   0 +++++ +++ 1658309631   0 -747844300   0 +++++ +++ 614890684   0
Latency               171ms     262us     155ms     173ms      27us     139ms

second run: w=48MB/s, rw=27MB/s, r=59MB/s

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
helios4       4024M           48.6m  16 27.5m  11           59.4m  13 147.9   5
Latency                        1193ms    1148ms             93477us     356ms
Version  1.98       ------Sequential Create------ --------Random Create--------
helios4             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1 -932939759   0 +++++ +++ 827095864   0 1905523812   0 +++++ +++ -1495813491   0
Latency               181ms     270us     214ms     238ms      21us     140ms

Well, the result is not very convincing. With the encryption offloaded to the CESA unit, the results are worse than before.

Table of Results with Bonnie++

All setups use a RAID 1 and the EXT4 filesystem with a block size of 4096.

The exception is the EXT4 filesystem created by OMV. I’m not sure which block size was used as I created it with the OVM web GUI and I didn’t check it in the terminal. However, I assume that the default block size of 4096 was used.

Nonetheless, if another block size than 4096 was used, this might explain the worse performance of the setup of OMV with EXT4 on LVM on RAID 1 versus the plain EXT4 on the RAID 1; but perhaps this is just the indirection layer of LVM?

Setup	LUKS sector size	encryption	write MB/s	read-write MB/s	read MB/s
OMV - EXT4 on LVM	no luks	none	146	78	177
EXT4	no luks	none	160	96	160
EXT4 on LUKS	512	aes-xts-plain64	64	34	53
EXT4 on LUKS	4096	aes-xts-plain64	73	37	59
EXT4 on LUKS (CESA)	4096	aes-cbc-essiv:sha256	48	29	60

Regarding the CESA unit, there exists a HTTPS Benchmark in the official wiki.

Out of Scope

There may be many aspects, which affect the measurement itself as well as how you evaluate the results. For instance, I did not look at the energy usage or CPU usage while the HTTPS benchmark takes the CPU usage into account.

Personally, I settled on the RAID 1 with an EXT4 filesystem on a LUKS container (with the correct sector size, of course). This provides encryption, which is a must-have, and reasonable speed. Furthermore, no other software but ssh is required, which is already running anyway.

Send RAID Alerts with my Helios4 via Email

2021-05-03

The goal is to be able to send RAID alerts from my Helios4 to my email address and because my personal IP address provided by my ISP is on a default blocklist and therefore not allowed to send mail, I quickly created a workaround. I already use a VPS, which is set up to send mail, so I figured I just need to connect my Helios4 to my VPS and tell the server to send the email instead.

On the VPS I created an extra user for this, so I can clean this up quickly if it went sideways (but it didn’t) and for some other constraints such as limiting this public key to only executing one script (more on that later).

The new user on the server is quickly created with:

sudo useradd -m heliosmail

Then I created a new keypair (ssh-keygen -t ed25519) on the Helios4 and copied the public key to /home/heliosmail/.ssh/authorized_keys on my server. The user heliosmail (if authenticated with this public key) should only be able to execute one command, so we modify the authorized_keys file slightly. The reason is that the SD card of the Helios4 is not encrypted itself (only the data on the RAID) and if that keypair got compromised, then I don’t want anyone to be able to log into my VPS. Specifically, I inserted this part in front of the public key: command="/home/heliosmail/mail.sh $SSH_ORIGINAL_COMMAND"

command="/home/heliosmail/mail.sh $SSH_ORIGINAL_COMMAND" ssh-ed25519 AAAAC3NzaC1lZ...

Now, when the user heliosmail logs in (actually, authenticates successfully), the mail.sh script is automatically executed. The variable SSH_ORIGINAL_COMMAND contains the original command which was issued by the user. So you could log this if you wanted, but I use this variable to pass an argument (the event why an email is now being sent) to the script.

The mail.sh script has the following content:

#!/bin/bash

EVENT="$1"

# first, strip underscores
CLEAN=${EVENT//_/}
# next, replace spaces with underscores
CLEAN=${CLEAN// /_}
# now, clean out anything that's not alphanumeric or an underscore
CLEAN=${CLEAN//[^a-zA-Z0-9_]/}
# finally get the spaces back because I like them.
CLEAN=${CLEAN//_/ }

(
echo "To: my@email.com"
echo "Subject: Helios RAID alert"
echo "Look at the RAID."
echo "Event: $CLEAN"
echo "."
) | sendmail -t

The first argument which is passed into the script is the event. This is the reason why mdadm triggered the configured script on the Helios4 (see below). And because I pass the argument to the echo command, it should be sanitized in order to avoid command injection.

Then I decided to modify the script which is run by mdadm. This is configured in /etc/mdadm/mdadm.conf on the line which starts with PROGRAM. The line starting with MAILADDR specifies the mail address and I commented it out because the Helios4 itself is not sending any mail. ¹

PROGRAM /usr/sbin/mdadm-fault-led.sh

Actually, this script makes an LED blink on the Helios (or switch it on permanently if a critical error occurred). The only line which I added to this script is the one after the EVENT variable is defined.

#!/bin/bash
#
# Make Red Fault LED (LED2) reports mdadm error events.
#
EVENT=$1

# I added the following line to connect to my VPS
su -c "ssh cloud $EVENT" max >/dev/null
...

I use su in order to change to the user max because the ssh pubkey authentication is only configured for this user and not for root. The -c flag specifies the command which should be run and the $EVENT is the reason why this script was triggered in the first place and I want to see this in the email as described earlier.

The ssh cloud part works because cloud is defined in ~/.ssh/config and $EVENT would be the actual command which should be executed when connected to the server (stored in the SSH_ORIGINAL_COMMAND variable on the server).

Inserting this line of code at the beginning also means that an email for every event is being sent. If this is not desirable, you can move the inserted line into one or more of the following if statements which check for certain events. You can also check the official docs on the Helios4 wiki.

# Active component device of an array has been marked as faulty OR A newly noticed array appears to be degraded.
if [[ $EVENT == "Fail" || $EVENT == "DegradedArray" ]]; then
    echo none > $TRIGGER
    echo 1 > $BRIGHTNESS
fi

# An md array started reconstruction
if [ $EVENT == "RebuildStarted" ]; then
    echo timer > $TRIGGER
    echo 1 > $BRIGHTNESS
fi

# An md array that was rebuilding, isn't any more, either because it finished normally or was aborted.
if [ $EVENT == "RebuildFinished" ]; then
    echo none > $TRIGGER
    echo 0 > $BRIGHTNESS
fi

# Test RED Fault LED
if [ $EVENT == "TestMessage" ]; then
    echo timer > $TRIGGER
    echo 1 > $BRIGHTNESS
    sleep 5
    echo 0 > $BRIGHTNESS
fi

That’s it!

If everything works, you should get an email if you run

sudo mdadm --monitor --scan --test -1

which triggers the event "TestMessage". You should know that if the script /usr/sbin/mdadm-fault-led.sh gets updated, you may need to make the above modifications again.

This should not matter but if you run sudo mdadm --monitor --scan --test -1 and sendmail is not installed, you’ll get a message saying that “sendmail could not be found”. ↩︎

Reverse SSH connection

2019-11-14

I have a raspberry pi which is running pihole and some other things and I want to be able to connect to it from wherever I am without exposing it directly to the internet. So, I thought I could make this possible with an SSH connection form my raspberry pi to my VPS and, at the same time, forwarding the SSH port. Then I can use my VPS as a jumphost by connecting to the VPS and then to the forwarded SSH port from pi.

Generating SSH key pairs

First, let’s generate the needed key pairs for the vps and the raspberry pi. I like to create folders in .ssh to keep my keys sorted.

I use Curve 25519 for pubkey authentication.

ssh-keygen -t ed25519

Generate two keys. My directory structure looks like this

-- .ssh/
  |- config
  |- pihole/
    |- id_ed25519
    |- id_ed25519.pub
  |- vps/
    |- id_ed25519
    |- id_ed25519.pub

with the following SSH config

Host vps
    HostName VPS_IP
    User USERNAME
    Port PORT
    IdentityFile ~/.ssh/vps/id_ed25519

Host pihole
    HostName LOCAL_IP
    user USERNAME
    IdentityFile ~/.ssh/pihole/id_ed25519

Great, now I can connect to my vps with ssh vps and to my raspberry pi with ssh pihole.

Raspberry Pi

I want to make sure that my raspberry pi is always connecting to my vps, e.g. if SSH crashed or if I restarted my pi. I use a systemd service and SSH public key authentication to accomplish this.

Go to /etc/systemd/system/ and create a ssh-reverse.service file with the following content:

[Unit]
Description=Reverse SSH connection
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/ssh -vvv -g -N -T -o "ServerAliveInterval 10" -o "ExitOnForwardFailure yes" -R 22221:localhost:22 -l USERNAME -p PORT -i /home/USERNAME/.ssh/id_ed25519 VPS_IP
Restart=always
RestartSec=5s

[Install]
WantedBy=default.target

Replace USERNAME, PORT, VPS_IP and the path to the IdentityFile (-i) with your values.

A few important options (more info: man ssh_config)

-R 22221:localhost:22 This forwards the SSH port on my pi (localhost:22) to the port 22221 on my vps
ExitOnForwardFailure yes If forwarding of the port fails, we want to exit
ServerAliveInterval 10 My pi sends a packet every 10 seconds to keep the connection alive

If this unit is not running, we try restarting the connection every 5 seconds. Make sure, that you already accepted the public key of your VPS on the raspberry pi by connecting to it manually. Otherwise the systemd unit will fail over and over again.

Then enable the service

sudo systemctl start ssh-reverse.service
sudo systemctl enable ssh-reverse.service

SSH configuration

Let’s write the SSH configuration to connect to the raspberry pi via the VPS.

The configuration for the connection to the VPS is already in place and, in my case, it’s called vps. So, if I do ssh vps I will be connected to my VPS. Great. Now let’s use the VPS as jump host.

Write a new Host configuration in ~/.ssh/config. I called it pihole-remote so that I can still connect to my raspberry pi via ssh pihole in my LAN when I’m home.

Host pihole-remote
    ProxyJump vps
    HostName localhost
    user PI_USER
    Port 22221
    IdentityFile ~/.ssh/pihole/id_ed25519
    LocalForward 8888 localhost:80

Now I can connect to my raspberry pi by using my VPS as a jump host. I’m also fowarding localhost:80 to 8888, so that I can access my pihole webinterface in my browser whenever I am connected to my pi with SSH.

Resources

0001-01-01

This page contains links to organizations/people/tech stuff that I find interesting. If you know something interesting that’s not on here, tell me.

Civic Organisations

Journalism - Humans

Michael Moore. Movies/documentaries include
Chris Hedges. Covering US foreign policy, economic realities, and civil liberties in American society.
John Pilger

Documentaries

Deep Web (2015) About the Silk Road and Ross Ulbricht

Tech Independence

Detailed instructions by Derek Sivers

Git Hosting

If you’re looking for other places than GitHub to host source code:

Blogs and Websites I like // Blog Roll

Derek Sivers. Very interesting life; musician, circus performer, entrepeneur, slow thinker, likes a different point of view
Alexey Guzey
Realitätsabzweig - Frank Rieger (German) as well as his post We lost the war. Welcome to the world of tomorrow (English) on his blog Knowledge brings Fear
Fefe’s Blog - Felix von Leitner (German)
Verfassungsblog/On Matters Constitutional Many well-versed and detailed posts on EU law, regulation, and politics.