EuroBSDcon 2022 Vienna

I have been meaning to attend EuroBSDCon for quite some time now, but wanted to do so in an environmentally conscious way. Practically this boils down to not flying.

In 2019 this meant I could not attend EuroBSDcon in Lillehammer, Norway; I was just not naive enough to travel 24 hours by train with 7 transits and an equal number of possibilities to miss a connecting train.

In 2020 and 2021 COVID-19 simplified any travel deliberations by putting a full stop to most in-person conferences, including EuroBSDcon. Though at the end of 2021 the EuroBSDcon Foundation announced that EuroBSDcon 2022 would be held in Vienna, Austria. And as it turns out, ÖBB, the Austrian Federal Railways, operates a night train from Amsterdam, The Netherlands to Vienna, Austria. With a 75% lower carbon footprint compared to flying the night train would be my ticket to EuroBSDcon 2022

Travel

Traveling by night train (Nighjet in ÖBB terms) is rather uneventful. And that is a good thing! Although the bunks were a little bit harder than I was comfortable with, I slept reasonably well. It is a pleasant experience to board a train, go to bed, and wake up practically at your destination.

Tutorials

The first two days of EuroBSDcon consisted of tutorials. I had chosen to attend Marshall Kirk McKusick’s Introduction to the Kernel Services and I/O System of the FreeBSD Open-Source Operating System on the first day and An Introduction to the Filesystems and Networking in the FreeBSD Open-Source Operating System on the second day.

McKusick is an excellent teacher and presenter. In two days time he was able to demystify what has alway been a black box to me: the FreeBSD kernel. It made me realize that the FreeBSD kernel is probably one of the better documented pieces of software out there.

Another thing that I enjoyed greatly were the many stories and bits of history that McKusick shared with us. Unix (BSD) has a rich history and having it recounted by one of the primary players is wonderful thing.

Talks

Talks generally provide a good a insight into what is happening in a community. The talks at EuroBSDcon certainly did so. I won’t go over all the talks that I attended, just the ones that that stood out to me.

Day One

On FreeBSD, compliance, and getting up from under the bus - Eirik Øverby

I was particularly impressed by one of the first talks, On FreeBSD, compliance, and getting up from under the bus by Eirik Øverby of Modirum. In this talk Eirik recounted the story of his company having to deal with increasingly sophisticated and massive DDOS attacks. He not only covered how they dealt with it technically, but also the toll it took on him, other team members and even their families. At times I thought I could still see the pain of it all on his face. When you read about DDOS attacks in the news, you hardly ever get to see the effect it has on people. This talk did and it left an impression on me.

It was painful to see that the real solution to the problem Modirum was facing, and still is facing, is entirely the making of the credit card issuers themselves. Their fall-back option, when 3D Secure is not available, is to not require it. This motivates criminals to DDOS services, such that of Modirum, to take them off-line. With these security services off-line, criminals can use their stolen credit cards without a problem.

So why do the credit card issuers have such a lousy fall-back option? Turns out it is cheaper for them to reimburse the payments made with these stolen credit cards then it is to stop processing credit cards at all when 3D Secure is off-line. This is what perpetuates these DDOS attacks.

Scaling ZFS for NVMe - Alan Jude

ZFS is an amazing file system. It is robust, flexible and yet still easy to use. During the tutorials it had also became clear to me that, internally, is it an inherently complex piece of software. This talk indirectly emphasized that by showing that it is not exactly straightforward to let ZFS perform well, let alone optimally, on storage pools exclusively made up of NVMe drives. This is not at all strange if you consider that ZFS was conceived in a time when storage made up of spinning disks provided the best cost/capacity ratio and solid-state drives were prohibitively expensive. But as with many things problems need to be identified first before they can be solved. And this talk certainly demonstrated the problem clearly.

GPU passthrough with bhyve - Corvin Köhne

I had met Corvin earlier on during the hallway track. That brief chat had me intrigued sufficiently to attend his talk.

Corvin works at Beckhoff, a company that develops industrial automation solutions. Most, if not all, control software runs on Windows. That has been the industrial norm and is what their customers expect. But at Beckhof they run their Windows instances virtualized on Bhyve and for that they need GPU passthrough. Corvin has been making steady progress on Bhyve GPU passthrough. He demonstrated it working by running Windows graphical benchmarks at 85-90% of the performance compared to running it directly on the hardware. An achievement that got the attendees cheering. Corvin does however need a bit of help getting his Bhyve patches reviewed.  

Day Two

Ctrl-Alt-Del: Learning to Love Legacy Code - Dylan Beattie

Dylan had an entertaining talk about legacy code. He emphasized that the world runs on legacy code, and outlined what is needed to deal with it. He captured that in an all too familiar keyboard shortcut: Ctrl-Alt-Del: you need to be able to Control it, Alter it, and eventually Delete it.  

While discussing these three topics he touched upon many good points. I will only be emphasizing a couple of them here as he has been given this talk for a number of years now and it can easily be found on YouTube.

Dylan showed three definitions of legacy code:

Not only did these definitions resonate well with me, it also made me realize how definitions can influence and alter perspectives. While I tended to equate legacy code simply with old code or code using old technologies, I now find that it can easily be applicable to fairly new code as well.

This might come in handy in organizations where technical arguments do not always carry the weight they arguably should. The negative connotation associated with the term legacy code can then be wielded as an argument to address the issues that are inherent to legacy code.

Many years ago I had a similar and even stronger change in pespective when I read Code as Design: Three Essays by Jack W. Reeves. I often point to these essays when non-technical managers do not seem to understand why software development is so difficult to estimate correctly. I generally love these changes of perspectives and the insight that comes along with it.

Dylan made another important observation:

Sometimes we are more interested in the technology then we are in being able to solve the problem.

This struck a nerve. In the early 2000’s, after having completed a somewhat strenuous project, I read Fred Brooks’ The Mythical Man-Month. The anniversary edition of that book has a chapter No Silver Bullet. In it a distinction is made between:

According to Wikipedia, accidental complexity relates to problems which engineers create and can fix whereas essential complexity is caused by the problem to be solved, and nothing can remove it.

Ever since having read these terms and their definitions I see everything in software development in those terms. It has convinced me that one of the major things holding us back in software development is the sharp rise of accidental complexity; accidental complexity introduced by developers being more interested in technology then the problem needing to be solved. These days I am spending more and more time solving accidental complexity issues, then I am creating actual functionality to solve the problem at hand (essential complexity). It takes away the joy of software development.

Dylan also touched upon hard constraints. In other industries, e.g. construction, aviation, aerospace, it is common to have hard constraints on what is being build. Not so in the software industry. We hardly ever set limits — with the possible exception of embedded systems — on code size or features. Software is extremely malleable. Without hard constraints software grows unwieldy.

There are many more excellent points that Dylan touched upon. Do look up his talk on YouTube. It is worth it.

Other Talks

Now that I am reluctantly working more and more with Docker and Kubernetes I was very interested in what FreeBSD had to offer on top of jails. Hence I attended two interesting talks:

I had never heard of pot before. But due to Luca Pyzzamiglio’s presentation I now have. I will definitely consider it once I get the opportunity to do some FreeBSD deployments again.

Equally interesting was Yan Ka Chiu’s presentation. I always find it interesting to hear how people are using FreeBSD in production. On top of that Yan Ka Chiu gave a good overview of the current container options (including pot) on FreeBSD.

While I haven’t worked much with OpenBSD — I ran it very briefly on my Thinkpad T480 before installing FreeBSD — I am alway enamored by its straightforwardness and simplicity. In an age where software always seems to get more complex OpenBSD’s stance is a very respectable one. The talk Taming the fox - Landry Breuil provided a neat insight into pledge and unveil. It emphasized once again OpenBSD’s elegant solutions to problems.

Conclusion

This was my first BSD conference. It was fun, interesting and stimulating. Next year, EuroBSDcon will be held in Coimbra, Portugal. Unfortunately, I do not yet see how I can get there from The Netherlands in a environmentally conscious way. This means I probably won’t be attending.