In this latest interview, Theo examines the past five years of OpenBSD development. He also discusses the OpenBSD 3.9 theme song, "Blob!", detailing what blobs are, why OpenBSD avoids them, and how OpenBSD developers work to reverse engineer them. Looking to the development process, Theo talks about recent and future "mini-hackathons", small and focused OpenBSD development gatherings. Finally, Theo also discusses the OpenBSD project's funding issues, and the response to requests for funding from users of the project's OpenSSH software.
Background: Jeremy Andrews: Your last interview with KernelTrap was back in November of 2001, shortly before OpenBSD 3.0 was released. Can you summarize some of the major improvements to OpenBSD between 3.0 and the current stable OpenBSD 3.8 release?
Theo de Raadt: That is almost 5 years, since we add "0.1" to our release number every 6 months. Our pf packet filter was born in 2001, sparc64 support showed up around 3.1, our W^X and other address space randomization changes around 3.4, amd64 support around 3.5, and after that all of our bgp/ospf routing daemons, much more recently we added many wireless drivers (staying ahead of Linux even) and now we are attempting to do more RAID management, temperature and other types of sensors... too much stuff.
Jeremy Andrews: How has the number of active OpenBSD developers changed over the years?
Theo de Raadt: The developer count increases and decreases, but mostly it hovers around 80 semi-active or active people. The hackathons and tree-unlocks (ie. when in the release cycle we branch the tree) create bursts of activity and then it seem like a lot of people. Sometimes development can seem a bit slow because of various reasons, but it always recovers.
Jeremy Andrews: What are some of the major challenges that the OpenBSD project has faced since OpenBSD 3.0?
Theo de Raadt: Well, internal to our project the biggest problem always is -- and likely will be -- that we do not have enough developers to keep up with the constant supply of new devices which vendors produce. But we try. At least, it appears our user community is fairly happy with our efforts. We try to focus our effort on the most important devices.
That brings us to the large external problem we have -- the most important devices to support sometimes are completely undocumented. It is quite a strange situation, to see the most common devices undocumented, while vendors of more fringe devices normally are very willing to provide documentation. In certain fields of operation, such as ethernet or i2c, devices which lack documentation are rare. Unfortunately a few vendors like Marvell and Broadcom are trying to lock ethernet up again. Some Linux (and recently FreeBSD too) developers are willing to sign NDAs so that a few people get the documentation, and I believe that this is the largest problem facing the kernel side of the open source community today.
Jeremy Andrews: Has the OpenBSD project succesfully convinced any vendors to release documentation that was previously refused?
Theo de Raadt: Yes, this has happened a few times. But normally this has been because the earlier refusal came from the wrong place inside the company. When we succeed later, it is because we have found a better way into the company.
Jeremy Andrews: How does OpenBSD convince vendors that they're better off releasing documentation?
Theo de Raadt: Well we tell them various angles on the truth, to make them understand. The problem is that the message you must give to a vendor depends largely on which particular person in the company you are talking to. Is he management? A decision maker? Someone who gains from making a risky move? Or someone who is afraid? Is he an engineer who needs ammunition for politics inside the company? Or have you fallen into the pit of PR despair, where the person is just supposed to shut you down? Each of these people needs to hear a different message. In the end of course, we firmly believe these sub-messages are all part of the same message -- giving us documentation allows us to support your devices so that some more consumers know that your devices are supported.
The problem is that normal business practices in the industry, these days, do not consider the end-user the consumer. Instead, the chipset vendors consider companies like Dell or IBM or HP to be their customer. You, the end user, are the product that these large VARs sell to the chipset vendors, because you become locked to the chipset vendor's product..
And I suppose it is therefore clear why the small vendors are willing to give us documentation for their chips.
Jeremy Andrews: Can you offer some specific success stories of the OpenBSD project regarding documentation?
Theo de Raadt: I think the biggest success is that most Taiwanese vendors give us documentation almost immediately. A little while ago we even had one (JMicron) contact us completely out of the blue, with full documentation and hardware.
Jeremy Andrews: Which specific vendors do you find to be the most difficult to work with?
Theo de Raadt: Well, TI has never returned a mail regarding their wireless chipset. Broadcom has never replied either, and even though driver support for their chip is now possible because of reverse engineering by some Linux folk, there is still the issue of no free re-distribution of the microcode, so this is a lot like the Intel situation. Intel talked to us, but they still refuse to make the microcode re-distributable; I really think it is such a simple thing we ask for. Mostly, it is American companies who I think feel entitled to not change their behaviour.
Jeremy Andrews: Which vendors stand out as being the most helpful to projects like OpenBSD?
Theo de Raadt: The little vendors.
Jeremy Andrews: OpenBSD 3.9 is scheduled for release on May 1, 2006. What are some of the new features and overall improvements that we can look forward to in this release?
Theo de Raadt: The big improvements are the thousands of little changes that are mostly not big Our project really believes in making small progressive changes instead of dangerous redesigns. This also allows us to lock our release schedule to 6 months, and I think that this in turn encourages the developers to think of how to develop carefully.
Jeremy Andrews: Each OpenBSD release includes a theme song that goes along with the release's artwork. OpenBSD 3.9's theme song is titled "Blob!". Can you explain what binary blobs are and why they're a problem?
Theo de Raadt: Vendors often try to hand off two kinds of binary code to us, which they expect we will happily incorporate into our system (and then, hopefully, we will shut up).
The first kind to mention is firmwares. Firmwares (like for instance on a Intel wireless card, or a such) are binary pieces of code that will run on the little processor that is on the wireless card. As an operating system, we need to load the code out to the card. To include a firmware in OpenBSD, we simply need a nice copyright statement from the vendor that lets us distribute the firmware. Some vendors won't even go that far, though.
The second kind of binary data vendors feed us are blobs. This is code that is expected to be linked against the operating system and run on the host processor. There are many problems with this. First off, can we trust the code to do what it should do? I don't think so. If there is a bug, can we fix it? No, as developers our hands are tied, and if our user community runs into bugs it just makes us look bad. Therefore when faced with the choice of supporting a device very poorly (as the blob would force us to) we instead choose to wait until we (or someone else) can reverse engineer it or.
Jeremy Andrews: What is it about binary firmware that you're willing to ship it, versus binary blobs? How can you trust the firmware binary to do what it should do? And what if the firmware has a bug?
Theo de Raadt: Quite honestly I prefer chips which have no firmware, and instead use correctly designed hardware logic, which our driver must then drive. Note that most ethernet chipsets do not use a processor, but many scsi chipsets do. Most IDE chipsets do not, but for wireless devices ... it is about half and half. This clearly has to do with the complexity of the data flow problem being dealt with.
But in the end, if we wish to support any such devices, we must be practical. We must accept the risk that there is a flaw in the firmware. (Is that not what many of us have been coping with for years now with Prism wireless chipsets and their firmware update tools?) But the legal climate is a real problem for us -- that is why we must get copyright permission to distribute the firmware images. Once they are distributed... at least the device works.
Of course, also note that we don't want to become Hermes (the architecture of the Lucent/Prism/Symbol chip) assembly language programmers... we have more than enough to do. Just a specific example. Please, people, don't load us up with more tasks
Jeremy Andrews: Blobs seem especially common with wireless ethernet cards and graphics cards, why is that?
Theo de Raadt: Graphics cards have gotten to this point because of their complexity. But these blobs also cope with lots of bugs in the devices. These bugs are because graphics cards are devloped very quickly now, and the hope is that software would work around the hardware bugs.
I don't know why any wireless cards use blobs. In fact, very few do. They should just document their chips. There's a lot of hogwash flying around about FCC rules, but if that was a concern of theirs they should just design their chips to lock the channels in hardware. But of course, noone in Taiwan does. So did the US vendors just tie the own hands? Certainly, we don't feel like we are constricted. The minute we are able to support another wireless device, we immediately ship the driver.
Jeremy Andrews: Can you explain more about these FCC rules that wireless manufacturers refer to?
Theo de Raadt: Basically the rules say that the devices must be closed boxes; that people should not be able to tune them to transmit outside of the specified frequencies. Now the vendors have gone and created devices which can sometimes do that. But I don't see how it is our fault, as an operating system group that makes free code that interoperates with a user's device, if these vendors went and created flawed devices. Let the FCC go after the vendors who made the flawed devices. In any case, most our users and developers do not live in the country where the FCC operates.
Jeremy Andrews: How is OpenBSD able to ship without any binary blobs?
Theo de Raadt: In most cases we have reverse engineered the code. In other cases we have actually gotten Linux source code that implements the functionality. In at least one case we have even gotten a glimpse at some vendor code which turned out to be BSD licensed, and this was code found lying around on the net.
Jeremy Andrews: What blob-based wireless cards remain unsupported by OpenBSD?
Theo de Raadt: Just a few of the newest Atheros devices. The problem in the wireless domain for us is more about firmwares without distribution rights. If Intel and Broadcom would give their firmware out so that we could include it, the users would benefit. But again, these chipset vendors do not feel that there is any relationship between them and the users.
Jeremy Andrews: Why would a chipset vendor be unwilling to distribute their firmware? What do they have to gain by not distributing it?
Theo de Raadt: I just don't understand. I mean, they let Microsoft and IBM and the various VARs distribute their firmware. So why not us? Why not the Linux vendors? Oh, but a few Linux vendors will distribute the firmwares, under some rather restrictive rules.
Jeremy Andrews: Why do you suppose binary blobs have become so common?
Theo de Raadt: Well, I think they really have not become that common at all. All the time we see more vendors that are open. It is just that right now the most common devices are made by a very small set of vendors who are very strictly trying to lock down their market in this way. But it could go very backwards for them quickly, too. For instance, the (newish) Realtek Gigabit Ethernet chips are not too bad at all, and there's lots of documentation. So maybe the Taiwanese products are a little bit later to the market, but they are simpler and robust once they do make it to the market.
Jeremy Andrews: Are these documented Taiwanese products that you refer to becoming more common?
Theo de Raadt: Ralink thinks that they shipped 14 million wireless chips in 2005. That's a very significant market presence. Of course, most of those do not go into PC's these days. Again this comes down to what the VARs decide should be put into consumer products.
One must also remember that Realtek continues to sell more ethernet chips every year than everyone else. And their gigabit chip isn't as bad as their old stuff used to be. But what if Dell, HP, Asus and such continue to push only chips which lack documentation? It is not just dire for us -- it is also bad for Linux.
Jeremy Andrews: What recommendations would you offer to user's of operating systems that include binary blobs? That is, what can they do to help discourage the spread of binary blobs while still using their favorite operating system?
Theo de Raadt: I don't know what to tell such people. Their operating system supplier is deciding this for them. Even in the Linux community there are distributions that refuse to include binary blobs, or at least, don't encourage it.
That said, almost all these systems are setup so that these blobs can be loaded. Quite often they even just say it is some LKM that supports some feature. The users never ask for source.
Jeremy Andrews: What are some examples of drivers that have been reverse engineered by the OpenBSD project?
Theo de Raadt: Oh there's too many to mention. The problem is that we define reverse engineering to apply to almost any situation where we lack vendor documentation. If we have to build our driver by reading mysterious code from somewhere else, say for example a Linux driver that someone wrote based on documentation they had under a non-disclosure agreement, we would still consider it reverse engineering -- because typically such code totally blows! Typically such code is full of unexplained magic numbers, and it is a serious effort to learn enough, and then create a driver for our use.
Jeremy Andrews: Have OpenBSD developers ever been threatened by companies whose products are being reverse engineered?
Theo de Raadt: No.
Jeremy Andrews: How do you motivate your developers to reverse engineer drivers?
Theo de Raadt: Once in a while I mention a project to some person. But most of of the time I become aware of it when someone like Jonathan Gray or Reyk Floeter or Damien Bergamini says "I have started working on this"....
Jeremy Andrews: What is the state of graphics card support in OpenBSD? Are graphics card drivers being reverse engineered like ethernet drivers?
Theo de Raadt: Well, that is more a job for the X developers. Though I have helped them get some documentation for some chips as well. I don't know how things are going there. It is sad to me though that everyone has ignored the work of Loic Duflot who basically showed how horrid the X server security model is.
Jeremy Andrews: Each year OpenBSD sponsors a "hackathon", flying developers in from all over the world. How does the project pay for these events, and what is gained?
Theo de Raadt: The hotel is paid for either by the project directly out of our CD sales money. Twice, DARPA money paid for the event. Another time, NLnet gave us a donation that helped a fair bit. And finally, another time a company made a very large contribution to thank us for a feature pf had just gained (the overload keyword).
Jeremy Andrews: Why are these events so important to OpenBSD?
Theo de Raadt: I think it is clear that putting people together in a room makes a lot of things move smoother.
Jeremy Andrews: You've also made mention of "mini-hackathons". How do these work, and are any currently planned?
Theo de Raadt: A few years ago we held a mini-hackathon for pf specifically, after a cansecwest conference in Vancouver which a few developers were attending. The mini-hackathon was held in a cabin near the town of Sechelt, in the woods, up the Sunshine coast; it even took a ferry ride to get there. I think there were about 15 people, staying in giant hunting tents that our hunting developer Bob brought from Edmonton. The cabin had power, but no net! There was a DSL link just under 1k away, up a hill, but it was at the maximum distance. We cut with a machete and rolled fiber through the bramble between the two buildings, and then we had net and could do commits! We were afraid that the deer would break the fiber, but the next day we saw them avoiding it, very carefully stepping over it...
The pf hackathon was very successful. It took a few years, but last November before the OpenCon conference in Venice we held a "ports" hackathon on San Servelo island in the Venician bay. That was absolutely incredible, and very productive, with the remaining package upgrade stuff being completed at the event.
Having learned what works (and kind of what doesn't) we are now planning about 4 mini-hackathons for the coming year
Jeremy Andrews: What is the focus of each of these upcoming mini-hackathons?
Theo de Raadt: There will be one focused on our new ipsecctl tool, another just for OpenSSH. The first one to be held in June will focus on finishing the multipath routing support in the kernel, and I have a feeling that quite a few more routing daemon improvements will happen as well.
Jeremy Andrews: What is ipsecctl, and what is in store for it at the mini-hackathon?
Theo de Raadt: Hans-Joerg Hoexer and Mathieu Sauve-Frankel have designed and built a new configuration tool for IPSEC, that is modeled a bit on how pf and /etc/pf.conf work. It was already in 3.8, was further improved for 3.9, but there is still a lot of work to do, in the end it will make IPSEC much easier for regular mortal to use.
Jeremy Andrews: OpenSSH seems to do what it was designed for quite well. What more do you have in mind for it?
Theo de Raadt: Well, OpenSSH is very important. So the big changes happening there are to improve performance and increase the security. As we become aware of more problems in the C language, we are trying to be very agressive to make the code cleaner. Just the standard OpenBSD proactive auditing process.
Jeremy Andrews: Once OpenBSD 3.9 is out the door, what are the plans for OpenBSD 4.0?
Theo de Raadt: Well it is still early in the 4.0 cycle. There are some crazy projects being worked on, but we will have to see what is ready for release. I believe that some level of ACPI support (maybe battery support, maybe at least better interrupt routing) will make it. And who knows what else.
Jeremy Andrews: The OpenBSD project recently had some media attention due to being low on funding. What are project funds used on, and where does the project get its funding from?
Theo de Raadt: First, I must clear up some misconceptions. We received some funding for a 2 year period when DARPA provided us with some funding -- essentially they paid the salaries of 5 people to work completely fulltime, bought about $30k in hardware, and paid for 3 hackathons). The original grant amount might have sounded positively gigantic at $2 million, but I swear I will never get over how incredibly much money a University acting as a middle man between DARPA and us can bleed the flow of financing. It was just astounding.
So, traditionally all our funding has come from user donations and users buying our CDs (our other products don't really make us much money). Obviously, that has not been a lot of money. We have however continued to make ends meet, and only cried for help when we started hitting problems.
Jeremy Andrews: I use OpenSSH every single day. I use it at home, and have used it at every computer job I've ever worked. I imagine this is true for a lot of people. What was the reaction to your recent request for donations from people and companies that benefit from OpenSSH?
Theo de Raadt: There were many reactions.
Some people thought that the tie between OpenBSD and OpenSSH is the problem, and thus they would not donate. Those selfish people apparently don't realize that OpenSSH-p is maintained by OpenBSD people, who don't need to do so, of course. Roughly stated, painting with some broad strokes, the Linux vendors flat out refused to help. They have not even really replied to requests. The commercial Unix vendors have tried to stay away from funding us as well, hiding in their castles, especially when users of our software sent them requests for action. Hundreds of people donated! Smoothwall, Mozilla, and GoDaddy made some large contributions, as large users of OpenSSH. A few other large players (users, not Unix/Linux vendors) have something happening inside their accounting departments. I don't understand why the Unix/Linux vendors (and Cisco) who ship our product are avoiding us. Maybe they are afraid to give to the first project, because then other projects will come asking too.
I think that contributions should have come first from the vendors, secondly from the corporate users, and thirdly from individual users. But the response has been almost entirely the opposite, with almost a 15 to 1 dollar ratio in favor of the little people. Thanks a lot, little people!
Jeremy Andrews: You refer to OpenSSH-p, the portable version of OpenSSH that runs on other operating systems besides OpenBSD. How much effort is involved in maintaining OpenSSH-p?
Theo de Raadt: A fair bit of effort, though it is mostly handled by 3 people. It is not getting easier, though, because many of the newer things we want to do (described above) will be rather agressive code re-organizations.
Jeremy Andrews: You have a reputation among non-OpenBSD projects as being difficult to get along with. Why do you suppose this is, and how does it affect you?
Theo de Raadt: I think lots of other projects have difficult people. Why is strlcpy still not in glibc? Because one person decided to be difficult. I am very easy to get along with, but I don't have time to waste being nice to people who are being stupid.
Jeremy Andrews: You have been known to occasionally make anti-Linux statements. What is it about Linux and the Linux community that draws your ire?
Theo de Raadt: I don't have a problem with Linux; I just don't use it. Nor do I think it is a newer and better or brighter or has less calories; everything we build is turds, we just move them around or shine them or have a different view on which way they should be rolled. I'm just tired of the various evangelical approach being taken by so many of the Linux users and developers. "Can't we all just get along?" If they keep throwing their ire at me, I will continue to call things as they are.
Jeremy Andrews: Why haven't we seen an OpenCC, or OpenX?
Theo de Raadt: To a certain extent we already have an OpenX, since Matthieu Herrb works in the OpenBSD X tree to develop new things. This is why our xterm has been privilege revoking for years now, and our X server is privilege separating.
I would love a new C compiler, but mostly because I feel that the gcc model is unmaintainable in the long term. What the gcc people have created is a compiler that is heavy on optimization. The result of their imperfect efforts thus is a compiler that generates incorrect code from time to time, and this affects us all. That also makes it very slow. I would love to see a new C compiler that was fully compliant, did minimal optimization, was small and fast, and high quality. But there is nothing in the open source sphere for that today. Nothing, nothing at all. We have investigated about 5 choices.
Jeremy Andrews: A high quality fully compliant C compiler that's small and fast, that sounds exactly like something we'd expect from the OpenBSD project. Is there any chance we'll hear about a C compiler mini-hackathon some day?
Theo de Raadt: I don't think we have found the right people to do this yet. Maybe we are picking up some skills with the effort we are putting into lint right now. It sure is teaching us how little people actually know about modern C.
Jeremy Andrews: You invest a tremendous amount of time and energy into creating a free, functional and secure operating system. Why is this so important to you?
Theo de Raadt: Whenever I work in a piece of code, I always get frustrated when I see very complicated API's designed by people... who are inexperienced, or so I used to think. Actually it turns out that designing correct (translation: SIMPLE) APIs is one of the most difficult problems in software development. The Unix kernel:userland API is such a great example of clever interface, and early on I became addicted to learning how to design things so simple, yet so powerful, especially compared to the Amiga, which I was experienced with beforehands. Don't get me wrong -- there are little flaws and mistakes, or at least, corner cases that one can never get just right. Over the last 20 years this has led me to where I am. In our group, I always stick my nose into any new API which someone builds, to try to see if I can help make sure it is just right.
That's the technical side. The other side is that I've got this great team who I get to play with every day. Why wouldn't that be important?
Jeremy Andrews: You live an adventurous life outside of leading the OpenBSD project, doing a lot of hiking and biking. What is it that attracts you to this lifestyle?
Theo de Raadt: Well, less biking these days. Eventually the eyes get tired of the screen, the wrists of the keyboard, the brain of the semantic arguments, and it is time to go explore some mountains!