Foreword — This article presents key findings from an embedded Linux developer survey written by Dr. Joachim Henkel and student Mark Tins of the University of Munich, with input from MIT's Eric von Hippel. The survey was posted at LinuxDevices.com and elsewhere in late 2003/early 2004.

Unlike commercial surveys from market research companies and analysts interested in how and where embedded Linux is used, or vendor market shares and revenues, this survey aimed to discover how the embedded Linux development process actually works. These findings focus mainly on source code sharing policies.

---

Munich/MIT Survey: The Development of Embedded Linux

by Joachim Henkel and Mark Tins

As readers of LinuxDevices.com are well aware, the use of Linux in embedded devices has increased enormously over the last several years. Surveys by various market research firms have confirmed this trend, but none have looked so far at the development process of embedded Linux.

Why is this development process an important issue? Embedded Linux — by which we mean modules and extensions to standard Linux that make it suitable for use in embedded devices — is unusual among open source projects in that most development contributions come from commercial firms, not hobbyists. Now, this may also be true for some other open source projects, and IBM and Red Hat, among others, contribute hugely to standard Linux. However, these firms have a strategic interest in establishing Linux as a widely used operating system. While some larger vendors of embedded Linux may have a similar interest, the situation clearly differs. Device manufacturers contributing to embedded Linux basically want to sell their hardware, and need some/any embedded operating system in it. Their technology needs differ strongly. Hence, embedded Linux can be described as a loose collection of very heterogeneous projects, to which many firms contribute.

Against this backdrop, several questions arise. First, why would firms contribute at all to the development of embedded Linux? After all, the GPL makes the protection of one's intellectual property embodied in embedded Linux rather difficult. Due to the GPL's licensing terms, the code becomes largely a “public good,” and economists would predict that too little of such a good would be provided.

Second, do firms that develop embedded Linux keep secret as much of the code as they can, or do they voluntarily reveal it? (Note that, despite the GPL, firms have some leeway in revealing their code: they could reveal it at once or only when the device comes to market, could sell it only to their customers, or could hide drivers in binary loadable modules.) More precisely: what code is revealed, what is kept secret? And what are the reasons to act one way or the other?

Third, who decides? Is there a company policy in place, or is it up to the individual programmer to reveal the code they write? What influence does open source culture exert on programmers, and how does this culture fit with that of a hardware manufacturing company?

We addressed these questions in a survey aimed at developers of embedded Linux. Of the 268 valid responses, more than half came from readers of LinuxDevices.com — thanks! The survey was online from November 2003 to March 2004, and people from 39 different countries took part. Descriptive results are summarized in a paper that you can download. We will discuss the most important results in the following, and also mention a few regression analyses not contained in the paper.

In our sample, 15 percent of participants identified themselves as “hobbyists.” Given that a hobby programmer is more likely to take part in a survey than a professional developer busy to meet a deadline, the share of hobbyists among all embedded Linux developers is quite certainly below 10 percent. So indeed, this software comes from commercial firms much more than from hobbyists. On the other hand, the distinction between hobby and profession is not that clear: most programmers working on embedded Linux as part of their job continue to work on it in their spare time.

Now, how much of what people develop for embedded Linux is made public? Since much of this code is so specific to a particular device that nobody except the device manufacturer would care about it, we asked for “embedded Linux developments by your firm that are potentially useful for others.” Among 176 respondents, the average amount of code revealed was 62 percent. (Figure 1). Leaving out universities and hobbyists, this number drops to 49 percent, but this is still a lot: consider that device manufacturers, the largest group among the participants, usually come from a proprietary culture and are used to revealing none of their developments.

The obvious next question is, has the amount of code that is revealed changed over time? In 1999 and 2000 there was a lot of hype around open source, and one might suspect that revealing has decreased since then — and might decrease further in the future. Surprisingly, the contrary is the case: only 10 percent of 142 respondents working for commercial firms said that their firm currently reveals less than in 2000, while 41 percent found the amount unchanged and 49 percent said it had increased (Figure 2). Hence, the fear that embedded Linux becomes ever more proprietary with less and less code being revealed seems unfounded.

An interesting detail is that the increase in the amount of revealed code was strongest for component manufacturers, where 12 of 13 respondents saw an increase. This finding confirms statements from interviews we had conducted with industry experts: buyers increasingly put pressure on manufacturers of components to provide the source code for drivers. Manufacturers seem to yield to this pressure, or recognize on their own advantages of revealing their source code.

So, firms do reveal a lot of code for embedded Linux, and this amount has increased since 2000. But what are the reasons for them to do it? Is it the hope for external development support, marketing intentions, or the individual programmer's open source enthusiasm? We offered a number of potential reasons to participants and asked for their agreement on a five-level scale. For hardware manufacturers (i.e., devices or components; for others, see the paper), the statement “my company reveals code because the GPL requires it” met with the highest agreement. That was to be expected — after all, promoting the sharing of code is the explicit intention of the GPL. However, next follow reasons that have to do with external development support: “others make bugfixes and reveal them”; “it reduces our maintenance effort when the code becomes part of the standard code base”; “others develop the code further and reveal their improvements in turn”; and, “we want to appear as a good OSS player.” The last one is related to external development support, since that is a precondition for receiving such support. Hence, it becomes clear that, despite the fact that most contributions come from commercial firms, an informal development collaboration between embedded Linux developers does take place, and firms seem to benefit from it.

The above is not to say, though, that revealing source is a pure business decision. Among the developers' personal reasons to reveal code, the reason “I consider it fair to give back to the community” met with by far the highest agreement. So, a sense of community does very well play a role.

Now let's zoom a bit into the picture. The average numbers given above for the share of revealed code hide the fact that, for each category of firms, individual responses vary widely, between 1 percent and 100 percent (Figure 3). Most of this variation can not be explained with our data — we do not know enough about the type of code the firm developes, nor about the attitudes of management towards open source. However, some preliminary results from regression analysis are available. For example, the firm's experience with embedded Linux, measured in years, has a significant positive influence on the share of code that is revealed. Apparently, it takes some time for a commercial firm, typically used to keeping private whatever its engineers develop, to become familiar and comfortable with the open source model. Also, a variable that measured the developer's enjoyment of programming and pride in their work has a significant positive effect. Not surprisingly, a restrictive firm policy with respect to revealing code has a significant negative influence.

It is impossible to discuss all aspects of the survey in this article; please refer to the paper. However, before closing, we would like to address the issue of firm policy towards open source participation (again only for the largest group, hardware manufacturers). Respondents were offered five statements relating to firm policy, and were asked to tick those they agreed with. About 10 percent each stated that their company was either “very restrictive in revealing code,” or that, in contrast, it “encourages” revealing. Thirteen percent ticked “It is my responsibility to reveal the source code,” and 18 percent ticked, “I make suggestions as to what we could make public, and discuss each case with my supervisor.” These two statements underline the role of the individual programmer as the person who decides about revealing, or at least does the lobbying for it. The highest agreement, however, was with the statement, “There is no official policy,” with 24 percent. Given that active participation in the open source process can entail considerable benefits but also risks, thinking about a consistent firm policy on this issue would be time well spent.

To conclude, we found that firms developing embedded Linux do make a considerable share of their developments voluntarily public, and derive various benefits from it. The code they reveal is typically generic, but can also be specific and differentiating. Now, the operating system is typically not what differentiates an embedded device — differentiation instead comes from the hardware or the application software. Hence, the revealing of code for the operating system is certainly less surprising than that of application code. Nonetheless, sharing is anything but normal — despite the fact that the open source process can yield considerable gains in development efficiency, with limited risk of losing competitive edge (at least, on the level of the operating system). But, voluntary revealing is not something commercial firms are typically good at. As one participant put it: “It is new territory for this company. The spirit is willing, but the legal staff is weak.” We recommend that firms using or considering embedded Linux analyze the benefits and risks of active participation in open source development processes, and develop a consistent policy on this issue. There are some risks, but benefits clearly dominate.

Complete survey results are available for download, here.

---

About the authors

Joachim Henkel is an assistant professor at the Institute for Innovation Research, Technology Management and Entrepreneurship at the Ludwig-Maximilians-University Munich (LMU). His research areas are open source software, user innovation, venture capital, and e-payment. In 2002, he spent six months as visiting scholar at MIT's Sloan School of Management, where he initiated a research project on embedded Linux. Joachim received a degree in physics from the University of Bonn and a Ph.D. in economics from the University of Mannheim. After his Ph.D. in 1997, he worked for two years with the consulting firm Bain & Company before returning to academia.

Mark Tins finished his studies of economics and business management at the University of Munich with this research work about embedded Linux. His main subjects were Innovation Research, Technology Management, and computer science. As an exchange student, he spent a semester in Ohio/USA, and one in Belgium. He is now searching for a job in the area of "New Technologies.”

The survey authors can be reached at joachimh at mit dot edu, and mark dot tins at campus dot lmu dot de.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.