Most LinuxDevices.com readers are well aware of how difficult it is to judge the utility of competing software development methods in objective and quantifiable ways. The proliferation of languages, software engineering methodologies, and tools suggests that it is not easy to determine which way is best — clearly, if one way were verifiably best, in a results-oriented field the inferior methods would… quickly be culled out in favor of the one superior method. We spend considerable time experimenting to find our own best tools and methods; few of us would contend that these decisions can be determined by purely objective means.

Consider, then, the difficulties faced by Jerry Krasner, who recently published a report on a study intended to determine whether Linux or Windows is the better operating system for developing embedded systems. This determination had to be conducted and reported in a way that would convince developers of the report's validity, yet with public knowledge of the fact that Microsoft had paid for both the study and the report.

As we shall see, this task proved to be formidable.

In writing a rebuttal to the report, the first question that arises is: what to rebut? This twenty-eight page report attempts to be at least three things:

• As its title suggests, it attempts to describe a framework for evaluating embedded development platforms. As we shall see, the framework as defined is not very useful.
• It reports on data purporting to show Windows superior to Linux as an embedded system development platform.
• Considerable space is expended on qualitative discussions of Linux vs. Windows and open source vs. proprietary software. Nothing new is presented here, and this article will say no more about this material, except when it contradicts other more substantive material.

Taking a look at the framework, the first thing that strikes me is how narrowly it is focused. It attempts to measure two factors in embedded system development: time to market (TTM) and total cost of development (TCD), defined as developer months times an assumed developer monthly cost. The framework also considers “Associated Costs” (AC), which is essentially the cost of the toolkit used. TTM and TCD are probably most important, with the study's TCD figures prompting a rash of press headlines of the form “Linux is 4X More Expensive Than Windows!”.

Now, let us start with a basic principle: a framework for evaluating embedded development platforms must properly measure product development success. TTM and TCD are indeed very important measures of success. But there are many more:

• Product differentiation. A quickly and cheaply developed product that is no better than competing products already on the market is seldom successful.
• Product cost. An operating system that can be deployed on cheaper hardware will have a competitive advantage over those that require higher-end hardware.
• Toolkit continuity. Many organizations strive to develop a platform over which they have control and which they can use on multiple devices and multiple versions of the same device. An operating system and toolkit that permits this is advantageous.

And so on. No framework that measures only a subset of these factors can be considered useful. It could be, as John Lettice has pointed out, that developers of larger, more innovative, products tend to choose Linux because of the control and flexibility it offers, while developers of cookie-cutter devices tend to choose Windows because of the help it provides. It could be that developers of high volume (and hence especially cost sensitive) devices tend to choose Linux because they believe it can be deployed on cheaper hardware; such development efforts typically trade off higher engineering costs in order to attain lower production costs. It could also be that larger organizations (with larger, perhaps less efficient engineering teams) lean toward Linux because they want toolkit control and continuity; while small, highly-motivated, and hard-working startup companies lean toward Windows because it helps them get their product to market quicker.

I don't know if any of these speculations is accurate, but no framework that fails to account for all of the factors determining development success can be considered persuasive. And since these factors are often quite nebulous and hard to quantify, the prospects for devising a truly persuasive framework are not good. One should not be surprised by this. Engineering productivity is notoriously difficult to measure, and this framework is, after all, an attempt to measure the effect of toolkit choice on software engineering productivity.

So Dr. Krasner's framework is unpersuasive. A look at the way in which the study was conducted, as well as a quick look at the data, also reveal some defects.

First, it is hard to take seriously a study that was funded by an organization, in this case Microsoft, with a significant stake in its outcome. One can only guess how many previous studies were conducted, then quietly dropped because the results were not favorable to the sponsor. Second, the survey was conducted by a market research firm that Krasner and Microsoft will not disclose. Conducting market research properly requires qualifications and experience that not all organizations have. We are not allowed to judge whether the company that conducted this survey was qualified to do so. Third, the projects studied for TTM and TCD were chosen in a decidedly non-random way; they were chosen from press releases published at LinuxDevices.com and WindowsForDevices.com, and from references supplied by the vendors. One can hardly expect this selection method to produce a sample fully representative of embedded systems. Fourth, little detail about the raw data was provided, allegedly for reasons of confidentiality. Finally, the survey questions were not disclosed, so one cannot judge whether the survey contained leading questions. Overall, it seems clear that only those already inclined to accept the reported results will find this survey persuasive.

It is difficult to address the data for TTM and TCD directly as only the sketchiest outline of the data is provided. But the study's very results argue against its validity. Consider that in most embedded software development efforts, only a small portion of time is spent on platform issues. In virtually every project I've been associated with over many years, one engineer has selected the development environment, brought it up on the target hardware, and introduced the other engineers to its use. From that point on, everyone involved is focused on the application rather than the environment. Platform issues constitute only a small proportion of the effort expended on all but the simplest cookie-cutter devices. An exception to this rule is the environment's debugger, which indeed can be expected to have an impact on productivity during the application development phase. But debuggers on most platforms have long since reached the point of maturity and usability. This includes those available for Linux. One can argue about the learning curve associated with specific debuggers (and the slickness of Microsoft's IDE is obvious compared to, say, GDB with DDD; though other front ends including KDevelop are comparable to Microsoft's) but I, for one, have yet to encounter a single graphical debugger that I could not start using productively within a few hours. Unless you are developing something truly remarkably simplistic, these few hours will not be a significant factor in your product development time.

What, then, are we to do with a study purporting to prove that the choice of Linux over Windows results in nearly four times the development cost? Experienced embedded device designers know intuitively that you simply cannot get this result by choosing one development platform over another. I have worked with some truly brain-damaged development platforms over the years, but not one that I could not eventually become accustomed to and begin focusing on application development. The platform choice is simply not important enough to give a 4X difference in productivity; at least if we are, as in this case, comparing 32-bit platforms that provide high-level languages and source code debugging.

When a study produces such wildly improbable results, the study organizers should question whether it was conducted correctly. At the very least, the strangeness of the results should be acknowledged in the writeup and some attempt at explanation should be made. It is puzzling that Microsoft and Dr. Krasner apparently failed to realize that the reported results would offend the intuition of experienced embedded device developers.

TTM and TCD are probably the most sensational results, and these are the hardest to verify because survey information and raw data were not provided. On the other hand, the methods for obtaining and analyzing the data for AC, as well as the raw data itself, are supplied in the report. Thus it is possible to critique this part of the report in more detail.

Dr. Krasner's handling of Associated Costs is insidious as it uses the very flexibility of Linux against itself. As LinuxDevices.com readers know, a wide range of options exists for constructing embedded Linux systems, from do-it-yourself approaches using freely available tools and components, to all-encompassing commercial IDEs similar in many ways to Microsoft's products. In section II Dr. Krasner presents the familiar discussion of the open source software development model, in which the user gets freedom and source code, but takes on the responsibility of choosing components and supporting them. The author correctly points out that many developers would prefer to focus on their application, relying on a vendor to handle the environment.

In this way, Dr. Krasner moves the report from its advertised focus on measuring toolkit productivity to the familiar and tired debate about open source vs. proprietary software, with Linux in the open source role and Windows in the proprietary role. The gist of the discussion is that a developer must choose Windows, and be supported by Microsoft during development; or Linux, and be on your own. This is clearly a false choice — fully supported toolkits are available for deploying embedded Linux systems. So the entire discussion of supported vs. unsupported products is misleading — only Linux provides both options, and only Linux provides the fully open source option for those developers who want it.

So in section II the author assumes that all Linux use is DIY and thus unsupported. Then in section IV he presents the concept of “Associated Costs”, which are the costs of the tools used. One might suppose that, having assumed all Linux use is DIY, he would report the cost as zero. But no, he averages the costs of the commercial distributions and reports that figure. Apparently, for the purposes of determining the level of support all Linux users are DIY, but for the purpose of determining tool costs all Linux users purchase commercial supported distributions.

I cannot imagine why such a quirky, obviously incorrect method for determining AC was used. In comparison to TTM and TCD, the cost of the tools used should be fairly straightforward to determine — ask the developers what they used and how much they paid. This approach would still have the limitations often associated with such surveys: it relies on the memory and truthfulness of the study subject, and the subjects themselves are selected in a non-random way. But within these limitations, it would have been possible to get an idea of the average cost of a Linux distribution used in the projects surveyed. Also, it is well known that the prices of high end specialized software packages are usually negotiable. Asking the users what they actually paid would be more reliable than using list prices. But the author constructed a list of embedded Linux distributions, then simply reported the average list price of the products on his list. Furthermore, he made no attempt to normalize this price according to the number of survey projects using the toolkits. I suppose I could release my own toolkit for developing Windows-based devices, price it at $19,005, average this cost with Microsoft's $995 product, and report that the average cost of embedded Windows development kits is $10,000, undeterred by the fact that Microsoft has 100% of the market and I have zero.

Further damaging is the sloppy data acquired for the Associated Costs determination. Included in the list of embedded Linux distributions is LynxOS, which besides being the most expensive toolkit listed also has nothing whatever to do with Linux. Experienced embedded system developers know that LynxOS is a conventional proprietary real-time operating system that predates Linux. Also listed are two versions of Hard Hat, though MontaVista's products have been called simply “MontaVista Linux” for a long time; this simple mistake calls into question what products are actually being referred to, and whether the pricing data is current. Finally, Trolltech's Qtopia SDK is listed, though Qtopia is an application framework, not an embedded Linux distribution.

Similar problems are found in Dr. Krasner's analysis of runtime component technology costs, as Rick Lehbaum pointed out in an earlier critique. Most or all of these categories have good open source alternatives available, but few are listed. Indeed, all of the categories have numerous commercial options available — the options listed cannot be considered even a start. For example, various ways of obtaining real-time performace are available beyond the FSMLabs options listed. And again, Dr. Krasner inexplicably listed LynxOS, which is indeed a real-time kernel, but has nothing to do with Linux.

I won't address every category specifically. Suffice to say that the world is awash in good open source web browsers, PIMs, encryption libraries, and so on. The component classes chosen for comparison have a clear PDA emphasis, so I only point out that many of these needs can be met with nothing more than the excellent Opie project, the fork of Qtopia that contains good PIMs, multimedia decoding, the Konqueror browser adapted to a PDA form factor, and more.

It is clear, then, that the small amount of verifiable data in the report has some significant flaws. The Associated Costs determination is fundamentally flawed both by its methods and its execution. The vulnerability of this part of the survey calls into question the validity of the TTM and TCD part, as one can hardly assume that those poorly documented parts were carefully and correctly designed and executed when the AC data is so easily shown to be invalid.

As we have seen, the Microsoft-funded study reported by Dr. Krasner is flawed on just about every level we examine it. First, it presents itself as a “cost estimation framework for evaluating embedded development platforms”, but spends most of its time repeating tired open source vs. proprietary software arguments rather than developing the framework. Second, it is no use “evaluating embedded development platforms” based strictly on “cost estimation” because time-to-market and cost are not the only determinants of success. Also, the time-to-market and cost data is impossible to verify and provides results that are wildly improbable, calling into question the validity of those results. Finally, this perception of unreliability is increased by looking at the small amount of verifiable data, which is easily found to be full of errors.

---

About the author: Jerry Epplin is Technical Editor of LinuxDevices.com and principal of EmbeddedSpace, an embedded systems consulting company. He's been playing with and working with Linux since . . . uh, well, . . . he's not sure when, but you didn't have to choose between KDE and Gnome back then because neither existed.

---

Add your voice to this discussion!

---

Do you have comments or questions on this story, or on the EMF report?

talkback here
But please, before you jump into this discussion, read the full report! You can download it from the EMF website (requires free registration).

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.