Article: Reflections on the EMF embedded Windows vs. embedded Linux report
Aug 7, 2003 — by LinuxDevices Staff — from the LinuxDevices Archive — viewsEarlier this month, market research firm Embedded Market Forecasters (EMF) released a report which claims embedded development projects based on Microsoft's “Windows Embedded” operating system platforms (specifically, Windows CE .NET and Windows XP Embedded) are completed 43% faster and at 68% lower cost, on average, compared with similar projects… using Embedded Linux. The study, titled Total Cost of Development: A comprehensive cost estimation framework for evaluating embedded development platforms, derives its quantitative conclusions from a cost-based framework for comparing embedded operating system development alternatives that was developed by the report's author, Dr. Jerry Krasner.
The report includes data from a survey of 100 manufacturers using 32-bit processors in a range of embedded projects and applications — 50 using various implementations of embedded Linux, and 50 using Microsoft's Windows Embedded platforms (Windows CE .NET and Windows XP Embedded). The devices and applications included in the source data reportedly covered consumer electronics, handheld computers, industrial controllers, network gateways, point-of-sale kiosks, set-top boxes, thin clients, and others. The report estimates “total cost of development” for each project by multiplying the average embedded design project time-to-market by the software engineering team size and cost.
Origin of the report
The development of the cost measurement framework and the writing of the report appear to have been largely funded by Microsoft. Krasner said Microsoft paid him to develop the framework and analyze the data, and that the report is based on data collected by an undisclosed, but “reputable”, third-party market research firm.
“Microsoft paid me for my time to develop the framework and to analyze the data from a very reputable, very large, third party,” Krasner said.
Krasner said Microsoft approached him about producing the report because he had been publishing anecdotal evidence for some time that indicated that the total cost of using Linux in embedded projects was much higher than was generally understood.
Krasner said his sense that using embedded Linux might be more costly than using embedded Windows grew from a number of conversations with company executives who were disappointed with their experiences with using (or trying to use) embedded Linux in their development projects. In attempting to quantify those executives' concerns, and provide what he felt would be a better way to show the true project development costs, he developed the “Total Cost of Development” framework, Krasner said.
My understanding is that Microsoft initially worked with the “reputable”, but undisclosed, third-party market research firm to develop a 100-question survey, and the research firm conducted the survey and compiled the responses. That data was then given to Krasner, for use in his report.
Executive summary — download of full report
An executive summary of the report along with a link to download the full report (free, but requires registration), is available here, at WindowsForDevices.com
I have attempted to be objective in my review of the EMF report. In reviewing the report, I arrived at the conclusion that it does not treat embedded Linux in a fair and balanced manner. Many of my reasons for reaching that conclusion are given below, in the form of a series of questions, concerns, and comments based on my review of the EMF report.
I welcome the addition of opinions from both individuals and companies in the embedded market — on all sides of the issue — via the talkback thread which is provided at the bottom of this article. Additionally, guest editorials are invited, and will be reviewed for possible publication on these pages.
Now, for my reflections on the EMF report . . .
Why is the title of this document “Total Cost of Development: A comprehensive cost estimation framework for evaluating embedded development platforms”, when it's primarily a report comparing Windows to Linux in embedded development (using the TCD framework)?
Less than a quarter of the document deals with the proposed Total Cost of Development framework, with much of the contents of even those pages taken up with discussion about open source and embedded Linux. I don't know about you, but I tend to become really skeptical about the reliability of the contents of “research reports” that purport to be something other than what they are — it makes one tend to doubt the objectivity of the report. Why not simply use a title that makes it clear what the report is attempting to do, like: “A comparison of 'Total Cost of Development' for Windows Embedded and Embedded Linux”. Or even: “Debunking the Myth of 'Free' Software Using a New 'Total Cost of Development' Metric” It's really not about “Total Cost of Development” in general, as indicated by the title. Why not just name it like it is?
Why does the report not state right up front that it was funded by Microsoft?
When I first saw the report, I called Krasner on the phone and asked him whether Microsoft had funded the report and the surveys on which it is based — to which Krasner replied that Microsoft had paid him to develop the framework, analyze the data, and write the report. He also explained that Microsoft had hired a “highly reputable” research firm to collect the data for computing TCD values for the opposing OSes (excerpts of the data appear in Appendix A of the EMF report).
Surely, the name of the company which devised and funded the study was worthy of mention in the report, particularly when that company is clearly the primary beneficiary of the report's conclusions.
Why does Krasner dismiss, without explanation, the royalty costs for XP Embedded, which represents half of the Windows Embedded data on which the principal finding of the report is based?
Krasner inserts the following interesting simplification into his analysis in section of the report titled “Runtime Royalty Cost Considerations”:
“For the purposes of runtime royalty comparison, only Windows CE .NET and embedded Linux will be considered” (Pg. 15, first paragraph)
How can this be justified, when the study is clearly positioned as a comparison between embedded Linux and Windows Embedded including both CE and XP Embedded?
The study frequently states that it is comparing both Windows CE .NET and Windows XP Embedded. Specifically, 50 embedded Linux projects are compared to 50 embedded Windows projects that consist of 25 CE .NET projects and 25 XP Embedded projects. But when it comes to a discussion of royalties, the comparison with embedded Linux is restricted to CE .NET, whose pricing was recently drastically reduced by Microsoft to bargain-basement by embedded OS norms (presumably to compete with embedded Linux) — to a quantity one price of just $2.60 (in 10,000 unit volume) and around $9 (10,000 unit volume) for the “Pro” full-up version (Pg. 16, Table 6).
Why did XP Embedded magically disappear with the waiving of the “for the purposes of runtime royalty comparison” wand? Why not instead eliminate CE instead of XP Embedded from the runtime royalty discussion, since the functionality of Linux is actually more on a par with that of XP Embedded rather than the more restrictive CE .NET? After all, according to Microsoft, CE has around 300 components whereas XP Embedded has nearly ten thousand — making Linux more comparable with the much more expensive XP Embedded, for many applications. Interestingly, the royalty cost of XP Embedded does not appear anywhere in the report, even though that OS contributes fully 50% of the Appendix A TCD data for Windows Embedded, on which much of the report is based.
Regarding XP Embedded royalties, I recently asked Microsoft what a “typical” royalty price for XP Embedded was, in volume. They declined to give specific pricing, but did say that it runs roughly around $100 per system. (Interesting observation: if you use the fact that CE contains 300 OS “components” whereas XP Embedded contains around 9000 components, the resulting 30:1 ratio applied to royalties would predict a $90 for XP Embedded vs. the $3 price of CE core. I wonder how many “components” Linux has?)
Krasner (and Microsoft) would probably hasten to point out that the heart of the report — the comparison between the “Total Cost of Development” of embedded Windows with that of embedded Linux — doesn't depend on the royalty rates; and that would be correct. So why does Krasner weaken the objectivity of his report by summarily throwing out the fact that half of the data behind the heart of the report is associated with royalties that are 3,000 percent higher than the other half (i.e. XP Embedded is ~$90 vs. ~$3 for CE).
Why are components like real-time, web browser, and others shown as requiring royalties for embedded Linux, when free versions of those components (many more) are available?
Real-time and web browsing alone account for $17 worth of royalties in otherwise royalty-free embedded Linux devices, making embedded Linux more expensive than CE .NET. But is it true? Absolutely not! There are multiple free sources of real-time functionality (including the preemption option available for use with the standard 2.4 and later linux kernel), and multiple free browser solutions; plus, ones you can license and pay royalties for — take your choice! One naturally wonders: How many of the other royalty-bearing components required by embedded Linux in Table 6 (page 16) are also bogus?
Does it seem reasonable that Table 6 exclusively lists functions that are contained within Windows CE and not in Linux, and does not show any Linux functions that are missing from Windows CE?
To understand what I'm driving at, consider the fact that in promoting both of its Windows Embedded OSes, Microsoft notes that Windows CE has around 300 functions, as compared to around 9,000 components in Windows XP Embedded. Does it seem reasonable that every useful software function contained in Linux is contained within the 300 components of Windows CE? Wouldn't you expect table entitled “Embedded Technology Component Pricing” to include at least some Linux functions that aren't in Windows CE?
Of course there were five canceled Linux projects, and no canceled Windows projects, in the sample! Here's why . . .
The report states that four sources were used by the survey-takers to locate embedded Linux and embedded Windows projects: news releases or listings at LinuxDevices.com and WindowsForDevices.com, and reference customers from embedded Linux distributors and Microsoft. But isn't it the case that only LinuxDevices.com currently provides a deep archive of news and articles pertaining to projects that have not been successful? Wouldn't that naturally skew the data on canceled projects for the sampled data? Of course, WindowsForDevices.com will eventually also provide significant archival data on canceled projects, but the site isn't even one year old — in contrast to its older embedded Linux sibling which has been around since 1999 and boasts an archive of some 5,000 stories.
Does Section III really provide a balanced comparison of development environments?
In Section III, Krasner sets out to define the development environments and tools needed for embedded system development, and purports to summarize what is available from both Microsoft and the embedded Linux market. He points that “Many RTOS companies have spent considerable time in assembling third party and custom designed tools into custom IDEs to support their RTOSes,” and that “this is also true of the Windows Embedded operating system development tool chain and the windows Visual Studio .NET application development environment.” But what of the embedded Linux offerings? For these, Krasner simply dismisses them with a comment that “although tools to develop Linux applications are readily available, there are few true integrated development environments for Linux,” and adds that “there are even fewer that take into consideration the unique requirements of the embedded designer.”
But hold on a moment! Just how does “few” and “fewer” equate to less than one? That is, first Krasner says Microsoft has an IDE and an application development environment; and then he says there are “few true IDEs for Linux” of which “even fewer” are intended for embedded development, implying that some are for embedded development. Help me out here: just how does Microsoft come out ahead of the embedded Linux market via that logic?
Following that solid line of reasoning, Krasner proceeds to say “This can be a significant issue in creating a productive and supportable development environment for application design under Linux.” Really? How does that follow from the prior faulty logic?
Regarding training and tools
Later on in Section III, we learn that “There are many tools and training programs available for developing Windows Embedded applications,” followed by some of the “many” items available from Microsoft being listed. This is compared to embedded Linux support available from Red Hat and MontaVista, as well as this interesting tidbit: “The Free Software Foundation offers bundled pre-compiled GNU/Linux product development packages from its web site for purchase and download. These packages include the GNU/Linux source files and GCC compiler and debugger tools.” (Really? Correct me if I'm wrong, but I wasn't aware that the FSF was actually in the business of distributing or selling Linux software.)
But what about all the other embedded Linux vendors, training shops, and tool vendors? I can't help wondering if the story might have been told very differently given a bit more research. Remember that website from whom the information about embedded Linux projects was obtained? I think you'll find hundreds of training, support, tools, classes, books, and other support to be available to stack up against Microsoft's limited support. Well, that's my opinion, but I'm not a research analyst.
One interesting table missing from the report is a comparison of the number of BSPs available for Linux vs those available for either CE or XP Embedded (or both together). Add them up, including all the embedded Linux vendors currently in business as well as the active free projects, and I think you'll find a huge advantage for hardware support in favor of embedded Linux. But instead, all we get from this report is that one Microsoft toolkit/IDE is “more” than a few or less-than-a-few toolkits/IDEs from the embedded Linux market.
“Let's compare typical development environments for Windows embedded and embedded Linux”
Section III then sets out to paint the picture of the Windows Embedded vs. Embedded Linux development environments. But what we get is simply a comparison between Microsoft's sophisticated IDE and build environment (really, it is quite good), with the freely downloadable GNU toolchain and associated tools. But when you use Linux in an embedded application, who says you're limited to what is freely downloadable? Krasner certainly doesn't think that's the case — which is why pages 12-14, Tables 4 and 5, and Appendices C and D are chock full of information on the “associated” costs of using FSMLabs, Lineo, LynuxWorks, Microcross, MontaVista, Red Hat, Trolltech, Viosoft, and others. But why aren't the IDEs and build environments of these companies, plus the ones of Metrowerks (Codewarrior) and TimeSys (based on Eclipse) included in the comparison of “typical development environments”. Perhaps it's because the typical projects used for the TCD calculation were largely based on the results of roll-your-own embedded Linux approaches rather than on commercially supplied embedded Linux. (More on that later.)
And so on . . .
Section III proceeds to compare “average” costs of Microsoft vs. embedded Linux tools, using Microsoft's well-publicized $995 tools model in comparison with “vendor provided, or published, terms and conditions.” The data is more thoroughly tabulated in Appendix C, and makes an interesting read. One wonders how up-to-date and complete the information is, after noting the inclusion of Lineo, the absence of Metrowerks, the absence of TimeSys, and the absence of a host of others.
Admittedly, there is a large and continually growing number of commercial and noncommercial software, tools, and support alternatives available in the embedded Linux space. Nevertheless, it's important to not skimp on the research, especially when the very presence so much support a single embedded OS is one of the significant advantages of that OS, and is frequently cited as a key reason companies and their developers are drawn to embedded Linux.
The big picture . . .
But enough of the small talk. When I stand back far enough, and look at the report as a whole, I see two big things: (1) the basic framework for evaluating “embedded development platforms” (taken from the title), i.e. the “Total Cost of Development” (TCD) model, that Krasner has put forward in the report; and (2) the actual survey data used to compute TCD for the two opposing embedded OSes considered in the report (i.e. the numbers for time-to-market and average number of software developers during the TTM).
There are many comments you can make when you look at the survey data tabulated in Appendix A-1, A-2, and A-3, and summarized in Table 7. A few come to mind, including: “Why would anyone use embedded Linux when their projects are likely to cost three times as much as with embedded Windows?”, and “You've got to be kidding — do you expect me to believe that?”
But my big concern is that the data used for the TCD comparison is based on an unfair comparison between embedded Linux and Windows Embedded. Here's why: The embedded Linux projects consist of the combination of projects based on noncommercially downloaded and unsupported sources (consisting, themselves, of a mixture of standard linux implementations and ones oriented toward embedded applications), along with projects that were based on commercially supplied and supported IDEs and toolkits (e.g. FSMLabs, MontaVista, Lineo, etc.). In contrast, the Windows embedded projects were all accomplished using Microsoft's commercially supplied/supported IDEs/toolkits.
Data gathered by LinuxDevices.com indicates that over half of all embedded linux projects are being accomplished without the benefit of commercial embedded Linux IDEs/toolkits and support. I thus suspected that this might be the case with the projects included in the data of Appendix A-3, and therefore requested clarification from Jerry Krasner, the author of the report. Krasner kindly provided the following pair of responses to my request:
- Initial response: “Of the 45 embedded Linux OEM respondents: 29 got their copies by download — 17 from company sites (e.g., Red Hat site, SGI site, Arm Linux site) and 12 from non-commercial sites (e.g., kernel.org); 18 OEMs purchased their version directly from a Linux vendor.
- Later, additional response: “Of the 45 OEMs using Linux 12 rolled their own and started from a generic distribution from places like kernel.org or from a silicon distributor. The rest used a commercial distribution of Linux (2.4 and later except for 2 cases which used the 2.2 kernel). CE was versions 4.0 or 4.1 and XPe was just XPe without any service pack releases. Looking at the data rows (which are unfortunately confidential under NDA), it actually appears as if the commercial distributions did achieve a faster time to market (13.3mo.) vs. the roll your own (15.9 mo.) but required more software engineers on average (16.4 vs. 10.8). This netted out to a slightly higher TCD for commercial over roll-your-own (218 vs. 172 DMM). It's hard to draw conclusions herein, especially since commercial distributions tended to be used on more difficult types of designs.”
I'm not at all surprised at the first breakdown, namely that 29 of the 45 projects obtained their embedded Linux software by downloading it from either company sites or “non-commercial” sites. This tracks well with the findings of LinuxDevices.com, that more than half of all embedded Linux projects are conducted without the benefit of commercially supplied/supported IDEs/toolkits.
Additionally, consider the idea that these projects were probably kicked off, on average, nearly two years ago, and possibly longer ago than that. (Add up the time from starting the project, to publication of the listings or customer reference stories, to the collection of data by the survey takers, to publication of the report.) Clearly, even the projects that were conducted using commercially supplied/supported IDEs/toolkits were probably using relatively immature development environments and toolkits. Don't forget, commercial embedded Linux has only been around for about three and a half years. So many of these projects, particularly the ones with the worst TCD values, were likely to have started right around (or even before) the dawn of the embedded Linux age!
Conclusions
My intent is not to diminish the embedded products that Microsoft has developed. They offer one of the finest IDE/toolkits available today. But the comparison of embedded toolkits and OS alternatives demands thorough, balanced research that remains on target and makes apples-to-apples comparisons. In my opinion, the EMF report, in its current form, is neither thorough nor balanced, and its conclusions are flawed.
But there is some potential “good”. The report strikes me as most interesting from the perspective of the proposed TCD model, along with the challenge to figure out whether embedded Linux development takes longer and requires more developers than does development with Windows Embedded OSes. While this noble goal is not achieved, other researchers may want to consider the TCD model in their work.
Update: Subsequent to publication of this article, we received a two-page statement from Dr. Jerry Krasner, author of the EMF report, in an email titled “Microsoft/EMF position statement.” That statement is available here.
To provide added perspective, this section offers a roundup of additional articles published on the web which relate to the EMF report . . .
- VDC whitepaper compares Linux, Windows embedded development
- Developer rebuts MS-funded embedded Windows vs. Linux study
- LynuxWorks challenges findings of EMF embedded Windows vs. embedded Linux report
- Windows device development faster, cheaper than Linux?
Be sure to check back periodically for the latest updates.
Do you have comments or questions on this story, or on the EMF report?
But please, before you jump into this discussion, read the full report! You can download it from the EMF website (requires free registration).
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.