Embedded Report Riles Linux Community
Sep 3, 2003 — by LinuxDevices Staff — from the LinuxDevices Archive — viewsThe following article has been reproduced with the permission of Software Development Times . . .
Developers claim EMF's Microsoft-funded study spreads more “half-truths”
It's embedded Windows versus embedded Linux, round two. Shortly after publishing a fiercely contested report in January 2002 that one Linux developer at the time characterized as "marketing-generated half-truths," Microsoft Corp. funded a new study comparing its Windows CE and Windows XP Embedded operating systems with Linux.
Not surprisingly, the new study shows that embedded projects using Microsoft's operating systems are 68 percent cheaper, and are completed in 43 percent less time than projects using Linux. Reaction from the Linux community has been loud and sustained–again no surprise–and mainly alleges flawed logic and research methods and blatant omission of relevant data.
The report, titled "Total Cost of Development: A comprehensive cost estimation framework for evaluating embedded development platforms," was published by Embedded Market Forecasters (EMF), a market-research firm based in Framingham, Mass.
Report methodology
According to EMF, the report results from a survey conducted between March and October of 2002 of 100 "randomly selected OEMs that had chosen either Windows embedded or embedded Linux across a wide range of device types and applications." All projects involved 32-bit architectures.
Exactly half of respondents were using various distributions of embedded Linux, the report said, and of the remaining 50, 30 were using Windows CE 4.0 or 4.1, and 20 were using Windows XP Embedded. Few additional selection details were provided. Total cost of development was calculated by multiplying the project's time-to-market by the size and cost of the engineering team, excluding hardware designers and testers. A separate EMF survey queried 456 embedded developers about project starts, cancellations and the time in between. Results were cross-tabulated according to operating system for analysis.
Report conclusions
The report concluded that Linux projects require an average of about 14.3 months to complete, while projects using embedded Windows finish in about 8.1 months. Further, the study showed that the cost of developing a device with the open-source operating system was roughly US$1.5 million, while Windows-based projects came in at around $479,000 on average. The cost per developer was determined by polling embedded development recruiters.
“Total Cost of Development” by platform, Embedded Market Forecasters
Linux experts refute, Microsoft defends report
Referring to EMF's findings as incomplete, Bob Morris, vice president of sales and marketing at Linux developer LinuxWorks Inc., said in a prepared statement that the research failed to consider several major factors that impact product design. "For instance, the report does not analyze projects according to their size and complexity–two factors that significantly affect time-to-market and overall cost," he said.
More pointed were observations by Rick Lehrbaum, executive editor of ZDNet's LinuxDevices.com, a Linux community Web portal from which many of the Linux OEMs were selected for the survey. "The major problem with the report is that it paints an inaccurate picture due to insufficient and old data. It may be an accurate representation of the data points, but I think the data points are the problem," he said.
Lehrbaum said he believes the report suffers from a failure to demonstrate a causative relationship among the sample project types, and the false conclusions that inevitably result. "If you take 1,000 samples of Linux and Windows projects, and the Linux projects take longer to get to market, it doesn't prove that Linux takes longer, but only that those projects were more complex."
"We feel that [such] conclusions would be consistent with the EMF report," said Todd Brix, lead product manager for Microsoft's Embedded Devices Group. He defended the study and its selection methodology. "Many OEMs are not comfortable revealing their selection of embedded operating system. It is common practice across the industry [therefore] to measure a product's progress based on public information. A variety of project sizes and devices with varying degrees of complexity were included in the study, [ranging] from consumer electronics to handheld computers and point-of-sale terminals to set-top boxes." The report does not describe the precise selection method used.
According to Lehrbaum, two of the primary sources for developers to survey were the 3-year-old LinuxDevices.com, and comparative newcomer Windows4Devices.com portal, which is about a year old. The other two sources were vendor references and news releases. The depth and scope of sample developers, he said, were therefore compromised because "at LinuxDevices, we [report] cool projects. It doesn't matter if they worked or didn't; we find out what they tried and we tell people." Three of the four other sources don't do that, he added, "so there was a major contribution on [only] one side of the issue."
Again Brix defended the report's methodology. "According to EMF, a sample size of 100 is statistically sufficient to draw the broad and significant conclusions reached in this study of a large comparative difference in both time-to-market and in total cost of development between embedded Linux and Windows Embedded."
Acknowledged omission
Another sticking point was the omission of XP Embedded royalties when calculating project costs. "It seemed odd to dismiss the royalty costs of XP Embedded. This paints an unfair picture because [XP royalties] are way higher than those for Windows CE .NET," which he said are as much as $10 per device compared with about $100 for XP.
Jerry Krasner, vice president of market intelligence at EMF, admitted culpability for the omission. "The XP argument is correct. When I was looking at vertical applications, what really surprised me was that 60 percent of OEMs [using Linux] were paying runtime licenses to somebody. That indicates that Linux ain't free. So I tried to show what [Windows] CE offered for free and what you have to pay for with Linux. But in doing so, I didn't look at the XP model. And XP does offer an awful lot that Windows CE doesn't, although the runtime is much more expensive."
Corroborating report
Brix pointed to another recent study conducted by research firm Venture Development Corp., which also concluded that embedded projects involving Windows are completed in less time than those that use Linux, though the VDC study showed far more parity between the two.
Time to Market, Venture Dev. Corp.
About the author: Edward J. Correia is Senior News Editor of Software Development Times.
Copyright © 2003 BZ Media, LLC. All rights reserved. Reproduced by LinuxDevices.com with permission.
Do you have comments or questions on this story, or on the EMF report?
But please, before you jump into this discussion, read the full report! You can download it from the EMF website (requires free registration).
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.