At the end of November Microsoft posted a document on its Web site comparing the Windows XP Embedded and embedded Linux operating systems, with the emphasis on the technical and business inferiority of Linux. Embedded Linux distributors argue that the document not only distorts the value of Linux, but contains inaccuracies.

The following is a point-by-point response to Microsoft's attack from Lineo, one of the companies of which Microsoft is most critical. Excerpts from Microsoft's document are in bold. The responses were written for ZDNet UK by Tim Bird, Lineo's chief technical officer; Dave Beal, senior product manager for tools and operating systems; and Garnet Brown, senior product manager for smart handheld devices.

Be sure to also review the following related stories . . .

• Microsoft throws down XP gauntlet at the feet of Embedded Linux
  
• LynuxWorks responds to Microsoft attack on Embedded Linux
  
• Lineo rebuttal to Microsoft attack on Embedded Linux — Part II
  
• Microsoft takes potshots at (Embedded) Linux

---

Lineo responds to Microsoft attack on Embedded Linux

---

Microsoft: Embedded Linux offers a standard kernel but no standard device level application programming interface (API). There are multiple implementations of other major OS components so developers end up working with different programming environments and tools for each device, decreasing efficiency, limiting code re-use and increasing application development time. For example, there are at least five different window managers and at least four competing browsers, increasing programming complexity and reducing the pool of available developers.

Lineo: The fact that there are “multiple implementations of major OS components so developers end up working with different programming environments and tools” is heavily slanted to the “Microsoft provides everything you need” viewpoint. Let's look at history. Microsoft has tied the Web browser and windowing environment to the underlying operating system and defines these items as major OS components. Apparently they do not believe in product differentiation or choice.

Lineo, and (presumably) the rest of the Linux development world, has always thought that choice and the ability to differentiate your product from your competitor's product was a good thing. We recognize that the definition of major OS components is, except for the Linux kernel itself, arbitrary and dependent on the requirements of the product being developed. We do not attempt to dictate to our customers what must be included in the final configuration. For example, many embedded products do not require a GUI or a Web browser, therefore it is presumptuous for us to conclude that these are “major” components which (presumably) are required in every system. Instead, Lineo's Embedix Linux offers a “core” set of features necessary to provide a fully functional operating system (by Lineo's definition), allowing the developer to pick and choose elements that distinguish a product from the competition. Flexibility allows customers to innovate.

It is worthwhile to point out that the Opera Web browser, supported by the Sharp Zaurus SL-5000D handheld device, is much more feature-rich and compliant with Internet standards than the standard Web browser included with PocketPC. So, the developer actually has more of a choice when using Linux.

The Opera browser for the embedded space is essentially the same code base as their desktop version of their browser. Microsoft has two very distinct code bases for PIE and IE. The Opera browser can do Cascading Style sheets, Java scripting and Java applet support, but PIE does not. Opera browser also does SSL and is much faster than the Microsoft competition.
Essentially there is a big chasm between the code base and APIs on the Microsoft desktop and their CE code base in the embedded space and there is continuity from the Linux side. This holds true for the GUI system, the browser, the JVM and the Embedix kernel itself.

The window manager situation is similar. On Linux there are different window managers with different characteristics (footprint, widget set, etc.) that may or may not make them applicable to your specific application. For those who want a deeply embedded device with a small footprint, the developer can choose Century Software's Microwindows, or Swell Software's PEG (Portable Embedded GUI). For those who want full PDA-like features, the developer can choose TrollTech's QT-Embedded. Lineo does not presume to know our customer's requirements better than our customers. We offer choices so that they can best build the products they envision for the market they seek to address.

There is no common integrated development environment (IDE) for Linux. OS development is command-line driven and applications development requires a new set of tools for each device. Developers must either build their own tool-chain from piecing together Open Source tools or opt for a specific vendor's costly toolset.

Lineo: Microsoft chooses to use the fact that there is no monopoly within Linux against us, saying that the accessibility to a number of different options is a disadvantage. It is true that Linux elements are provided from a large number of sources, including alternate contributing vendors and the open source development community. Linux users have the possibility to use one of many Integrated Development Environments. The developer can select one based on feature set, cost, and familiarity. It is important to recognise that, regardless of your IDE choice, the underlying compile/build tool chains are the same. Thus, each developer can create interoperable code regardless of the specific IDE which they choose. They can choose the specific IDE they are most comfortable with.

With regard to the statement that developers must either “build…. or opt for a specific vendor's costly toolset,” Lineo's board support packages, which include a ported, tested and validated Linux kernel and operating system services combined with the complete development tool chain (compilers, linkers) is currently priced at $1,495. Costly doesn't quite seem to be the appropriate word when weighed against the fees charged by Microsoft.

There is also a choice here between choosing a huge feature packed multi-faceted development system from Microsoft that manipulates MS OS binaries, versus a cleaner and simpler model from Linux where the tools are to the point and the source is open. Microsoft can't engineer any tool large enough or with enough features to compensate developers for the lack of source code access.

Each embedded Linux distribution vendor certifies support for only a limited number of CPUs and drivers on its own distribution. Red Hat's v7.1 has support for only four USB and zero 1394 device drivers. From a list of over 300 network card drivers available for Linux, Red Hat's v7.1 Hardware Compatibility List (HCL) currently indicates support for only two “Red Hat Ready” drivers and only 27 “compatible” drivers. In addition, there are no drivers listed for any embedded CPU beyond the x86 processor and there is no embedded-specific HCL.

Lineo: Microsoft agrees that generic Linux provides support for hundreds of generic drivers as indicated by Microsoft above. Leading with that statement, it is a mystery to us why Microsoft would choose to qualify each vendor's offering based on what is supported by a single vendor's desktop distribution. We can't believe Microsoft doesn't understand that in the embedded world, support for discrete network cards is generally not required. Instead, networking hardware is generally provided directly on board (rather than via a separate network card). The open source nature of Linux allows anyone to modify existing drivers (often, quickly) to support new hardware as it becomes available. For example, each of Lineo's Board Support Packages, is specifically optimized for the target board's specific hardware, and includes network drivers as appropriate to their architectures.

Pre-configured solutions are available, but are either limited or distribution-specific and come at an extra cost from a commercial Linux vendor. Lineo's SDK supports only four pre-configurations — two for x86 and two for PowerPC. Lineo's Embedix BDKs have some pre-configurations, but they cost an additional $595 to $1,495 and are CPU and board specific.

Lineo: Microsoft's information is dated here. Presently, Lineo supports many boards across the primary embedded processors, including: x86, PPC, MIPS, ARM, SH3 and MPC5272. It is true that these pre-configurations (or Board Support Packages in Lineo terms) come at additional cost above our base tool-chain but our BSPs also provide a fully functional build tool-chain (compiler, linker, etc.), debug support (for all Linux memory/application spaces), deployment wizards (to easily move your code from the development platform to the target).

It is also important to note that Lineo's pre-configurations are not functionally organized as are Microsoft's (i.e. set-top box, Windows thin client, etc.) but they are organized around the target hardware to provide everything you need for full functionality and development capability.

Note also that they are comparing pre-configurations to Board Support Packages. In reality, a single board support package from Lineo often includes multiple pre-configurations (in the form of state definition files). So the number of pre-configurations supported on Embedix is much larger than the number of BSPs.

(Windows XP has a) comprehensive OS foundation with proven performance and reliability…

Lineo: Microsoft claims “comprehensive OS foundation with proven performance and reliability” for an operating system that was only recently released to the desktop and has certainly not yet shown “proven performance and reliability”. Those who have used Embedded NT (a minimal desktop operating system for embedded use–as is planned for XP Embedded) are likely have many stories of “proven performance and reliability”.

Linux was never designed to address the needs of the embedded market….

Lineo: Windows XP was originally designed for the desktop. Linux was not “originally” designed for the embedded market, but specific features and versions of Linux for the embedded market have been in development (and “design”) for many years. UClinux is probably the best example of this, whereby (by “design”) Linux was ported to processors without MMU. Besides kernel features specifically designed for the embedded market, there are also alternate version of libraries, utilities, and other programs. Linux has been much more customized for the embedded market than Windows XP. Microsoft should stop looking at desktop Linux in making these comparisons.

There is no documented third-party reliability testing evidence for Embedded Linux. Any claims made about reliability are typically specific to a particular distribution and largely anecdotal. Also, Embedded Linux offers nothing comparable to Windows XP Embedded's driver signing functionality and certification, and commercial distributions are not comparable in their certifications of drivers. For example, on Red Hat's current Hardware Compatibility List, there are less than a few hundred certified drivers and most of those are complete systems, as opposed to hardware components or peripherals.

Lineo: Linux, unlike a closed operating system, does not need to provide a list of certified drivers. Why? Because with Linux, the source code is open, allowing anyone (you, me, your development team) to fully test and validate the driver in a specific application space. No one has to take the word of a vendor regarding the stability or suitability of their software for a potentially mission critical application.

Windows XP Embedded delivers a feature-rich multimedia experience. Full support for DirectX 8 provides superior graphics rendering and performance. Direct3D — advanced support for interactive 3-D graphics applications. Windows Media 8 for industry-leading codecs and Digital Rights Management (DRM). Support for advanced graphical functionality including ClearType fonts and multiscreen. DVD video support.

Lineo: The APIs claimed here are, for the most part, proprietary and closed. Linux supports not only industry standardized multimedia formats for music and video, but also supports mainstream commercial and 3rd party multimedia plug-ins with an open API set.

To get the functionality, quality, performance, codec support and DRM delivered in Windows XP Embedded, OEMs will need to license an array of third party components including codecs, DRM and renderers (players) that generally come with incremental licensing fees… The MP3 basic decoder costs about $0.50 per-unit. If an OEM wants the enhanced functionality of MP3 Pro, it will pay an additional $2.50 per-unit royalty. For MP3 Pro encode and decode, costs can run an incremental $7.50 per-unit.

Lineo: And what is the royalty schedule for XP Embedded that will presumably include MP3 and a lot of other features that you don't really need? While there are many commercial solutions to Linux, there are also many freely available open source components also. The choice is yours. Linux provides the freedom to pay for only those components that you really need. Or maybe less. Is Microsoft asserting what the licensing cost will be for something they don't own?

Linux does not offer an Internet browser that is equivalent to the features in Internet Explorer 6.0. Third party alternatives such as Opera and Netscape are still not feature-rich and require an OEM to pay incremental per-unit royalties. If Java support is required, the OEM will need to license a Java Virtual Machine (JVM) and pay per-unit royalties.

Lineo: This is quite accurate. Why? Because Internet Explorer 6.0 includes a number of Microsoft extensions to the W3C specification. Microsoft is not compliant with the existing standards.

Embedded Linux provides options. If you want a feature or specific solution, you are able to select that solution. Embedded Linux does not force you to procure elements that are not applicable to your product.

IE 6.0 has a huge footprint and is slow compared with the Opera browser. To find a more appropriate browser for the embedded space Microsoft offers Pocket IE (PIE), which is a different code base than IE and has neither the features nor third party support of embedded Linux browsers or of IE. In order to do an apples-to-apples comparison, you would have to compare the subset of features they include from IE6 with browsers available for Embedded Linux.

In large quantities, an OEM will incur an additional $8 per unit to license Opera for Linux. As well, there is no email, instant messaging and news support in Opera for Linux.

Lineo: Opera's Web browser has matured significantly since this paper was originally authored. In addition, those who want stand-alone email, instant messaging and news support will find tens of competent, freely available applications to handle these tasks. Don't let Microsoft quote prices for Linux application software. Let the application vendor software do that. IE6 costs $12 per unit royalty, which is more than $8.

Networking support in embedded Linux simply is not as comprehensive and feature-rich as that in Windows. Linux does not offer comparable 802.11b zero-configuration. As well, Red Hat's HCL shows no supported drivers for 802.11b wireless — only non-supported Open Source drivers. Linux offers no RDP support for thin clients or the 802.1x wireless security protocol.

Lineo: The Microsoft author should review the state of the current Linux 2.4 kernel. Linux grew up on the Internet. Its support for networking services is comprehensive. Due to the open source nature of Linux, elements that may not yet be supported can be added by any competent engineer. Just because Red Hat doesn't support a driver doesn't mean it isn't there and usable. This has more to do with Red Hat test bandwidth than Linux maturity. They are picking nits here.

Multilanguage availability in Linux is extremely limited for both OS localization and application development. Localizing the OS means rebuilding it in the desired language.

Lineo: The OS should never be localized in the embedded space. You localize the user interface, not the OS. Even desktop Linux versions don't localize the OS–it's not necessary.

Although the author seems to be focusing on comparative items appropriate to the hand-held / PDA market, he fails to understand the true configurability of Linux and multiple options available to developers. This configurability provides the customer with extreme ability to innovate, but because not every component originates from the same enterprise, it does mean that a little research is required to determine exactly how you would like to best differentiate your product.

For example, TrollTech's “QT”, the desktop and application framework environment, which is used by the Sharp Zaurus SL-5000D hand held mobile device, supports:

• Languages with single-byte encodings (European Latin-1 and KOI8-R, etc.) and multi-byte encodings (East Asian EUC-JP, etc.) are supported.

• Support for the “complex” encodings — those requiring right-to-left input or complex character composition (eg. Arabic, Hebrew, and Thai script) is implemented, but the range of Indic scripts (Hindi, Devanagari, Bengali, etc.) is still under development.

To find and fix security flaws, Linux relies heavily on the Open Source community and source access, assuming that the “many eyes” of the Open Source community can equal a more secure OS. Even vendors such as Red Hat cannot guarantee the reliability and security of their OS. Red Hat's Security and Exchange Commission (SEC) filing states that they may not be able to test their software effectively because independent software programmers, over whom Red Hat has no control, contribute to Open Source programs.

Lineo: SEC filings always contain very alarming risk statements, even when they are unlikely, in the interest of full disclosure to the investment community. The reality of Open Source software and the “many eyes” theory is realized by comparing the track record of security reports for Microsoft OSes and Linux. Microsoft's Windows XP End User License Agreement is a telling report of Microsoft security record.

The open source nature of Linux allows the developer to track down any latent bugs or defects immediately after they become apparent. It further allows the developer to fix those bugs immediately. When compared with the alternative–that is, contacting the device manufacturer or Microsoft, entering a bug report, contracting with Microsoft for a bug patch (among the potentially hundreds of others submitted), and finally waiting for that patch to be delivered, the advantages of Linux are clear.

Linux provides support for APM, but that support depends on code that must be contained in the system's BIOS. In addition, Linux provides only limited ACPI support. Linux does not support plug and play and ACPI relies on such functionality. Adapting Linux to deliver this support will require major changes to the driver model — work that is time intensive.

Lineo: Lineo has provided customers with power management solutions specific to their embedded devices. This support has been implemented at both kernel and application software layers, and includes device sleep, wake-on networking, CPU idle modes and other technologies to make Linux usable on devices with stringent power consumption requirements. Comparing desktop APM support to that offered by devices such as the Sharp Zaurus SL-5000D is a mistake.

Linux has no strategy for integrating applications and services across devices, PCs, servers, and the Web.

Lineo: Microsoft references a number of supported technologies here, most of which are Microsoft proprietary. Linux supports a number of open source equivalents for many of these services.

Any aggregated device management needs to come from a third party at an additional cost.

Lineo: Once again, the developer must evaluate the needed additional cost components of Linux against the per-unit “one size fits all” cost of XP Embedded.

Linux looks to Java as a distributed applications development environment. Java provides only a set of least-common denominator services across different OSes. Java was originally designed to be interpreted, and only gained the ability to compile down to native code after a major retrofit. It fails to take advantage of the unique capabilities of the underlying OS.

Lineo: Translation: .Net is a proprietary solution targeted at only Windows platforms.

Java was designed for use with a single programming language — Java. Developers have no option to choose the best tool for the task and cannot use knowledge or code from other languages.

Lineo: There are multiple Java Development Environments now available for, and compatible with, Linux. Java is a cross-platform, open standard (unlike .Net) technology that has been approved and agreed upon by key members of corporations and organizations.

Java can be used for distributed applications development, but so can numerous other languages (including many of the same languages that Microsoft promotes in its .Net initiative–like perl and python). For compiled code, the APIs available for embedded Linux versions are identical across most CPU architectures, and only differ slightly even between major CPU variations (like between MMU-less and MMU-full processors). The various versions of Linux available for the embedded space have much more in common with each other than do all the different OSes released by Microsoft for the embedded space.

Linux solutions from any vendor typically are fraught with numerous incremental intellectual property, support and potential per-unit license or royalty costs that add up to significant licensing and support expense.

Lineo: As opposed to a single, large, one size fits all per-unit cost. Linux provides you with the ability to use commercially available code or freely available Linux code.

Lineo charges for some IP, including the boot-loader, code optimizer, reflective memory, debugger and pre-configured device-specific distributions. Royalty fees can run from $1 to $15 per-unit for this incremental IP.

Lineo: Lineo does charge for some IP, but royalties are not charged for any of the items that Microsoft has listed. Microsoft is distorting the price that Lineo charges and the terms of the licensing.

Lineo charges for IP including: VxWorks & pSOS+ API compatibility layer, vertical solutions for smart handheld devices, residential gateways, digital TV. Once again, the customer only selects those IP elements they desire. The core Linux elements — including the features necessary to create a large number of embedded devices — remain royalty free.

Linux tools pricing is expensive and a key source of revenue for embedded Linux vendors. Lineo sells its SDK for $5,495 per seat. Anything other than installation support costs extra, and for support you have to call Lineo, Caldera and/or Metrowerks depending on your particular issue. There is no single source of support.

Lineo: Lineo's tool chain includes cross-compilers, linkers, debuggers, memory monitors, kernel configuration tools, OS services configuration tools, library reduction/optimization tools, deployment wizards and more. To address the on-going FUD about Linux, Lineo has recently adjusted its pricing model where the Embedix Application Builder is available for $2,995 (including all tools described above). This is available for an operating system that comes without royalties or other per-unit costs.

The support needs of Linux and of a closed operating are completely different. The open source nature of Linux means that any competent engineer can review the OS code and derive a solution themselves, often within hours of detecting the problem. Closed solutions such as XP Embedded effectively tie the hands of engineers, forcing them to consult Microsoft support, for problems that are often minor. But in-fact, Lineo does indeed provide a single point of support for Lineo's Embedix tool-chain.

An OEM is required to sort through a potential legal morass of licensing issues around the GPL if it wants to protect its intellectual property rights. This creates extra costs from both a development and legal perspective. An example of this risk can be taken from Nvidia. An Nvidia programmer, in the course of developing a driver for one of its products, used a portion of code from a freely available video driver. The developer failed to realize the code was licensed under the GPL and would therefore require Nvidia to release the source code for its entire driver. Because Nvidia did not want to release the source code to its commercial software, the company incurred substantial costs to develop a new driver that did not contain the GPL code.

Lineo: The open source licenses are often spoken of as a disadvantage but they are truly a strong advantage. What they offer you is a choice: Do you want to use existing, freely available code (and contribute the resultant product back to the community) to quickly produce your driver or application or do you want to program from the ground up–increasing the development cycle, but avoiding the derivative work clauses of the open source licenses? This choice is not offered by a closed operating system such as XP Embedded.

Lineo provides its GPL compliance toolkit that reviews your complete, deployable software configuration and architecture. The GPL Compliance tool flags potential open source licensing violations relative to applications, services, libraries and specific library linking techniques. This tool allows any line-level engineer to quickly (within minutes) evaluate their system in order to spot licensing violations at an early stage of the development process. They can then decide whether to proceed along the same path, releasing the resultant code under the open source license, or re-build the application to avoid the licensing issues completely.

One of the main purposes of the tool is to combat the FUD that Microsoft and others spread about open source licenses. Doing any software engineering work requires responsible engineering practices and some oversight to avoid copying or using unfairly the work of others, or infringing on another party's intellectual property. Pretending that this is a situation unique to open source development is one of Microsoft's worst lies. Microsoft does nothing to assist customers with IP management issues, even though their customers face some of the same legal and patent infringement issues that exist for open source embedded developers.

Product support and maintenance guarantees are only as good as the vendor standing behind them. Lineo offers free 30 day installation support. Anything else costs extra and may require a call to Caldera or Metrowerks depending on the issue. Support is a profit center for embedded Linux vendors. Red Hat charges $1,999 per-seat for Linux hosts and a minimum of $2,500 per-seat for non-Linux hosts GNUPro toolset support. In addition, an OEM would need to buy OS support–Red Hat's 5 incident basic support costs $949 per year…

Lineo: Linux does not provide sustaining revenues in the form of per-unit charges, as does XP Embedded. Consequently, we can't amortize support costs within the royalty structure. This requires Lineo to charge for support issues.

While the Open Source community can help with development; the burden of testing and supporting a particular OS implementation long term falls to the OEM. While commercial Linux vendors do offer support, if an OEM makes modifications or adds additional features to a commercial Linux vendor's distribution, support may not be available or may cost more.

Lineo: Under Windows XP, the developer does not have access to the source code and is not even permitted to make modifications.

With over 400 developers, four products teams and a worldwide sales, marketing and channel development team, Microsoft is focused on our customers and partners for Windows Embedded… Recently published data from “Open Source: The Unauthorized White Papers” by Donald Rosenberg outlines that fewer than 200 people have been accepted as contributors to the Linux kernel, and of the 2,200 people who have contributed to the rest of Linux, 91.4 percent provided only one or two pieces of code. This list gets even shorter in the embedded space.
Lineo: Microsoft compares the sheer number of developers between the complete Windows Embedded (another interesting note–they diverge from the use of “XP Embedded” term and refer to four product teams) and Linux, especially as Microsoft has all but defined XP Embedded to include the GUI and the Internet browser (see above).

Microsoft has historically blurred the distinction between what is an application and what is a component of the operating system. If Microsoft were to apply the same broad definition of “what is Linux?” that they used to define XP Embedded (i.e. to include GUI, Web browser, Java, etc.), we believe that the numbers game would look very different. But what's the point? Their own statistics show that Linux has thousands of contributors, while the whole of the Windows embedded product teams consist of 400 developers.

Linux Is Not Free. While the Linux kernel may be available at no charge under Open Source, the total cost profile of an embedded device is a function of multiple factors, including: Opportunity cost of investing in OS development versus differentiating via form factor design and/or value-add applications and services. Etc.

Lineo: Microsoft uses the term “…total cost of an embedded device”. Clearly these issues apply regardless of which operating system one chooses — Linux or XP Embedded. What they neglect to include is the per-unit cost, and the tools cost.

So called “royalty-free” OS software licenses may apply to only a limited set of functionality and may be offset by higher upfront costs and incremental royalties for proprietary, value-added software.

Lineo: Lineo provides fully open source Linux with no per-unit costs. The developer is free to modify and use thousands of existing applications for their specific device. License bearing components are negotiated in a manner which best suits the customer and Lineo. XP Embedded uses a one size fits all technique with a commensurate royalty structure.

Copyright © 2001, CNET Networks Inc. All rights reserved. Reproduced with permission.

---

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.