Foreword: In this guest editorial, FSMLabs CEO Victor Yodaiken relates his company's experiences in the real-time Linux market to Wind River's efforts to reposition embedded software development as “device software optimization,” or DSO.

As background, be sure to read our DSO special report, which offers a comprehensive timeline that chronicles the arrival, evolution, resistance to, and growing adoption of DSO as an industry-defining term.

---

Software reuse and DSO — breaking the rules of embedded software development

by Victor Yodaiken

One of our customers is going to production with a very sophisticated software defined radio that provides a textbook case of what we call “software reuse,” and what Wind River calls “device software optimization” (aka “DSO”).

The customer is Harris Systems. They began the project with a good algorithm and then prototyped, tested, and validated using x86 laptop computers and PCMCIA wireless cards. The same software was picked up and moved to rugged quad and dual PowerPC VME single-board computers from Curtiss-Wright with specially designed radios. Later versions may take advantage of multi-core chips. RTCore gave Harris developers low microsecond decoupled hard real-time, and Linux gave them everything from networking and advanced server file systems to open source routing code (provided by NRL) and network test code.

FSMLabs software has now been incorporated in products ranging from cell phone handsets to welding robots, and from computer graphics systems to aerospace simulators, and Harris developers exemplify what we've seen in successful development projects — mostly in accord with what Wind River is calling DSO.

Harris finished their incredibly complex project ahead of schedule and within a very constricted budget by breaking all the standard rules of embedded software development:

• They didn't wait for specialized hardware to show up before starting development to ensure they started late. They developed software and validated the system with off-the-shelf notebook computers and PC cards [story]
• They used microprocessors, including multi-processors and smart software to avoid requirements for specialized hardware instead of fighting Moore's law. Quad PowerPCs are hard to beat with custom hardware.
• They didn't spend a single second of engineering time writing boot loaders or BSPs — they purchased software that booted up just on its own. “Boot loaders and BSPs are a solved problem”!
• They recognized early that they had a hard real-time requirement and addressed it in design instead of hoping it would go away or that they could configure their way out of it.
• They used the wealth of powerful services, tools, and applications that already exist for Linux — both free software and proprietary. In fact, they chose between buy, download, and develop internally on the basis of a comprehensive understanding of cost and time tradeoffs instead of relying on “how things were always done” or some marketing spin. In other words, they focused on the value add.

At last year's ESC, Wind River's CEO, Ken Klein, provided an excellent analysis of the problems encountered by developers who do not break the rules of embedded development. Klein said:

• More than half of embedded designs are completed 3-4 months behind schedule
• A third of them don't meet basic performance requirements
• 24 percent — nearly a quarter — get cancelled
• Of the ones that actually make it to market, two thirds are over budget, often way over budget

Embedded software has become complex in the last two decades, and methods that worked in days when hardware was 90 percent of the complexity and value, when development times were leisurely, and where systems were neither networked nor complicated are now unbearably costly and ineffective.

As Klein and others have pointed out, much of the problem is that embedded development teams are spending time and effort on “reinventing the wheel.” The solution is software reuse — or “DSO,” if you like more complicated terms.

Wind River and its collaborators in the DSO project are attempting to assemble a supply of reusable components that run on their operating systems, but Linux already provides a superb medium for reusing the wealth of software developed for the enterprise. The fundamental design principle of RTLinux was to provide hard real-time as a “bolt-on” component to a standard operating systems platform. For the last 12 years, we have been telling new customers their design goal should be to minimize the real-time code so that they can improve its reliability, and to make their application re-use as much as possible of the services and applications in Linux (or BSD, or even Windows) domain.

RTLinux/RTCoreBSD architecture
The quintessential example of an RTLinux data acquisition program is a simple real-time thread that loops periodically and writes data down an RT-fifo to Linux, where a single line of shell script can determine whether data goes to a flash file system, a journaling file system, over the network in a possibly encrypted TCP/IP stream, or to an application.

FSMLabs cellphone customers have reused Nucleus GSM/GPRS stacks and QT graphical environments in the same system. Our simulation customers are using many of the tools in Fedora Core 5 with X-windows high end graphics and Linux's support for 16 or more processors in a multi-core environment. Our telecom customers are reusing the OSDL's carrier grade Linux with contributions from many sources, instead of having their engineers redevelop all of this open standard technology from scratch. Many of our customers are attaching their embedded applications to Windows software (using networks or VMware), to provide operator interfaces or other sophisticated services that are increasingly important in embedded applications. We have robotics and control customers who reuse our parallel port driver to prototype or even replace digital I/O pins.

Software reuse is not just the process of incorporating existing software components in a new project; it also covers things like developing on a PC or notebook while hardware is being created. A commitment to software reuse forces engineers and managers to decide whether some component or service that can be locally developed will have compelling advantages over a reused component or service. If a compelling advantage cannot be identified, then the obvious question becomes — “why are we wasting our time with it?” (or, “why are we wasting our money paying people to work on it?”).

Allocation of code in real-time system
Some organizations have replaced “develop in house” with “download,” and this works in many cases. The amount of functional, high quality, free software on Linux and BSD is astonishing.

However, it is easy for organizations to get dragged away from their actual project, and to sink into a swamp of maintaining areas outside their area of expertise. Linux or Apache or even Perl can be a nightmare to maintain and test if there is not a match between internal expertise, time available, and level of criticality. Catastrophes caused by development groups attempting to learn the ins and outs of a million line open source project in their spare time are the basis of all those Microsoft-sponsored studies showing high total cost of ownership for Linux.

The question is, where is the balance in the download versus purchase (or purchase support) tradeoff? The DSO member companies and Microsoft would probably advocate a more conservative choice than FSMLabs, while we would be more conservative than, for example, those dwindling number of development groups that can stretch development times and not consider salary costs.

All of us are dealing with a fundamental change in the “value add” calculation for embedded software. Wind River's John Bruggeman notes: “We think there's a fundamental shift in the marketplace, a shift driven by end users, or customers of devices,” and then goes on to mention technical requirements.

Red Hat's Michael Tiemann discussed this shift more deeply in an article published by LinuxDevices.com four years ago. Tiemann wrote:

The modern (soon to be “archaic”) embedded development model is to select a microprocessor, design a hardware platform, and license an RTOS (not necessarily in that order) — then assemble or license additional software components, develop a proprietary “application”, and once debugged, sell the integrated widgetry to a buyer who supposedly wants the widget independent of any software or hardware componentry contained within the widget. Ever since this model was pioneered by James Ready and his VRTX RTOS, virtually the entire industry has inadvertently evolved to a point where licensors are treating 100 percent of the resulting development process as their proprietary value, rather than recognizing that 80 percent of what they do is actually quite common across the industry (or at least their slice of it). The result is that companies are paying the full proprietary cost of their development process while only 20 percent of their development process is truly value-add. The industry is therefore paying five times what it should for product development, and this 400 percent overage is causing an industry challenged by current economic conditions to slip from marginal profitability to substantial unprofitability. In this analysis, the fact that Linux can be licensed free of charge (thereby eliminating the entire cost of the proprietary RTOS) changes the equation almost not at all, yet companies are behaving as though it changes everything.

Any development project should start with an analysis of why the project is worth doing — what exactly is it that the developers will add to the project that makes it a good idea to pay them to work on it. This analysis can then be used to help focus engineering effort on the “value add.”

If your device is a simulator or a VOIP gateway, then boot loaders, file storage, and window managers may be important components of your system. But they are commodity components — they don't provide a justification for the effort of creating your product. What Tiemann noted, was that most of the software inside modern embedded systems is commodity software — solving problems that have already been solved. Linux and BSD reflect this reality, as does RTLinux, which was designed to make real-time into a component of a platform OS instead of a capability of a niche RTOS (real-time operating system).

In explaining DSO, Bruggeman says that “rich content” and “communications” are driving new customer requirements, but we see customer requirements also extending to powerful databases, interoperability, storage systems, multi-processor and multi-core management, standard middleware, and so on. Embedded developers now want the platform that enterprise developers have, plus they have specifically embedded requirements for space and power limits, ruggedness, and — more often than some suspect — hard real-time. Linux and BSD bring all that enterprise computing platform power to embedded developers automatically. DSO is Wind River's attempt to catch up with the new era.

---

About the author: Victor Yodaiken, CEO and Co-Founder of FSMLabs, came up with the basic technology of RTLinux, a technology that adds hard-real-time performance to Linux. Yodaiken began his career in 1983 as one of the chief developers of Auragen's distributed fault-tolerant UNIX, and he had an active consulting business before starting FSMLabs. He has also worked in academia, as a professor and department chair at New Mexico Tech, and as a research professor and post-doctoral fellow at the University of Massachusetts in Amherst. Currently he is an adjunct faculty member at the University of New Mexico.

---

Do you have comments on this article?

---

talkback here

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.