From the streets of the Silicon Valley, to Wall Street, and of course to Sand Hill Road, “Linux” was the catchword of 1999. All signs point to Linux continuing as both a hot financial and technological prospect into the new millennium. Putting IPO fever aside, there are strong technical and business reasons to consider using Linux for your next embedded development project. There are measurable engineering advantages, like broad hardware support, scalability, excellent performance, high reliability, and open APIs.

There are enticing business reasons to employ Linux — no run-time royalties, free or low-cost available software components, and freely available source code. And, there are somewhat intangible reasons to put Linux into your next internet appliance, control system, or communications switch — engineers like Linux and want to put Linux experience on their resumes, easing today's hiring crunch, and so do marketers, who are eager to package products with “Linux Inside”.

This white paper reviews these and other reasons for building your next embedded application with Linux, and provides a glimpse of what to expect from embedded Linux in the coming year.

Hardware Support — the Elusive Device Driver

Finding device drivers for embedded designs based on a traditional embedded OS, running on COTS hardware, can be a frustrating exercise. In the desktop and server world, new devices emerge every three months or less. Peripheral vendors augment the OS offering with drivers and utilities that accompany their video cards, network interfaces, and other devices. Smaller, more resource-constrained embedded OS vendors struggle to keep a few dozen device drives and BSPs current — the authors once spent three days searching for a single-board computer whose complement of peripherals was fully supported by the OS in use. Imagine an embedded OS trying to pass the “Fry's Test” — walking down the aisle the local electronics superstore to find which devices, if any, a traditional embedded OS supports.

Traditional RTOS vendors employ two strategies to support the hardware chosen by their customers: either charge up to $30,000 in NRE/consulting fees (and then retain the driver code themselves), or “borrow” (you guessed it) Linux drivers from the open source community. Then, they still must invest substantial time and their customers' money to port the interfaces to their proprietary, closed systems.

Contrast the availability of drivers for Linux: with very few exceptions, device interface code for Linux appears quite regularly along with or even prior to the release of new devices. For example, two years ago a client in Australia needed an Ethernet chip driver for the proprietary OS they were using. The chip vendor, although already shipping silicon, still kept the specification under wraps. Rather than haggle with the vendor, together with a consultant we easily found a Linux driver for the device, and a port ensued. The client was so impressed that this year they switched to embedded Linux for their application OS.

Scaling Linux

Traditional embedded operating systems go to great lengths to tout the size and efficiency of their kernels. Even today, when memory is only available in increments of 4 MB, vendors still extol their “complete micro-kernel solutions” in 15 KB. Realistically, viable commercial OS configurations come in at 128-256 KB for a reasonably configured kernel, another 100-200 KB for a TCP/IP stack and sockets library, and for a web appliance, 50-150 KB for a HTTP server, plus a minimum 64 KB of working RAM. An embedded system software profile of 800 KB to 1 MB no longer looks gargantuan!

These same vendors point out that a desktop distribution of Linux runs into the hundreds of megabytes. Well, they are right. However, embedding an OS like Linux is more like lunching at a good cafeteria than dining at an upscale prix fixe establishment. When you embed Linux, you choose only those components that make sense for your application. Don't need a read/write file system? Don't use it! The same logic applies to networking, GUI, shells, and countless other utilities and libraries. If your project does need more functionality than fits into local non-volatile storage, you can craft either a tiny Linux boot loader, a stand-alone bootable Linux system, or a slim Linux kernel that pulls down additional modules and application code over a network, frequently in 500 KB or less!

Linux and Real-Time

Many (if not all) embedded applications have some sort of real-time performance requirement. However, the latency-sensitive portions of such embedded code are usually dwarfed by purely throughput-sensitive code that implements control algorithms, communications protocols, and user interfaces. Moreover, many of these real-time requirements prove to be “soft” — missing a deadline once in a while does not impact the overall system viability. Even when “hard” real-time deadlines do exist, the scope of deterministic response can be reduced to the driver level or overcome by the “real-fast” performance offered by combining Linux with Pentium and PowerPC silicon.

So, when developers choose to embed Linux to leverage the wealth of available networking, database, and interface software, real-time concerns often take a back seat. But should they? Using a general-purpose operating system (GPOS), like WindowsNT, for real-time and embedded applications, can spell disaster — recall the U.S. Navy's having to tow their Aegis destroyers back to port, repeatedly, because of crashed WindowsNT steering systems. A GPOS typically suffers from several challenges to real-time applicability: determinism in general, and response under load in specific. GPOS schedulers, optimized for time-sharing, can induce unpredictably long blocking times; drivers developed by a mix of GPOS-vendor engineers, peripheral-board vendors, and other third parties add their own variable latencies. The whole system may respond in a sprightly fashion with a few dozen threads and a handful of interrupting devices, but scale either into the signficantly and watch responsiveness drop and jitter hit the ceiling.

Linux, developed for desktops and servers, is also a GPOS, but enjoys a promising future in real-time embedded designs. Two primary paths exist to providing a real-time Linux: by inserting a second kernel into the system, and by refining the standard Linux scheduler and tuning Linux device drivers. The first approach, which has also been applied to WindowsNT, presumes that to use Linux for real-time, you must first “throw it out”.

The addition of a second OS, regardless of its putative real-time characteristics, vastly complicates both the development and run-time considerations of the embedded Linux developer. The second OS introduces

• A second set of RTOS-specific APIs
  
• The loss of Linux process and threads programming model
  
• A “flat” memory model without benefit of memory protection
  
• Starvation of applications running in the “normal” Linux environment
  
• Unnecessary stratification of Linux applications and drivers into ill-defined non-real-time and real-time classes

A much more sensible approach is to optimize the existing, open Linux code base to address the needs of actual applications, but first to characterize the performance of standard versions of Linux.

Before even attempting to enhance Linux responsiveness, it is key to measure its real-time performance, thoroughly, in terms familiar to real-time/embedded designers: worst-case interrupt latency, context switch, and maximum blocking times. Simply knowing how well Linux will respond on a particular platform in many cases obviates the need for additional re-architecting. Linux already enjoys provably superior compute and networking performance throughput, even when compared to supposedly lightweight RTOS products, implying good to excellent average response times, even under load. Expect such measurements to be published in the coming months, by MontaVista Software and others.

Developers whose requirements exceed such real-time characterizations in terms of the Linux kernel itself, need not despair. On-going work by engineers at MontaVista software and elsewhere in the Open Source community continues to refine overall performance and promises to offer greater determinism in both the kernel and interfaces.

Legacy Code

Application code comes to an embedded Linux environment from one of three sources: existing code written for a commercial RTOS, code written for an in-house proprietary kernel or executive, and code developed for UNIX-compatible workstations and servers.

Applications running on both commercial and in-house kernels share a key characteristic — proprietary APIs. While several off-the-shelf kernel products enjoy extensive design-in success, few if any offered programmers viable open applications programming interfaces, like POSIX or the cross-industry openness of the various Linux APIs. Moreover, embedded OS vendors had little interest in supporting open APIs or any other tools that might facilitate customers' porting applications from their kernels to those of other vendors or to open platforms. In-house kernels, developed for very specific target applications, either had no perceived need for openness and portability, or arose with and were perpetuated by “not invented here” (NIH) sensibilities.

Today, embedded and real-time systems designers, excited by the benefits of Linux and the Linux explosion in general, are eager to build their next-generation products on an embedded Linux. However excited, they still must face the challenge of porting existing code to Linux.

Porting UNIX applications to Linux often is as simple as typing “make” or using a GNU configuration package. The availability of POSIX (IEEE1003), System V, Berkeley, X11, and other standard APIs on Linux makes porting incredibly easier, if not always 100% transparent. Challenges arise from dependencies on semi-proprietary extensions (vendor and platform specific) or from use of superceded open APIs (like POSIX.4 or POSIX draft APIs). Such challenges are very much akin to those faced when porting proprietary code, as follows.

While many embedded developers use the occasion of moving to Linux as an opportunity to rearchitect their code, there still can arise the need to move existing programs with proprietary system and library calls, ad hoc/in-line device access, explicit interrupt/preemption disabling, etc. to Linux. In our experience, accommodating the two dozen most frequently-used calls, as well as programming practices, breaks out as follows, with accompanying solutions:

• Task creation, scheduling, management — use POSIX/Linux threads
  
• IPCs (queues, mutexes, etc.) — use POSIX .1 and POSIX .1b equivalents
  
• Networking — use equivalent Berkely sockets
  
• Pre-emption disable — migrate to driver / rearchitect
  
• In-line interrupt management — migrate to driver / rearchitect
  
• Direct memory access — either migrate to driver or use POSIX .1b mmap()

For developers not wishing to recode to track the above, inter-OS translation libraries do exist for at least large APIs subsets, and we at MontaVista will be providing like solutions.

The Embedded Linux Business Model

Embedded systems designers have always reluctantly faced the prospect of purchasing development tools. First comes the sticker shock from per-seat proprietary tool sets priced from $10,000 to $20,000. Next comes the haggling over run-time pricing. While some typically low-end embedded OS vendors offer run-time buy-out options, most impose a heavy deployment royalty, not just on the kernel but on every useful component. These run-time costs significantly raise real development costs, or impact the cost-of-goods-sold of your product, thinning sometimes precarious margins and threatening your time-to-volume.

Proprietary embedded OS vendors attempt to front-load their commercial relationships (even those charging royalties), and thereby have little or no interest or patience in seeing a project through to completion. “Drive-by selling” is common — repeat business comes from technology lock-in, not excellence in service. Therefore, most commercial embedded OS vendors enjoy only a 10-15% support renewal rate.

Contrast the embedded Linux business model. Instead of doling out intellectual property, embedded Linux suppliers offer subscriptions, professional services, and in MontaVista's case, a standardized embedded Linux distribution (Hard Hat Linux). Charging a reasonable fee for embedded Linux subscriptions, based upon a stable, validated distribution, represents a very viable long-term business strategy.

This new business model leverages the breadth of Linux technology and open source components to accelerate customer time-to-market, optimizes the cost of ownership of development resources, and places the focus on the customer-supplier relationship. Embedded Linux suppliers, like MontaVista, must (and do) prove their worth by combining Linux and open source credentials with embedded and real-time expertise, in a professional support organization. The benefit to you, the embedded developer, is a supplier who lives or dies upon your project's success.

The Future of Embedded Linux

To foster the mainstreaming of Linux in embedded applications, several enabling technologies will emerge in the next 12 months:

Embedded Processor Support: While at present most Linux distributions focus on Intel Pentium, the year 2000 will bring support for a variety of embedded 32-bit x86 CPUs and high integration PowerPCs from Motorola (8xx and 82xx) and IBM (4xx), a well as ports for MIPS chips popular in networking, and emerging CPUs like StrongARM.

Open Source Flash File Systems: Distribution-limited enabling flash technology available today will either move to GPL practices or be replaced by new software and hardware solutions.

CompactPCI Hot Swap Support: Ports of Linux today exist for CPCI systems from vendors like FORCE, Motorola, and Ziatech, with either no PCIMG hot swap or hardware-specific support. In 2000 common models for Linux hot swap aware drivers (HSADs) will support these and other vendors.

CPCI Backplane Networking: Today several backplane networking/messaging solutions exist for Linux on CPCI (e.g., MontaVista's Hard Hat Net). PCIMG is moving quickly to standardize schemes for passing raw and IP data packets over the CPCI chassis bus and Linux is likely to be the first OS to support this capability.

High Availability Frameworks: HA-specialists (like goahead), protocol stack vendors, hardware vendors, and Linux suppliers will variously partner to supply HA framework software, and to bind existing frameworks to specific hardware configurations, in increasingly non-chassis-specific ways. Expect to see HA framework components for Linux, like journaling file systems, clustering software, and policy databases.

Enhanced Real-Time Support: Linux will become more real-time/deterministic in performance, both through kernel substitution strategies and through enhancements to the standard kernel(s). The continuing trend of faster silicon is likely to make some real-time concerns moot.

More Options in Tools: Traditional cross development tool vendors will make available both workstation Linux hosting and embedded Linux targeting tool sets for C, C++, Java, IDEs, and high-level design tools that complement or compete with GNU.

Embedded Linux Tools: Expect to see tools targeted specifically at embedded Linux, including GUI-based scaling/configuration tools, small GUIs (not X-based), tiny web servers.

Open vs. Closed Source: The next year will bring both new open source technology options and availability of licensed intellectual property in a Linux context.

These developments and others, technical and markets-based, will bring embedded Linux into traditional domains of the proprietary RTOS. Linux, with its wealth of networking software, will very naturally accompany the Internet revolution as host to applications like Internet-enabled multi-media, SOHO office appliances, and home network gateways, as well as find its way into both shelf and card controllers for large communications switches, routers, and other infrastructure equipment. Industrial control, robotics, instrumentation — all traditional domains of the proprietary RTOS, have projects that are either implementing today or planning shortly to implement Linux-based systems.

Linux Caché

In the past, choice of embedded software components was a closely-held trade secret; implementation specifics, including the embedded OS, could confer significant competitive advantage. Marketers at embedded OS vendors despaired of ever being able to crow about design wins — either the project languished, or the application designer remained tight-lipped when it came to PR.

Compare new designs based upon Linux — when before have high-tech product marketers touted their companies' choice of operating system? “Linux Inside” is now appearing on the outside of products. When have manufacturers been able to use their embedded OS choice as a hiring incentive? Linux expertise is not just in high demand, it is a major employment draw across the country. And when have inherent economies freed the embedded developer from being a hostage to imperious tools and OS vendors? Today. That's when, with embedded Linux.

References

Raymond, Eric S. [1999]. “The Magic Cauldron”, from The Cathedral & the Bazaar. O'Reilly & Associates, Sebastopol, CA.

Ready, James. [1999]. “Running Linux on a Dual Processor/SBC CompactPCI System”. RTC — Software Updates, September.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.