During the course of the past week, there's been an increasingly lively discussion (and debate) about the real-time characteristics of Linux, and over which approach is best. As a result, questions like these are starting to become more and more common . . .

• What does “real-time” mean?
  
• Who (or what) needs real-time, anyhow?
  
• How much real-time is really needed?
  
• What are the… alternative ways to add real-time to Linux?

In this article, we'll try to clear away the smoke, and provide some perspective on the subject of real-time Linux.

Here's the plan. First, we'll define terms like real-time, hard real-time, soft real-time, and learn how real-time performance is characterized. Next, we'll learn what sorts of systems need real-time, and how much they need. Then, we'll compare some of the basic approaches used to add real-time to Linux. Stick around, 'cause at the end of the article, we intend to introduce you to a new, evolving “quick reference guide” to real-time Linux products and resources!

A little help from my (real-time) friends

Not being an expert in operating systems, I thought it best to lean on some embedded Linux (and other) heavyweights for guidance. Let's meet our “panel of experts” . . .

• David Beal — Product Marketing Manager at Lineo, a company that offers a Linux distribution (Embedix) for embedded and real-time applications.
  
• Mitch Bunnell — CTO and cofounder of LynuxWorks, a company that offers both a Linux distribution (BlueCat Linux) for embedded and real-time applications, and a POSIX-compliant proprietary (non open source) real-time operating system.
  
• Steve Furr — Senior Software Developer at QNX Software Systems, a company that offers a POSIX-compliant proprietary (non open source) real-time operating system.
  
• Douglass Locke — VP of Technology at TimeSys, a company that offers a Linux distribution (Linux/RT)for embedded and real-time applications.
  
• Jim Ready — CEO and founder, MontaVista Software, a company that offers a Linux distribution (Hard Hat Linux) for embedded and real-time applications.
  
• Victor Yodaiken — CEO and cofounder of FSMLabs; chief maintainer, RTLinux project. FSMLabs offers a real-time Linux distribution which based on RTLinux plus the Linux kernel.
  
• Myron Zimmerman — CTO and founder of VenturCom, a company that specializes in Windows NT and CE for embedded and real-time applications. Approximately 15 years ago, VenturCom developed the first real-time implementation of UNIX, called VENIX.

The questions, please

Here's the list of questions I posed to each of our real-time panelists . . .

• What is real-time?
  
• What is hard real-time?
  
• What is soft real-time?
  
• Who needs real-time? hard real-time? soft real-time?
  
• How do you characterize and specify real-time performance? (In other words, what specifications should be used to objectively compare real-time OSes?)

The replies from the panelists were both thoughtful and instructive. If you have the time and interest, I strongly suggest reading each panelist's answers. Use the following links . . .

• David Beal, Product Marketing Manager, Lineo — response
  
• Mitch Bunnell, CTO/cofounder, LynuxWorks response
  
• Steve Furr, Senior Software Developer, QNX Software Systems — response
  
• Douglass Locke, VP Technology, TimeSys — response
  
• Jim Ready, CEO and founder, MontaVista — response
  
• Victor Yodaiken, CEO and cofounder, FSMLabs — response
  
• Myron Zimmerman, CTO and founder, VenturCom — response

So . . . What is real-time, and who needs it?

After carefully studying the answers from our panel of real-time experts, I've put together composite answers to the real-time questions I posed. One thing I certainly learned, is that experts hardly ever agree on anything! (Read their responses, to see my point.) Nevertheless, there do appear to be some trends among the answers — so here's my attempt to balance the perspectives of seven “experts” from seven different companies (who pretty much all compete with each other, I might add) . . .

• What is real-time? — I think the IEEE definition (supplied by Douglass Locke) is a good one, and a clear one. Namely: “a real-time system is a system whose correctness includes its response time as well as its functional correctness.” The system doesn't just run software to plough through the process; it needs to do it in a timely manner. So, real-time systems care about timeliness, but just how timely isn't clear until you add a modifier to “real-time”. In other words, real-time is very much a gray region. “Response to what?” To “events”, which are either external events, communicated to the system via its input/output interfaces (which include things like Ethernet as well as audio, keyboard, or mouse input), or to timer-triggered events within the system itself.

• What is hard real-time? — the almost universal definition of “hard real-time” seems to be guaranteed worst-case response times. Here too, there's an aspect of grayness, in that the time-scale isn't built into the definition of hard real-time. Instead, a system is considered hard real-time if it meets required maximum worst-case times. In short, an operating system isn't hard real-time per se — instead, it may meet the hard real-time requirements of some applications, but not of others. Whether the OS must guarantee maximum response times measured in a few microseconds, a few milliseconds, or even a few seconds, depends entirely on the application.

• What is soft real-time? — soft real-time is probably best defined as real-time that isn't hard real-time. In other words, there are two kinds of real-time: hard and soft. Sort of like eggs. Put another way, soft real-time systems require timeliness, but not critically so. A soft real-time system is one in which its users are happier when the system responds optimally, but which is not considered to have “failed” when the system doesn't meet each and every desired response time. On average, a soft real-time system meets a particular response time goal.

• Who needs real-time? hard real-time? soft real-time? — Most computer systems have at least some real-time requirements. That is to say, there is an element of timeliness associated with their operation. These days, however, there are increasingly stringent requirements — largely due to the interconnectedness that has become the norm, rather than the exception.

  As an example, consider Voice-over-IP (VoIP) telephony, between two PCs. You can carry on a conversation even if some voice packets are being dropped. However, when too many packets don't make it through on time, you lose the ability to communicate — which really means the system has begun to fail. So even though this example seems to be one of soft real-time, since it's not catastrophic if a packet is dropped here or there, there's still a point where the application falls apart. Therefore, a hard real-time constraint exists — like a “floor”, beneath which the system fails — even in this otherwise soft real-time system.

  When we think of hard real-time systems, we tend to think of systems either with extremely tight or highly critical timing constraints, where missing deadlines will break things or cause loss of life. Examples like the Mars Lander, industrial automation, and autonomous robots come to mind. But actually, any system that is considered to fail when its particular worst-case timing requirements aren't met is a hard real-time system. Don't confuse the “hard” in hard real-time with “hard-to-do”, as in very short response-time requirements!

  Suppose you are considering using Linux in a real-time application. Linux was not designed to be a hard real-time operating system. It was intended for desktop and server use, where the response requirements are much more soft than hard. Demands like streaming media weren't around until recently. So Linux evolved as a “general purpose” (non real-time?) operating system, certainly not in the manner of the traditional real-time operating systems (RTOSes) like QNX or LynxOS.

  Today, the popularity and power of Linux are driving developers to try to use it in a growing number of embedded and real-time systems. Thanks to its open source, Linux is quickly responding to the demands placed on it — both in terms of reduced footprint for embedded, and in terms of improved real-time performance. And there are multiple approaches to meeting these needs, largely because there are many different markets and applications to which Linux is being adapted, which have differing requirements. Thanks to the open source development model of Linux, some of these adaptations will remain add-ons or patches, while others will find there way into the mainstream Linux kernel.

  At the moment, a number of scheduler and preemptability enhancements are being developed and experimented with. Whether these become part of the standard kernel remains to be seen. With increased demands coming from applications like streaming media and VoIP, we can expect the standard kernel to evolve in its responsiveness due to design improvements. Ongoing hardware improvements, including processor, memory, and disk performance increases, will also play an important role. However, none of these changes are likely to improve worst case response times to the tens-of-microseconds level. For now, it will be necessary to be content with multi-millisecond responsiveness from the kernel, even after adding various scheduler and preemptability patches. But this should be fully adequate for the “best efforts” bottom-line requirements of soft real-time apps like streaming multimedia, VoIP, and gaming.

  So what do you do if you really must meet timing constraints measured in the tens-of-microseconds range, while still using Linux as the system's OS? That's where solutions like RTLinux and RTAI come in. They provide multi-microsecond worst case response times, by running outside the Linux sphere. Essentially, they own the system and can respond to events nearly instantaneously, and without interference from Linux. RTLinux does this via a clever trick — it creates a software Interrupt controller which Linux thinks is the real thing (i.e. RTLinux emulates the system's Interrupt controller). In effect, having RTLinux (or RTAI) is like dedicating a separate processor to handle designated real-time events — since it literally commandeers the system when it needs to respond to a real-time event, in a manner that is transparent to Linux.

  But there is a price to pay, which is that you must design your software to use the RTLinux-provided functions for hard-real time processes, rather than the normal Linux kernel-provided functions. Yes, it's as though you have two processors in the system (a faster one, and a slower one); but it comes at the cost of two APIs — one for real-time, and one for normal system functions. In short, with RTLinux (or RTAI), you can't simply run Linux applications and have them magically benefit from the presence of RTLinux (or RTAI) in the system. However, the proponents of this approach contend that it's a better way to program real-time systems anyhow — i.e. partitioning the application into separate real-time and non real-time software. Maybe so, but I suspect most programmers would prefer to simply program in Linux if the Linux kernel could meet their timing constraints acceptably. Until we have multi-gigaherz processors, however, don't hold your breath waiting for microsecond level response times from the Linux kernel itself.

• How do you characterize and specify real-time performance? — if you reviewed the responses from our panelists, you will have noticed that our expert panelists waxed forth with a long list of buzzwords they like to use to rate their products vs. their competitors'. There doesn't seem to be a lot of consensus around a set of three or four key parameters that can be used to objectively characterize competing solutions. Even where the concepts appear to be similar, there isn't agreement on precisely how they're defined or measured.

  Not surprisingly, each vendor seems to prefer specs that show off the unique strengths and technical advantages of their product. But wait; before condemning that tendency, think about the fact that each vendor has probably chosen to focus on a particular set of niches, within the nearly limitless market for embedded systems. Naturally, they've optimized their offering to match the needs of their target customers. Taken as a whole, the result is a wide variety of choices for customers, supporting a broader range of applications than would otherwise be possible.

  So how do you decide whose solution is best for your project? Given the diversity that has resulted from the adaptation of Linux to an incredibly wide range of applications, plus the inherent complexity of the real-time systems themselves from both a hardware and software perspective, the best bet is to benchmark several alternatives on a system that models your hardware and software requirements.

  Which brings up one of the most wonderful aspects of Linux. What other operating system is supported by dozens of OS suppliers, who all support a common set of APIs? With what other solution do you get the opportunity to benchmark your application on multiple offerings without rewriting it?

And the winner is . . .

In my opinion, the recent highly visible discussions (some call them “scuffles”) over alternative real-time Linux architectures and approaches are great for Linux. They are a healthy symptom of a vibrant market, comprised of strong and creative competitors who continually contribute to a growing technology pool. What we have going on here, is a combination of democracy and capitalism at its best.

The effect of all these competitively motivated efforts to bring out the best solutions, is that the entire market has the opportunity to share the bright ideas of an energetic and enthusiastic group of individuals and companies with a common purpose — solving real world problems better than what has been done before. Because Linux is open source, the fruits of that process are shared in a way that yesterday's proprietary development models could not offer.

Who wins? We all do!

Introducing . . . the “Real-time Linux Quick Reference Guide”

You've probably realized, by now, that there are many ways to add real-time capabilities to Linux based systems.

Certainly, one method is to throw hardware at the problem, by running faster processors or employing specialized hardware. For example, specialized peripheral controllers and digital signal processors (DSPs) can offload critical real-time tasks from the main system CPU.

However, assuming you want to use the main system processor to manage real-time system events, there are a great many options from which to choose. Given the somewhat bewildering variety of alternatives, we've assembled the “Real-time Linux Quick Reference Guide” which we hope will assist you in locating Linux-based solutions that match your system requirements. The guide is available online, here, and will be updated periodically.

Don't forget to also take advantage of the tremendous amount of information available via the LinuxDevices.com search engine, by searching with “real-time” as the keyword. Try searching through our news, articles, links, and products databases.

---

Do you have questions or comments on this article? talkback here

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.