[Updated May 12, 2002] Foreword: In part three of an Embedded Linux Journal series of articles by Kevin Dankwardt on Real-time Linux, Dankwardt reviewed the sub-kernel approach as used in RTLinux and RTAI and provided some benchmark numbers. Following publication of Dankwardt's article . . .

• MontaVista Software's Kevin Morgan issued a response to Dankwardt's article in which he offered “a few clarifications (or points of view)”.
  
• Next, Victor Yodaiken and Matt Sherer (of FSMLabs) reacted to Kevin Morgan's response to Dankwardt's article, taking exception to Morgan's assertion that RTLinux is “not appropriate for the placement of comprehensive applications”.
  
• Then, Kevin Morgan (of MontaVista) clarified the status of MontaVista's kernel preemption enhancements and responded to several other issues raised in Yodaiken and Sherer's earlier comments.

In this round, Karim Yaghmour provides the RTAI point-of-view — drawing attention to the nature of the API, the usability of the methods, and distinctions in the overall openness of the specific approaches that are being compared.

I've been following the current RTLinux vs. Preempt debate at LinuxDevices.com from a distance, and I thought I would share some of my thoughts from the RTAI perspective.

First, I have no benchmarks to provide. I will, however, refer you to Kevin Morgan's words of wisdom on the use of benchmarks.

What I would like to point out above all, nevertheless, is that this debate seems to omit two crucial elements: the nature of the API provided; and the usability of the methods suggested. Interrupt latency is certainly important and so is I/O throughput, but there is more to real-time than this.

Of course, a careful programmer can design a Linux system with the proper process priorities and the appropriate custom device drivers to obtain hard-real-time behavior. If a developer can choose the exact processes that run on his kernel and understands their source code inside-out, then he can certainly deliver a deterministic software solution using the Linux kernel alone.

This isn't the problem. The problem is that the kernel alone doesn't make a Linux system. A developer therefore needs at least half a dozen other open source/free software packages to get the end system he's looking for. This is where the preemption/low-latency approaches hurt most, since most developers know little about what the additional software does.

Even with custom kernel drivers, there's still a big problem. Namely, neither the device driver API nor the user-level API provided by Linux are anywhere near deterministic. That's the crux of the problem.

As I said, if you know this API and the kernel's internals well enough, you can certainly obtain deterministic behavior — but most people can't afford the time to dig this deep in the kernel. For all they know, even if they did dig deep enough and understood how everything comes together, there is no guarantee that the next kernel version would behave the same.

Don't get me wrong; I'm not saying MontaVista's approach doesn't work. On the contrary, I'm sure they've delivered many functional products to their clients with this approach. Their successes, however, aren't necessarily useful to the open source community since they rely on the OS vendor's (i.e. MontaVista's) in-house know-how.

That being said, I'm not siding with Victor either. Among other things, RTLinux suffers from the same problem as MontaVista's approach, since it is mainly a product sold by a company and its users therefore depend on that company for their own product's success. Again, this doesn't mean FSMLabs can't deliver. They've probably helped a couple of clients indeed. But it remains that these clients, too, must rely on an OS vendor's (i.e. FSMLabs') in-house know-how.

As Kevin Dankwardt pointed out in his article, only the RTAI micro-kernel is “maintained in an open-source manner where independent outsiders have contributed.” (Kevin also mentions the Love preempt patch, but I'm only considering real-time micro-kernels here, since I don't think the preempt patch is useful to common 'mortals' who want hard-real-time in Linux.)

Beyond these most basic facts, RTAI provides a slew of features not found in any other open source hard-real-time Linux offering. These include . . .

• hard-real-time capabilities in user-space (with capabilities to access Linux's non-deterministic API)
  
• a deterministic API (both in kernel space and in user space)
  
• dynamic memory allocation
  
• real-time networking (UDP over ETH)
  
• etc . . .

What is most important about RTAI, nonetheless, is the fact that the people behind its implementation, its distribution, and its know-how do not belong to any single organization. Their knowledge and their code are available in open source to all those wishing to program hard-real-time applications in Linux.

The RTAI development team therefore firmly believes that your first stop in shopping for hard-real-time programming in Linux should be the RTAI project website. Of course, the use of the word 'shopping' is a figure of speech, since RTAI is entirely available in open source.

If MontaVista were to provide detailed information and actual usable examples of the deployment of their method, or if FSMLabs were to release their latest codebase in open source, we would be glad to point you to their work too.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.