In his response to Kevin Dankwardt's Embedded Linux Journal article entitled Real Time and Linux, Part 3: Sub-Kernels and Benchmarks MontaVista's Kevin Morgan writes that RTLinux is “not appropriate for the placement of… comprehensive applications”. But it's hard to figure out what's “not appropriate” about something that works as well as RTLinux.

RTLinux has been successfully used in production systems for eight years. Aircraft simulators, jet engine test equipment, complex data acquisition systems, robots, routers, machine tools — you name it, and RTLinux is running it.

There are, indeed, other proposals for real-time on Linux, but RTLinux is not a proposal. It is tested, stable, in production, ready to run, with a track record of offering microsecond-level performance on a wide range of processors and boards: everything from single board Elan SC520s and Compaq StrongARM based iPAQs, to SynergyMicro Quad PowerPC VME crates, to racks of Pentium IVs.

The endorsements of RTLinux by Red Hat, LynuxWorks, and RedSonic came because RTLinux delivers the performance that customers need, and RTLinux is supplying it today.

The performance of a “real-time” thread running in Linux will degrade rapidly when Linux is stressed, while the same thread under RTLinux will be rock steady — even if Linux crashes under the weight of its load! Try typing 'halt' on a machine running real-time code under RTLinux — you'll see that your real-time code continues to run perfectly even after the Linux kernel has halted. We get this result by an innovation — making Linux into a low priority preemptible thread of the real-time RTLinux core kernel.

The classical really simple RTLinux application consists of a real-time thread that collects data from a device, say every 100 microseconds, and dumps the data down to a Linux process — say, a process executing the shell script . . .

cat < /dev/rtf_daq | MyFFT | ssh [email protected] display

Linux and its utilities provide all the power and features to run the shell interpreter, the file system, the math libraries for computing a FFT, the encryption, authentication, and network connections from 'ssh', and so on — RTLinux core provides the precise timing. We get precise real-time in the RTLinux kernel and we get all the power of UNIX in Linux. We encourage modularity and we make it easy to re-use existing code and systems in real-time applications.

If you can write POSIX threads and shell scripts, you can put together a pretty powerful RTLinux application in a day.

Unfair?

Kevin Morgan argues that all this is unfair: the heavy lifting is done in Linux and if we tried to run a Perl interpreter or two in the RTLinux core kernel, performance would suffer. You could make the same argument about pulleys and levers: if you tried to lift the same weight directly, without using the pulley, then we'd see how strong you really are.

Trying to optimize an operating system for both real-time and non-real-time behavior is too hard, too costly, too expensive, and too frustrating — after all that effort, results are disappointing. Trying to build a monolithic real-time application that mixes time-critical and non-time-critical components is too error prone.

So indeed, RTLinux “cheats” by using the right tool for the job. We optimize the core kernel for real-time and we run a thread that is a general purpose operating system optimized for non-real-time. Instead of trying to build a race car that can move freight, we designed a system that allows us to have the race car work together with a freight train.

Of course it's unfair — that's what's so great about it! “Unfair”, “too simple”, “too fast”, “too reliable”, and “too cost effective”! We plead guilty on all counts. You get 50 microseconds worst case scheduling jitter on an Ampro Pentium SBC, 16 microseconds on an Apple PowerMac, 100 microseconds on a Synergy quad processor VME box, 80 microseconds on an AMD Elan SC520, 50 microseconds on a Embedded Planet RPX860, 23 microseconds on a Tyan/AMD K7 dual processor, and so on. And that's scheduling jitter — interrupt latency is far lower.

Kevin Morgan argues that instead of RTLinux people should try “to solve a real-time requirement more simply in a native Linux environment using the appropriate blend of custom I/O drivers and user level processes”. This “appropriate blend” is an unproven proposal — an “alpha” project on Source Forge. Furthermore, writing “custom I/O drivers” is more complex, not simpler than writing RTLinux threads.

If you get RTLinux today, you can have a 100 microsecond thread running this afternoon. If you get the Linux preemptive kernel today, you get a work-in-progress with a performance goal. Don't try to control a metal lathe or even a network switch with a “goal”.

But more than that, the simplicity argument is just wrong. RTLinux uses a dual kernel module both for performance and to make software engineering more efficient. Modularity is the key to making complex systems work reliably and to getting reliable software out the door fast. Some people, who are used to the traditional “put it all in the same pot” method of real-time programming, will argue that splitting the real-time software components from the non-real-time components is inconvenient — but, in fact, the modularity increases programmer productivity. Programming in the RTLinux real-time environment designed to accommodate real-time, and using the POSIX threads API is much less complicated than programming in the Linux driver environment where the programming interface changes as Linux develops.

Refer to the following table for a comparison of the two approaches:

Table 1: a comparison of the two approaches

More benchmarks

We also did some simple tests. For each CPU on the system, we spawn a thread that sits in a sampling loop. For the MontaVista code, we enable the hardware real-time clock interrupt at a 1024Hz rate (976 microseconds), and for RTLinux, we just reschedule the thread to have a 976 microsecond period. Each thread blocks, waiting for the period to expire, and samples time when execution begins again. By measuring the offset in time, we can see what the inherent jitter is under various loads.

Tests were run on an Athlon 1.2GHz machine with 256MB of RAM, a Realtek 8139 network card, an S3 Trio video card, and a Maxtor drive — a reasonably “off-the-shelf” system. The Linux distribution used was stock Red Hat 7.2. While this is a fairly large system for RTLinux and most embedded applications, it is more congruent with the systems used to test the preemptive kernel. Kernels used for the MontaVista tests started with a stock 2.4.16 kernel, and all RTLinux tests were done with the 2.4.16 kernel and toolchain that comes with the RTLinux/Pro development kit.

Most timing tests seem to be done against a quiescent system or for very few samples. For example, in Kevin Dankwardt's article, he seems to have looked at 1000 samples! When it comes to hard real-time demands, the worst case values are the only numbers that matter. The quiescent approach involves teaching your machine floor employees to clear the floor at 2:00pm on Thursdays, because this is when the device controlling the lathes rotates system logs, and might 'put off' controlling the lathe for a second or two. In order to test a real-world situation, we ran all tests with a user compiling a kernel, another machine pingflooding the test system over a 100 megabit network, and a concurrent set of six 'find /' processes working on the filesystem. The result is a highly thrashed system, and a set of numbers that give a true indication of what kind of timings you can expect in the field.

As you can see, even with the preemption patch, which, according to the config option, is appropriate when you are “building a kernel for a desktop system, embedded system or real-time system”, is anything but. The goal of the MontaVista approach is a system that is much better for real-time than standard Linux and gets 2-3x worse performance than RTLinux, but our results show that under load the “preemptive kernel” actually performs worse than the standard Linux kernel.

Raising the system's load further impaired interactivity under all kernels, but the timings of the real-time tasks remained solid with RTLinux, at 25 microseconds worst case scheduling latency.

Victor Yodaiken and Matt Sherer work for FSMLabs, the originator and commercial supplier of RTLinux.

---

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.