The father of real-time Linux talks to ELJ about the past, present and future of RTLinux.

Victor Yodaiken, project leader for RTLinux, is considered by many to be the father of “real-time Linux.” We first heard from him in his article “Introducting Real-Time Linux” which was published in the February 1997 issue of Linux Journal. Yodaiken, currently president of FSMLabs, a software company headquartered in New Mexico and with developers in the US and Europe, has been working on systems in both industry and academia since the early 1980s when he worked on one of the first commercial distributed fault-tolerant UNIX systems and wrote one of the first ports of UNIX to the 68000.

The Past

ELJ: Could you provide a brief history of RTLinux? Where did it come from? How did the idea for RTLinux arise?

Victor: I was interested in real-time for quite some time, and I was particularly interested in how to fit real-time into an operating system properly. I had also been doing some work on how to prove programs to be correct, program verification, which I think is eventually going to be critical for real-time systems.

ELJ: You were doing this as an instructor or professor at New Mexico Institute of Technology (NMT)?

Victor: I came up with RTLinux at NMT, but I had been working on real-time and related issues even back to my graduate student research. I started working at real-time programming to get a feeling for what were the hard things to understand, and it came to me that one of the hardest and most difficult things in trying to validate that a real-time program works, was to show that the non-real-time components didn't interfere with the real-time components. This led me to think about how you could make sure this didn't happen. Also, I was getting involved in Linux at the time, some of my students were working on the PowerPC Linux port. I was very happy with Linux. So, being interested in Linux, and being interested in real-time, and trying to figure out a way that you could make programs reliable by separating out the real-time and non-real-time parts…they all came together, and I thought of this “trick” for running Linux as a sub-thread of the real-time operating system and “fooling” it about what was happening with the interrupt controller.

ELJ: It's a brilliant technique! Really, it's one of the most elegant things you can do with computers because, in fact, they simulate hardware in software.

Victor: Some of the complaints we got about the patent were from people who said “this is so obvious that it's obviously been done before”, and I really sympathize with that because when I thought of this idea I thought, “this is so obvious that either someone did it before, or there's an obvious reason why it won't work”. Having known about virtual machines, which is a very old idea, predating actual real computers, I was sure that someone had done this. The closest I found was actually something I had known about–an operating system called MERT that was done at Bell Labs in the 70s.

ELJ: What did the MERT developers do? In what ways is RTLinux similar or different from what was done?

Victor: They did an entire virtual layer that ran a real-time operating system on top of an emulation of most of a processor. That was the big difference from other people who had done virtual machines (before). In many ways, the innovation in the RTLinux method is that it emulates such a small part. Everybody knew you could do this kind of stuff with virtual machines, but (the perception was that) virtual machines were very slow (or needed special hardware as in the IBM mainframes). The nice thing about RTLinux is that we have to emulate only this tiny little piece of the hardware (the interrupt controller). Everything else can go on the raw hardware–all the Linux I/O, all the general purpose systems I/O, goes directly to hardware. We don't have to emulate all that, we don't have to emulate very much at all. We just have to emulate this one little piece–interrupt control.

ELJ: How long did it take you to unplug yourself from the university and “go private”?

Victor: It took a long time, and actually RTLinux development really slowed down, because after we had done the first versions I got drafted to be the department chairman. So I spent two years doing RTLinux in my very, very little amount of spare time. I had all this idiotic administrative work, I was teaching, and I have a wife and children I like to see sometimes too, so I had almost no time to work on RTLinux. So development was pretty slow for a while. Finally, I decided that I really had to devote myself full time to RTLinux. I've been on leave (from the University) and working on it full time only since January [2000].

ELJ: Are you expecting to go back to the academic environment, or do you think you'll stay commercial?

Victor: I have a lot of research interests that I would still like to work on at some point, but making RTLinux an “industrial strength” product will come first. I'm going to stay with the company (FSMLabs) as long as it takes for that, full time. If I can keep one foot in academia at some point, that would be nice. But my agenda right now is to make RTLinux “industrial strength” and to make FSMLabs a going concern that's on its own feet and not necessarily dependent on me anymore. We're growing pretty fast, so I think we will get there.

The Present

ELJ: I understand you're nearly ready to release version 3 of RTLinux, which is currently in beta. Could you briefly describe what's new in that release?

Victor: We're closing in on a final version 3. There are a whole list of changes since version 2 . . .

• RTLinux now runs on PowerPC and on Alpha as well as x86, and we are hoping to release a MIPS version, too. We're getting amazing performance on PowerPC embedded systems and on the Alphas. We're impressed with those. We've seen better performance on PowerPC than on the Pentium systems, mainly because the interrupt controller's and timer's non-ISA systems are less intrusive than on the Pentium motherboard designs. We did the Alpha port because it was our most requested next chip. There are people who want to use the Alpha where they are very compute hungry–especially in things like engine control. In high-end real-time where you need a lot of compute power, people are really pushing the limits of what's available in current processors.
• We have some enhancements in SMP operation which allow you to dedicate processors to real-time event handling.
• We have a new ability for real-time signal handlers in user tasks, where the user process makes an rtlinux_sigaction which will work like the POSIX sigaction, except the signal handlers run in real-time and there's an extension to allow us to catch periodic interrupts. As a result, the user process can designate a function to operate within hard real-time deadlines. And those functions run in the address space of that process, so they can share data with the process, and call function libraries of the process. For example, you run a user program, and that program may need to collect data from an A-to-D device every 200 microseconds. The two ways of doing that are: writing an RTLinux thread that runs in kernel space and collects data from this device and dumps it through a pipe or via shared memory to the user space; or installing a RTLinux signal handler which reads data from the device into the process address space and makes it a lot easier, especially for simpler data acquisition systems and for systems with complex user libraries. One good example of this will be people who want to take OpenGL and other graphics libraries, read data in an do some processing on the data in real time–they'll be able to use this facility. Our customer for this was the Jim Hanson Creature Shop–we did it for them but we're going to make it available generally.
• Another thing is the use of the GDB (GNU debugger) with RTLinux threads. You can set up RTLinux threads so that when an RTLinux thread makes an arithmetic error — divide by zero or reaches for the wrong memory address — it will trap, and the whole RTLinux system will halt while Linux itself continues, and on the same machine you can run GDB and inspect the state of everything and do single stepping. All the facilities of GDB are available on that same machine. Also, you can use all of the standard front ends of GDB, such as DDD. As far as GDB is concerned, it's debugging something that's on a remote machine–but it's actually on the local machine.

The Future

ELJ: What are we likely to see in terms of new developments for RTLinux over, say, the next year?

Victor: We're going to provide real-time networking. We have a very strict subset of the Internet protocol (IP), and possibly UDP, and we want to make it possible to layer on top of that, especially for applications that need QoS (Quality of Service). There, we're looking at applications like machine control, moving video, VoIP (voice over IP), telecom, fault tolerance and clusters.

ELJ: Are you aware of the announcement from Lineo about their real-time networking support?

Victor: This is the release by David Schleef. It looks good and takes the immediate pressure off us to release something. I think for many applications it will be enormously useful. Our network initiative is focused in a different area. For instance, we'd like to have a networking system that has hooks for QoS and types of quality service guarantees.

ELJ: Other plans?

Victor: We want to do some fault tolerance tied into the networking. Also, we want to do some other ports. A MIPS port is underway, and probably a StrongARM port will be next, although SHARC is also interesting.

ELJ: Have you given some thought to supporting more of the wide variety of system-on-chip devices that are now available?

Victor: The problem is, there are so many of them. We want to encourage and help independent developers and manufacturers to make RTLinux work on these systems. For many of them, RTLinux should run straight off. We don't have too much of a driver problem (since most of the hardware is controlled by Linux). If the interrupt controller is compatible, we're fine. For many of the processors, those are on-chip, so it's not a problem. The problem with ARM and MIPS is that there are so many variations of the cores and of the controller devices, and ARM itself has some difficulties getting good Linux performance.

ELJ: Anything else?

Victor: Yes, we're looking at making a stand-alone RTLinux that can run without Linux entirely: a little bit extra initialization and self-support, so we can make a ROMable RTLinux that can come up first, or that can stand-alone on a very small system.

ELJ: Would it replace the BIOS?

Victor: We actually could be part of the BIOS. We could take RTLinux and mix it in with a BIOS of some sort. Have the BIOS do the initialization, just as Linux does the initialization for us now, and then pop up into RTLinux as the operating end of the BIOS.

ELJ: Would you expect RTLinux to be used without Linux, in that case?

Victor: Yes. It doesn't mean that the same POSIX API that you use in Linux would be available without Linux. There are two uses for stand-alone RTLinux. If you really have to squeeze something down to a small size, MiniRTL will run happily in 4MB right now, but we have some interest in seeing it run in smaller spaces with less functionality. The other use is in fault tolerant systems: if you have the RTLinux that could run on its own, and it (the Linux kernel) died, you would still have some usable system there. Alan Cox proposed, as a joke, that a real-time thread could do a pthread_join on the Linux thread and reboot Linux on a crash–but I like the idea, for real.

ELJ: What if you need device support? Would you have access to functions like a Linux driver for IDE hard drives?

Victor: What you'd do is put in what you needed. One of the goals has been to keep everything modular. You can have a bare system that you can put in a ROM, that does nothing other than touch a serial line every now and then. But if you want to put in IDE (drive support), you can do that too. If you wanted to put in IDE capability by calling the BIOS, that would not be hard to do.

ELJ: So the mechanism for calling BIOS functions is not too complicated?

Victor: No, I don't think so.

ELJ: Would it be easy to use Linux drivers?

Victor: Modifying a Linux driver to work with RTLinux is not very difficult right now. It's still a job that requires “by hand” work, and we're hoping to move beyond that. But it's not a very hard job to do that.

On Real-Time Standards

ELJ: What do you think about the various proposals for a real-time Linux standard that are floating around?

Victor: The intention with RTLinux version 3, is that you can write RTLinux threads in something that looks very much like the user-land threads environment. What we've done over the last year (starting from version 2 which gave us the initial pthreads environment), is integrate the POSIX threads/signals calls into our existing set of operations. So one example is if you want to catch an interrupt in version 2 (also supported in version 3), you make a call to rtl_requestirq. But in version 3 there's an alternative way to do this which is to use sigaction, which is the standard POSIX call. What we would like is for programmers who come to the RTLinux environment, and are familiar with the pthreads and POSIX programming environment, to have very little difficulty in figuring out how to write RTLinux programs. We would very much like them to not have to learn a new environment, and also particularly not have to learn much about how Linux works internally.

We don't want to force programmers to do Linux kernel programming. We want them to write things that are nice and straightforward like threads programs. There are two reasons: one is that people should not have to learn the complexities of Linux internals, which is always changing and is a very complicated environment; and the other is that real-time programming is hard enough without having to learn something else. So we want to give people a familiar environment to work in.

ELJ: So that's the impetus for using POSIX as the real-time function interface in RTLinux version 3?

Victor: Yes. The POSIX 1003.13 specification allowed us to conform to POSIX without having the performance compromises that you get if you go for the more heavyweight POSIX environment. The 1003.13 specification defines four classes of POSIX environments, ranging from a very minimal real-time-only system (called a PSE51, the “minimal real-time environment”) all the way up to a full-fledged POSIX system which is both real-time and non-real-time (PSE54), which is essentially what Linux is. PSE51 looks like a single POSIX process per processor with applications as threads and signal handlers. By focusing on that PSE51, we get the familiar environment of the POSIX threads programming interface but we don't have the performances compromises that we'd have to have if we went to the full-fledged POSIX environment. I'm really speaking about things like putting in POSIX file I/O, which there's no way to do in hard-real-time.

ELJ: So it sounds like POSIX 1003.13 in general, and PSE51 in specific, fit the RTLinux architecture very nicely.

Victor: Yes. We've preserved what I think is the biggest advance in RTLinux which is this paradigm that you separate out the real-time parts very vigorously from the non-real-time parts, you decouple the operation of the two components. We don't let the non-real-time interfere with the real-time. That's at the core of RTLinux. And we've kept that while being able to have this nice familiar environment of the POSIX threads and signals. What we realized was that we could conform to POSIX standards, and get a POSIX API without destroying our core value of this decoupling. With the decoupling, we get the performance. With RTLinux we always quote worst-case interrupt response times, and the reason we're able to do that is because the very complex and sophisticated non-real-time environment can't delay the real-time system. We had avoided POSIX in the past because POSIX seemed to make us mix all these things together. But the 1001.13 environments specification allows us to make a very simple real-time environment with a more complex environment, the Linux environment, as a sub-thread.

ELJ: Is anybody else implementing PSE51?

Victor: I think we're the first company to actually implement it, and we're the only people who are actively pushing it right now.

ELJ: And it's an approved standard?

Victor: Yes. What's nice is that we found an existing standard that we can fit into, without compromising performance. There is a real standards body. The standard is not the creature of any one particular company. It's not the whim of anybody's agenda. The standard is technically exactly what we needed, and we're very happy to find that and to work with that framework.

The Patent

ELJ: Okay, let's talk about the “infamous” RTLinux patent. What's up with that patent?

Victor: We're formalizing the license right now. What's clear is that people running applications on RTLinux are free and clear. They don't have a problem; they have a license to use the method and to run their applications on it–there will be no fees or royalties. We would like to find some way of using the patent to reinforce GPL, so that people don't take this method and build proprietary systems on it, without some additional license.

ELJ: Could that potentially be used to block RTAI (Real-Time Application Interface) from being LGPL(Library Genreral Public License)?

Victor: RTLinux is released under the GPL. I have an agreement with Linus that nobody who uses RTLinux with Linux will have to pay royalties. And we're considering how other licenses are going to work out. I'm very much in favor of making sure that people who are doing GPL work or noncommercial research work don't have to go through any paperwork or pay. We're in the process of formalizing how that's going to work out so that we don't get into trouble later on. And this is an area that nobody knows how all these laws interact. But in the meantime, people who use the system, who use RTLinux, don't have anything to worry about–there's a definite commitment there.

ELJ: When will the license become defined?

Victor: It's been very late, and I'm sorry about that, but we had delays that were not our doing. I would like it out there and I'm hoping we can get it through the lawyers in a few weeks.

ELJ: Have you seen a lot of negative reaction to the RTLinux patent?

Victor: I've received a lot of complaints, but also some cheers. (I was happy to see that the week after the patent got trashed on Slashdot, there was an even more hostile mob attacking Linus after the Transmeta announcement.)

Some of the hostility towards a patent is based on principle, people who on principle think software patents are wrong. I respect that position, even though I disagree with it. I think that software patents have obviously been abused, but I think they also have the potential of protecting individual innovators and small companies (like us) from having their work simply appropriated by bigger companies. That's what patents are supposed to do.

Some of the hostility that we see is the same hostility that you see towards GPL — people who don't like the fact that the GPL doesn't permit you to close off open source. I've had bitter complaints about the patent from people who work for huge companies with giant patent portfolios — complaints like “but I want to use this method on our proprietary operating system”. There is an attitude that free software is work that some people will do as a hobby so that others can make some money with it. You know, “thanks for the ideas guys, but now let the professionals take over”.

ELJ: To you, is GPL a fundamental principal, that you believe very strongly in?

Victor: It is something I believe in strongly–RTLinux is GPL and is available for free download on the net at the FSMLabs web site. On the other hand I'm not a GPL purist. I'm more in Linus Torvald's camp than in Richard Stallman's. I see GPL as something that has a lot of good force in it, it restores some balance to skill versus ownership. I very much respect Richard Stallman for his single-minded devotion to free software, because without that we would not have Linux and we would not have this very exciting and innovative software environment that we have right now. So I'm in favor of GPL, but I don't believe that all software should be GPL'd, I don't believe that GPL is the only way to have open source, and I don't believe that people are necessarily evil if they don't produce GPL software. In my opinion, GPL is a critical part of making sure software is a free market instead of a monopoly. I don't know if Richard intended it that way. GPL has been unfairly criticized as being anti-commerce, but GPL is more anti-monopoly and pro-competition. Since I've been flying a lot, I see what happens when one company owns all the gates and doesn't have to care about customer service. GPL gets rid of the gates, like it says on that T-shirt.

ELJ: As the copyright holder of RTLinux, you also will license RTLinux for specific proprietary (i.e., non-GPL) uses, right?

Victor: Sure. If people have hardware that has their proprietary innovations in it, and they need low level software to operate that, and they're going to package the software and the hardware, I don't see anything wrong with them selling a modified non-GPL version of RTLinux (the Linux part, of course, stays GPL). Of course, I'd like them to have a license from us when they do that. The principal of GPL is that if you're distributing software, you must distribute the information in that software, you must tell everybody how it works, and you must allow them to go and tinker with it. But if you're selling a printer, and it's a closed box, and you want to hide how you program the access to the FPGAs in there, I don't necessarily think you should have to release that source publicly. The printer example is to the point, because the Ghostscript folks have been selling non-GPL versions of Ghostscript software to printer manufacturers for many years.

ELJ: Then why not release it under LGPL, which allows users to have a choice between open and proprietary?

Victor: There's an important distinction here. I think that it's okay for people to buy licenses if they want to get out of GPL. They can buy licenses from the copyright holders. That's a reasonable requirement. You can tell people: if you want to work in the free software world, you can take the free software, but you're out there, you have to participate and you have to give back. If you don't want to be out in the free software world, you can purchase a license and get out of (the GPL restrictions).

ELJ: Thanks very much!

---

About the author: Rick Lehrbaum created LinuxDevices.com

---

Copyright © 2001 Specialized Systems Consultants, Inc. All rights reserved. Embedded Linux Journal Online is a cooperative project of Embedded Linux Journal and LinuxDevices.com.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.