Here's how to get started developing the code for your next high-altitude atmospheric research project, or anything else where a hard real-time task needs to communicate with other software.

RTLinux is a hard real-time OS, as opposed to soft real-time systems or those that make no scheduling guarantees whatsoever. RTLinux is a hard real-time kernel that runs Linux or BSD as an idle task, whenever there are no real-time demands.

The dual-kernel approach taken by RTLinux requires a slightly different approach to real-time programming. Real-time code is written as a kernel module that is managed by the real-time kernel. All user management code is run as a normal process managed by Linux, communicating through a variety of mechanisms. This code separation abstracts real-time code into a simpler code base, simplifying development of both the real-time code and the management interface.

Some real-time OS approaches try to force the kernel and user-space code to do well with both real-time and nonreal-time scheduling constraints. Instead, RTLinux takes the normal UNIX approach, where a tool is written to do one thing, and do it well, rather than cram everything into a one-size-fits-all system. This results in a simpler system and encourages simpler code, while simultaneously providing a deterministic real-time environment that is constrained only by the hardware powering it.

The RTLinux API

Those of you who have used real-time systems before know that every system has a “special” API, where “special” is usually replaced with a more colorful term. RTLinux's API, however, is based on POSIX PSE 51, which is a standard designed for embedded real-time systems. This capability enables developers to use the standard pthread_* calls within a real-time environment. This means that all of the POSIX calls, such as pthread_create(), are available to real-time code, along with all of the mutex calls, condition variables, etc. For cases that introduce nondeterminism and are not covered by the standard, RTLinux provides extensions to make life easier for the developer.

From the user-space perspective, the nonreal-time code is exactly like any other Linux process. Once the real-time components have been abstracted out, the remaining code is free to behave like any other application, without fear of interfering with real-time operations. A management front end can be written in GTK+ (or Qt, of course), talk to a remote database or even host an entire Oracle instance directly on the real-time system. Nothing done within Linux can affect the execution of real-time code. While this is not license to write sloppy user-space code, it does remove any worry of complications arising from someone mishandling a Java garbage collector or transferring a large file over NFS while the machine is controlling a robotic arm.

As a side note, code running in the real-time kernel also has access to the entire API of the Linux kernel. However, the Linux kernel was not designed with real-time constraints in mind, and many calls are not always safe. Indeterminate blocking may occur, depending on the call chain. It is up to the developer to decide which calls are safe for the environment and problem at hand, and RTLinux provides real-time equivalents of some commonly used functions. As most hard real-time problems involve direct interaction with hardware, having the kernel API available for dealing with tasks like PCI device initialization can prevent many headaches. As demonstrated later in this article, there are places where use of the kernel API is entirely safe, so all is not lost.

Communication between Components

How is real-time code managed, if it is off running on its own in the real-time kernel, while the rest is running as a normal Linux task? RTLinux provides a few answers to this problem and solves a variety of needs in differing situations.

First, the most common communication model is the real-time FIFO. Anyone who has used a normal FIFO under Linux (as created with mkfifo) is familiar with how this works. A process on one end writes to a FIFO, which appears as a normal file, while another one reads from the other end. With RTLinux, the reader might be a real-time process, while the writer is a user-space program shuttling directives to the real-time code through the FIFO or vice versa. In either case, the FIFO devices are normal character devices (/dev/rtf*), and both ends can interact with the device through normal POSIX calls, such as open(), close(), read() and write().

This approach works well for many applications because the user-space code simply has to work with a normal file for communication. From the real-time perspective, the calls to push data into the FIFO are nonblocking and have little impact on execution. However, as real-time code can never be blocked while waiting for user space to work through the data, the FIFO calls allow data to be overwritten, rather than block the real-time caller. This means that if the real-time kernel is under heavy pressure and never schedules the Linux thread (and by extension, your user-space code), the FIFO may fill before user space can read it. In this situation, the FIFO either should be allocated with more memory to buffer with, or it should flush the FIFO to prevent user space from getting dated or corrupted data.

Another IPC mechanism is the mbuff driver. This is a shared memory system that allows kernel and user-space code to share access to the same memory region. For some applications, this is a very good way to share information, especially when access methods might be nonsequential, as is the case with real-time FIFOs. Also, as with FIFOs, the real-time code cannot be blocked while user-space applications handle the data, so there are no synchronization methods between the two sides. Should this be needed by the programmer, it is possible to coordinate access via certain bit flags in the region, but one must be careful not to block real-time code unintentionally.

A third system that merits mention is RTLinux's softirq system. It is possible for real-time code to create a software-based IRQ, where a handler is installed under the Linux kernel that runs as if it were intercepting real hardware IRQs. But in this case, the IRQ is really generated by a driver in the real-time kernel. This is a very efficient method of safely accessing potentially blocking calls in the Linux kernel. For example, the RTLinux call rtl_printf(), which behaves like Linux's printk(), allows safe real-time printk() calls by pushing data into a buffer and signalling a software IRQ. The handler installed under Linux intercepts this and safely moves the data into the normal kernel ring buffer without potentially inducing blocking.

A Simple Example

Enough talk–let's go through a simple example, demonstrating real-time code, threading and usage of the real-time FIFOs. In order to run this example, you'll need to be running RTLinux, but configuration is detailed for you either in the RTLinux Open download or the RTLinux Professional distribution and won't be covered here. You can find both versions on-line, here.

At this point, we assume a running RTLinux kernel with the basic RTLinux modules loaded (those loaded with insrtl). Rather than dragging you through the normal “hello world”, we'll do something slightly different. Here, a user-space process will communicate to real-time code, directing each one to perform a different step in running a real-time “hello world”.

Before we dive into the code, let's briefly discuss the idea. As with most RTLinux applications, there are two components: a real-time module and a user-space management piece. In this case, the real-time code consists of two threads and a real-time FIFO handler. The FIFO handler watches a FIFO for commands from user space. As the control handler receives requests, it hands them off internally to the destination thread through another FIFO. The two destination threads generate the “hello” and “world” as periodic real-time code. The control commands that they receive from the FIFO handler direct each one to alternatively write “hello” and “world” to another FIFO. The user-space code simply reads the two FIFOs coming from the two hello world threads, and periodically pushes a command to the handler directing them to switch roles.

First, there is a simple header file that both the real-time and the user-space code share. This defines some of the values pushed across the FIFOs as commands:

 #define NOOP                         0
 #define HELLO                        1
 #define WORLD                        2
 #define STOP                         3
 
 struct my_msg_struct {
        int command;
        int task;
 };

The four states direct the actions of the real-time threads and are sent from user space to the real-time thread over the control FIFO, wrapped in the my_msg_struct structure. The task field directs whether the command should go to thread 1 or thread 2.

The Real-Time Code

First, we'll review the real-time component in this exercise. The fact that it is a kernel module shouldn't frighten you; this should look simple to anyone who has done POSIX code before:

This is just like header information you might see in any other application. There are two real-time threads generating the “hello” and “world”, so we store their thread IDs in the pthread_t tasks[2] array. This way we can use the IDs to cancel the threads during cleanup. The function declarations are standard forward declarations for our real-time thread and the control FIFO handler.

Listing 1. Initialization Code

Listing 1 shows all of the initialization code. As you can see, this is the method used in a normal Linux kernel module. In fact, all of the Linux kernel facilities are free for use at this point. As the real-time kernel does not directly support memory management, any preallocation should be done here. Real-time FIFO creation also needs to occur in this function, as after it completes, you are operating in real time.

The fact that we explicitly destroy existing FIFOs may come as a surprise, but it is a good idea in general. This makes sure that this code has explicit use of the FIFO, and that there are no resources left open from other modules. If two modules use the same FIFO by accident, the data will intermingle and appear to be garbage. So, we destroy the FIFOs and then create them as needed. The buffer size chosen here is arbitrary and should be tuned to your needs in practice. The FIFOs themselves each have a special use. The first is for user-space to real-time communication. The second two are for the control thread to push data to the real-time threads, and the last two allow the real-time threads to push data back to user space.

For those familiar with normal POSIX thread management, the next few steps should be straightforward. We initialize a thread attribute in order to set the thread priority and then pass it along with a call to pthread_create() in order to direct thread creation. We do this twice, as there are two real-time threads: one for “hello” and one for “world”.

The last line is an RTLinux-specific call; this registers a function handler for use with the control FIFO. As data is pushed into the control FIFO, this function is called to read it, interpret the data and push it on to the correct thread:

 void cleanup_module(void)
 {
     pthread_cancel (tasks[0]);
     pthread_join (tasks[0], NULL);
     pthread_cancel (tasks[1]);
     pthread_join (tasks[1], NULL);
     rtf_destroy(0);
     rtf_destroy(1);
     rtf_destroy(2);
     rtf_destroy(3);
     rtf_destroy(4);
 }

Like init_module(), this call is executed within the context of the Linux kernel. All we have to do here is cancel the two threads, join them and then destroy all of the real-time FIFOs we were using.

Listing 2. The Real-Time Thread in Its Entirety

Listing 2 shows the real-time thread in its entirety. We passed the task number along from the init_module() call, so we cast that from void. The FIFO this thread will use is going to be one up from the task id, as FIFO 0 is the control FIFO. In order to be safe, we don't want the threads filling the FIFO until user space is ready, so we stall the code by setting the command to NOOP. Following that, we make a call using an RTLinux extension to make the thread operate in periodic mode. This saves us the work of trying to schedule things in a more complicated fashion during the main loop. With this one call, the thread becomes periodic, getting scheduled every half-second (500,000,000 nanoseconds). On each scheduled wakeup, the code attempts to read from the control FIFO. As it was opened in nonblocking mode, it will return immediately if there is nothing to read, otherwise it will fill out the message structure. Based on the value present in the structure, it either writes a message to a FIFO or cleans up with the POSIX I/O calls and quits.

Listing 3. Real-Time Code–Handler for the Control FIFO

Listing 3 is the last bit of the real-time code and is the handler for the control FIFO. This is essentially a callback that is executed when there is pending data on a FIFO. In this case, the execution is simple. It reads a message from the control FIFO, and based on the task designated in the structure, it pushes the data on to the correct thread via another FIFO. The user-space application easily could have written to the thread directly over this second FIFO, but this step was taken in order to demonstrate the handlers and communication between real-time components. Note that we also have used the POSIX calls here, opening and closing the handler and interthread FIFO on each callback. While this adds overhead that could be avoided easily by opening the FIFOs once and tracking the open file descriptors, we use this approach to demonstrate the simplicity of the POSIX I/O calls.

The User-Space Code

This section is very simple. The user-space code consists of a small C application that interacts with a couple of files, doing reads and writes. These reads and writes to the FIFOs direct the real-time threads, controlling which one is in charge of “hello” and “world”, eventually triggering a shutdown.

Listing 4. A Normal User-Space Application

Listing 4 shows a normal user-space application. A few header files are needed, including our common definitions with the real-time code, along with other normal I/O headers. We declare a few variables and then open the real-time FIFOs. They are treated as normal files, so it's a simple matter of using normal open() calls on the /dev/rtf* entries. If you compare the FIFO numbers with the real-time code, you will see that /dev/rtf0 is the control FIFO, while 3 and 4 are the two channels to the real-time threads:

      msg.task = 0;
      msg.command = HELLO;
      write(ctl, &msg, sizeof(msg));
      msg.task = 1;
      msg.command = WORLD;
      write(ctl, &msg, sizeof(msg));
      hello_thread = 0;

If you refer back to the real-time code, we started the real-time threads in a stalled state, so that they don't actually do anything. The first thing we need to do is write commands to the control FIFO to start them up. To do this, we fill out a structure for thread 0, with a command value of HELLO, so that it will be told to write “hello”. Then the same is done for thread 1, telling it to print “world”. For future reference, we remember that thread 0 is in charge of “hello”.

Listing 5. The Main Loop

Listing 5 shows the main loop, and there aren't any real surprises. As there are two files we are watching, we set up a normal file descriptor set for select() to work with. The real-time code is running every half-second; here we sample more often, just to keep things simple. In a real environment where the intervals are tighter (tens of milliseconds or less), the data might be timestamped or transported in a different way.

Here, however, it's simpler to just poll for data. Once we read it, the loop dumps the string, and the FIFO it came from, to stdout. Just to demonstrate interactive FIFO handling, the code talks to the control FIFO every 20 iterations, telling the real-time threads to switch roles:

      msg.command = STOP;
      msg.task = 0;
      write(ctl, &msg, sizeof(msg));
      msg.task = 1;
      write(ctl, &msg, sizeof(msg));
 
      close(fd0);
      close(fd1);
      close(ctl);
 
      return 0;
 }

Once the main routine completes, there is one more step needed to shut down cleanly. The real-time threads need to be turned off. If we failed to do this and just exited, nothing bad would happen, other than the real-time code would continuously overwrite the FIFO buffers. Instead, we send a stop command to the control FIFO so that the threads stop working. As we will see in the demonstration, the module still will be loaded, but it will no longer be causing any significant resource utilization.

Running the Example

Running the example is very simple and is no different from any of the other example programs that come with RTLinux. As mentioned, we assume that at this point the normal RTLinux modules are loaded into the kernel. First, you need to load the real-time code:

 insmod thread_mod.o

Now the real-time code is up and running. In the real-time kernel, the threads already are configured as periodic and are executing, just not generating anything useful. Now, start the user-space application:

 ./thread_app
 FIFO 1: hello
 FIFO 2: world
 FIFO 1: hello
 ...
 FIFO 1: world
 FIFO 2: hello

This will continue until the user-space code completes 1,000 reads from the file descriptor set. Note that the FIFO outputs will reverse as the code directs the control thread to reverse the roles. This may not happen completely in sync, as the real-time kernel is under no obligation to handle user-space code if it doesn't have time. This means that the code to switch roles might only get through the first command, but not the one directed at the second thread. This does introduce potential complexity, but delaying user code in deference to real-time operation is rarely a bad thing, if ever.

Conclusion

That concludes a simple introduction to RTLinux, its concepts, API and a short example. It is a lot to digest at once but should serve only as a starting point. For more information on RTLinux, RTLinuxPro and other FSMLabs products, check out www.fsmlabs.com.

---

About the author: Matt Sherer works for FSMLabs, Inc., developers of RTLinux. He lives in Socorro, New Mexico, where he tries to maintain a balance between writing, coding and enjoying the Land of Enchantment. Matt can be reached at [email protected].

---

Copyright © 2002 Specialized Systems Consultants, Inc. All rights reserved. Embedded Linux Journal Online is a cooperative project of Embedded Linux Journal and LinuxDevices.com.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.