As Embedded Linux becomes established as a solid alternative to many proprietary OSes and RTOSes, demands on embedded Linux developers and providers are increasing. This detailed technical article by Nicholas McGuire sketches the top requirements for Embedded Linux systems including considerations of user interface, network capabilities, security issues, resource optimization, performance requirements… and issues, and compatibility and standards issues.

---

Introduction

Embedded Linux distributions have been around for quite a while. Single floppy distributions, mainly targeting the X86 architecture, like the Linux router project (lrp) and floppy firewall are well known by now. This first step into embedded Linux distributions was accompanied by a fair amount of 'home-brew' embedded Linux variants for custom devices, expanding the architecture range into PowerPC, MIPS, and ARM, from which development kits are now starting to evolve.

Embedded Linux is more and more becoming a usable and easy to handle Linux segment. But what is the position of embedded Linux? Where does it fit into the other embedded OS in the 32-bit market? In this article a few thoughts on the question “Why embedded Linux?” will be sketched out, positioning embedded Linux quite high up on the list of first-choice embedded OS and RTOS options.

The main challenges I believe will be to fit contradicting demands into embedded systems — these demands are . . .

• Simple user interface vs. in depth diagnostic and administrative interface
  
• High level of security vs. open and simple access to the system via network
  
• Resource constraints vs. high system complexity and low response time

---

Part 1: The main challenges in High-end Embedded OSes

What are the main challenges for system designers and programmers in the embedded world? The list given is definitely not complete and reflects lots of personal impressions — it is thus only one view under many — heated debates on what is required may be fought on mailing lists. Consider the following as 'one picture', hopefully offering constructive thoughts on the subject, even if not all may be applicable to some systems.

User Interface

A major point of criticism of embedded Linux systems is their lack of a simple user interface — generally embedded systems have an archaic touch to their user interface. But a tendency that is evolving is to split the user interface into two distinct sections — a simple to use 'system overview' that gives you a general 'system up and running' or 'call the technician' information. And a more in depth interface that allows you to diagnose system operations at an 'expert' level.

This split is not always done cleanly and is not always visible to the user, it will often run on one interface, but this split is anticipated by most interfaces of embedded devices — representing the actual operational demands. Simple to use for common operations — clear and instructive to the maintenance personnel in case of errors. Embedded Linux can provide both in a very high quality if designed to these goals from the very beginning on.

Many embedded Linux distributions offer a web-server giving OS-independent remote access to status information — at the same time maintenance via secure shell can allow insight into the system down to directly poking around in the kernel at runtime without disturbing the systems operation.

Operational Interface

HMI's as machine-tool designers like to call it or GUI's as OS developers will prefer are some sort of generally graphical based interface that should allow close to untrained personnel to inter-operate with specialized hard and software. A problem that arises here is that embedded systems are limited in available resources and fully developed X-Windows systems are very greedy with respect to RAM and CPU usage (if anybody tried out XFree 4.0 on a 486 without FPU . . . at 33MHz let me know how long the window-manager takes to “launch”).

So does this mean forget embedded Linux if you need a graphical interface? Nope! There are quite a lot of projects around: nano-X, tiny-X, and projects that give you direct access to the graphics display like libsvga or frame-buffer support in recent kernels.

Getting an acceptable graphics interface running on an embedded Linux platform is still a challenge even though IBM has shown that one can run Xclock on top of XFree86 in a system with no more than 8MB footprint, generally a 32MB storage device and 16MB RAM will be the bottom line (there are some PDA distributions though that are below that). The Operator Interface will be a simple scale down variant of a “standard” Linux desktop in many cases and this simplifies development greatly as the graphics libraries available for Linux cover a very wide range — with a new widget set emerging every few weeks.

Administrative Interface

Embedded Products have traditionally required skilled personnel to handle error situations or performance/setup issues. This basically is due to a non-standard operating-system model behind all these devices. The goal was to have a intuitive interface at the expert level (and many hours of training . . . ) which limited the potential scope of intervention and at the same time raised maintenance costs of such devices. Embedded Linux takes a different approach — you have a very large and seemingly complete operator interface — a more or less complete UNIX clone — and this allows operators to debug, analyze and intervene with great precession at the lowest level of the GNU/Linux OS.

The advantage is clear — you don't need to learn each product — it's a GNU/Linux system just like a multiprocessor-cluster, a web-server or a desktop system — one interface for the entire range of possible applications. This allows operators and technicians to focus on the specifics of each platform without great training efforts on a per-device basis. Even though the initial investment in training can be relatively high — all attempts to manage complex problems using simple interfaces are severely limited — POSIX II gives a complex and powerful interface to the operator that allows adequate response to a complex system.

Status and Error reporting

Checking the status of the fax-machine or an elevator is not a high-end administrative task and should not require any knowledge of details at all. To this end Linux offers the ability to communicate with users directly via the console (simply printk'ing errors on a text-console) or a web-interface as well as offering a OS-independent active response via voice, email, SMS or turning on a siren if one connects it to some general output pin of the system.

So the resources required for clean status and error reporting are available in Linux and embedded Linux but care must be taken as to what information can be displayed in response to errors as this naturally touches security issues. Error messages need to be clear and status information needs to be informative — “An application error occurred — [OK]” is not very helpful — on the other hand it is not always desirable if error messages include the exact version of the OS/Kernel/application and the TCP port on which it is listening . . . as this could reveal information that allows attacking such a system.

Network Capabilities

High end embedded Systems are not only required to offer remote administration in many cases, but in addition the demand for system update and fixes from remote sites is moving into the demands list. Linux and also embedded Linux offer many possibilities to satisfy these needs at a high level of efficiency flexibility and security, at the same time extending network related feature far beyond common demands.

Network resources

One of the strengths of GNU/Linux is its network capabilities. These include not only a wide support for protocols and networking hardware, but also a wide variety of servers and clients to communicate via network links. Naturally, a system that provides a large number of network resources also needs to provide appropriate security mechanisms to protect against unauthorized access. In this respect Linux has evolved very far — especially the latest 2.4.X kernels provide a highly configurable kernel with respect to network access.

Remote Administration

Reducing costs is a primary goal of much of the technical development effort being done. A major cost factor in embedded systems is long term maintenance costs. Not only the direct costs of servicing the devices on a routine basis, but also the indirect maintenance related costs of system down-times and system upgrades are an important factor. A reduction of these costs can be achieved if embedded systems have the ability of remote administration. This encompasses the following basic tasks:

• remote monitoring of system status (web-interface, logging to a central facility, etc.).
  
• remote access to the system in a secure manner allowing full system access. This can be done via encrypted connections.
  
• the ability of the system to contact administration/service personnel via mail/phone, based on well definable criteria.
  
• upgradeability of the system in a safe manner over the network, allowing not only full upgrades but also fixing of individual packages/services.

A GNU/Linux based embedded system is well suited for these tasks, providing well tested server and clients for encrypted connections, embeddable web-servers as well as system log facilities that are capable of remote logging in most cases. Outgoing calls from an embedded system, that are necessary to satisfy these criteria are also well established in GNU/Linux, allowing for connections to be established via any of the common network types available, including dialing out via a modem line.

Scanning the Potential

The previous section listed a number of tasks that a remote administratable system should be able to perform, but this is definitely not the full suite of offerings a GNU/Linux system will have in the network area. The degree of autonomy of an embedded system can be pushed up to that of a server system — allowing for dial-in support for proprietary protocols to fit into a non-UNIX environment smoothly. NFS, the network filesystem, can not only be incorporated as a client in an embedded system, but also as a server, allowing for a central server or administration system to mount the embedded system for monitoring and upgrade purposes. This way giving virtually unlimited access to an embedded system over the network.
At the same time, all of these services can be provided in a secure manner by running them over VPN's or encrypted lines. This capability of 'stacking' services is one of the strengths of GNU/Linux networking — and again, you don't need to rely on a specialized software package, you can rely on well-tested and widely deployed setups that will give you a maximum of security.

Security Issues

My personal belief is that not so much power consumption or processing speed but security will be the key issue in embedded systems in the near future. Reliability was one of the demands from the very beginning on — security, on the other hand, has been neglected. The more embedded systems become complex, offer extensive user intervention and utilize the ability to interact with local networks and the Internet, the more security related issues are emerging.

Linux Security

GNU/Linux for servers and desktop is well suited for sensitive computer systems. Its security mechanisms are challenged on a daily basis from script kiddies and 'professional' hackers. Although this is not a very pleasant way of getting your system tested, it is a very efficient way. A system that is deployed in a few hundred to maybe a thousand devices will hardly be tested as extensively as the GNU/Linux system.

This means that an embedded Linux or real-time Linux system is relying on the same mechanisms that are being used in servers and desktop systems. This high degree of testing and, at the same time, the full transparency of the mechanisms in use, due to source code availability, make a GNU/Linux system well-suited for systems with high security demands.

Standard services that a Linux system can provide:

• Firewalling and network filtering capabilities
  
• kernel based and user-space intrusion detection
  
• kernel level fine grain capabilities allowing for precise access control to system resources
  
• user level permissions and strong password protection
  
• secure network services
  
• well configurable system logging facilities

These possibilities taken together allow not only monitoring systems with respect to current actions taking place and intervening if theses are inappropriate, but also for detection of system tendencies and response to developments far before failure occurs. This tendency monitoring covers hardware (e.g. temperature detection or system RAM testing) as well as monitoring system parameters like free RAM, free disk-space or timing parameters within the system (e.g. network response time to ICMP package). A vast majority of the hardware related failures are not abrupt, but develop slowly and are on principle detectable — having an embedded OS/RTOS that can provide this service can improve the system reliability as well as the systems security.

Talking to devices

Most embedded systems will have some sort of specialized device that they are talking to, to perform the main system task — may this be a data-acquisition card or a stepper motor controller. These 'custom devices' are a crucial point in the embedded Linux area, as these will rarely rely on widely deployed drivers and have a limited test-budget available.

So to ensure the overall system security, a few simple rules need to be kept in mind when designing such drivers. Regular Linux device drivers operate in kernel space. They add functionality to the Linux kernel either as builtin drivers or as kernel modules — in either case there is no protection between your driver and the rest of the Linux kernel. In fact kernel modules are not really distinct entities once they are loaded, as they behave no differently than built-in driver-functions, the only difference being the initialization at runtime.

This makes it clear why device drivers are security relevant: a badly designed kernel module can degrade system performance all the way down to a rock-solid lock-up of the system. A really badly designed driver will not even give you a hint at what it was up to when it crashed. So drivers, especially custom drivers, must aim at being as transparent as possible.

To achieve this, a flexible system logging should be anticipated. This may be done via standard syslog features as well as via the /proc interface and ioctl functions to query status of devices. The latter also can be used to turn on debugging output during operations, a capability that, if well designed, can reduce trouble-shooting to a single email or phone call.

Aside from these logging and debugging capabilities, a driver design must take into account that there is no direct boundary between the driver and the rest of the kernel. That means the driver must do sanity checks on any commands it receives and in some cases on the data it is processing. These checks not only need to cover values/order and type of arguments passed, but also check on who is issuing these commands — the simple read-write-execute for user-group-other mechanism of file permissions is rarely enough for this task.

RTLinux devices are not that much different from regular Linux devices with respect to security considerations, but they differ enough that this difference should be mentioned explicitly. Noting this for RTLinux only, is due to the fact that my work covers this RTOS variant of Linux, but basically it should hold true for the other flavors of real-time Linux variants (corrections appreciated).

A simple example of setting up a secure RTLinux device would be a motor controller kernel module. This module must be loaded by a privileged user (the root user) and needs to be controlled during operation. To achieve this:

• Load the module and system boot via init script or inittab.
  
• change the permissions of a command FIFO (dev/rtfN) to allow a non-privileged user to access it.
  
• send a start/stop/control command via this FIFO as the unprivileged user.
  
• check the validity of the command and its arguments.
  
• log such events with timestamps and user/connection related information to the systems log facility.
  
• monitor the logged events and follow development of driver parameters during operation.
• document the system behavior in a way that deviation can be located in debug and log output.

If a scheme of this type is followed, then operating a system with custom devices will exhibit a fair level of security. Clearly, a non-standard device will also require an increased amount of documentation and instructions for the operator, as the behavior of non-standard devices can hardly be expected to be well-known even to knowledgeable administrators.

Kernel Capabilities

A feature of the Linux kernel that is slowly finding its way into device drivers and into applications is its ability to perform permission checks on requests at a more fine-grain level that the virtual filesystem layer (VFS) can.

Kernel capabilities are not limited to the normal filesystem permissions of read-write-execute for owner-group-others. Resorting to these capabilities in the kernel, allows controlling actions of the driver, such as introducing restriction on chown or releasing some restrictions like on ID checks when sending signals (which allows unprivileged users to send signals instead of making the entire process a privileged process). These capabilities require a cleanly designed security policy for the drivers. The name of this kernel feature says it very clearly: it's control of capabilities, not a security enhancement as such.

No system is secure or insecure, but some systems can be configured to be secure and others simply can't. The goal of any implementation using kernel capabilities for access control should be to replace global access settings by resource specific access restrictions. By this means, one can prevent the root user from accessing the device altogether as well as give an otherwise completely unprivileged user full access to a specific resource.

— Continued —

---

Story navigation . . .

• Part 1: The main challenges in High-end Embedded OSes
  
• Part 2: Resource Allocation
  
• Part 3: Operational Concepts
  
• Part 4: Compatibility and Standards Issues

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.