Part 3: Operational Concepts

During the development of embedded GNU/Linux projects a few main modes of operation have evolved. These modes will be briefly described in the next sections, showing the flexibility of embedded GNU/Linux. This flexibility is a product of the wide range of hardware Linux and embedded Linux has been deployed on — ranging from commodity… components embedded systems to dedicated hardware SBC's.

Networked Systems

Network capabilities was one of the early strengths of Linux — and very early in the development of Linux, specialized Linux distributions for disk-less clients have evolved. XTerminals based on low end commodity component computers have been around quite a while, from which specialized systems like the Linux Kiosk system evolved as an example of embedded Linux running via NFS-root filesystem.

In its latest version the Linux kernel is fully adapted to boot over the network and run via nfs-root filesystem, allowing for inexpensive and easy to configure embedded systems ranging from the noted kiosk system to embedded control applications that will boot via network and then run in a RAMDISK autonomously. The ability to operate in a disk-less mode is not only relevant for the administration, but also important for operation in harsh environments on the factory floor where hard-disks and fans are not reliable.

A further usage of the network capabilities of embedded Linux is allowing for a temporary increase of 'local' resources by accessing remote resources, may this be mounting an administrative filesystem adding an nfs-swap partition (a cruel thing to do . . .) or simply using network facilities for off-site logging. The network resources of Linux allow moving many resources and processing tasks away from the embedded system, thus simplifying administration and reducing local resource demands.

Performance

Performance issues with nfs-root filesystems and nfs mounted filesystems will rarely be a critical problem for embedded systems, as such a setup is never suitable for a mission-critical system or a system with high-security demands. Nfs-server and client in the Linux kernel is very tolerant towards even quite long network interruptions (even a few minutes of complete disconnection normally will be managed correctly), but this tolerance does not eliminate the performance problems and nfs-root definitely is only suitable for systems where the data-volume transfered is low.

A special case might be using nfs-root filesystems for development purposes, this is a common choice, as it eliminates resource constraints related to storage media and simplifies development. Development on nfs-root filesystems, though, must exclude benchmarking and reliability tests as the results definitely will be wrong. A stable nfs-root environment can offer a filesystem bandwidth well above a flash-media. On the other hand heavy nfs-traffic on an instable network or a highly loaded network will show false-negative results.

Security of NFS

The nfs-filesystem does not have the reputation of providing a high level of security. So nfs-root systems should not be used in areas where network security is low, or on critical systems altogether (for a Kiosk system it may be well suited though). There are secure solutions for network file-systems, like tunneling nfs or SMB via a VPN, but these do not allow for booting the system in this secure mode (at least not to my knowledge). Also SMB, which is a state-full protocol is clearly better than nfs, but again, I don't know of any bootable setup providing something like smb-root. For systems that might use a local boot-media and then mount applications, or log-partitions over the network both SMB and tunneled NFS are possible with an embedded GNU/Linux system.

RAMDISK Systems

RAMDISK systems are not Linux specific, but the implementation under Linux is quite flexible and for many embedded systems that have very slow ROM or media with a relatively low permissible number of read/write-cycles, a RAMDISK system can be an interesting solution. RAMDISKs reside in buffer cache, that is, they only will allocate the amount of memory that is currently really in use. The only limitation is that the maximum capacity is defined at kernel/module compile time. The RAMDISK itself behaves like a regular block-device; it can be formatted for any of the Linux filesystems and populated like any other block oriented storage device.

The specialties of Linux are related rather to the handling of the buffer cache, which is a very efficiently managed resource in the Linux kernel. Buffers are allocated on demand and freed only when the amount of free memory in the system drops below a defined level — this way the RAMDISK based filesystem can operate very efficiently in respect to actually allocated RAM.

To operate a RAMDISK system efficiently an appropriate filesystem must be chosen — there is no point in setting up a RAM-disk and then using reiserfs (at least in most cases this will not be sensible) a slim filesystem like minixfs, although old will be quite suitable for such a setup and yield and efficient use of resources (imposing minor restrictions with respect to maximum filename length and directory depth).

Performance

One of the reasons for using a RAMDISK is file-access performance; a RAMDISK can reach a read/write bandwidth comparable to a high-end SCSI device. This can substantially increase overall system performance. On the other hand, a RAMDISK does consume valuable system-RAM, generally a quite limited resource, so minimizing the filesystem size at runtime in a RAMDISK based system is performance critical. It is a slight exaggeration, but doubling available system-RAM in a low memory setup can improve overall performance as much as doubling CPU speed!

A nice feature available for Linux is to not only copy compressed filesystem images to a RAMDISK at boot time, but to actually let the kernel initialize a filesystem from scratch at bootup and populate it from standard tar.gz archives thereafter. The advantage of this is that the boot-media can contain each type of service in a separate archive, which then allows safe exchange of this package without influencing the base system. Naturally, exchanging the base archive or the kernel is still a risk but at least updating services — which is the more common problem — is possible at close to no risk. If such an update fails, you just login again and correct the setup.

With a filesystem image you generally have to replace the entire image; if this fails, the system will not come back online, and a service technician needs to be sent on site to correct the problem. To put the additional RAM requirement into relation to the services — a system providing a RTLinux kernel and running SSHD, inetd, syslogd/klogd, cron, thttpd, and a few getty processes will run in a 2.4MB RAM-disk, and require a total of no more than 4MB RAM.

Resource optimization

When using a RAMDISK system, a few optimization strategies are available that are hard to use in general systems or desktop systems. These optimizations are related to the files in a RAMDISK system only have a 'life-span' limited to the uptime of the system; at system reboot the filesystem is created from scratch. This allows removing many files after system bootup: init-script, some libs that might only be required during system startup and kernel modules that will not be unloaded during operation after system initialization has completed. The potential reduction of the filesystem is 30-40% on the test system built (e.g. MiniRTL).

Security

As everything else, the choice of the system setup also has security implications, a few of these with respect to RAMDISK systems should be noted here. System security and long term analysis relies on continuous system logs, writes to RAMDISKs are quick, but to an off-site storage media or a slow solid-state disk are delayed, system logs may thus be lost. A possible work around is to carefully select critical and non-critical logs, writing these along with other critical status data to a non-volatile media (e.g. NVRAM). This solution is quite limited as, in general, no large NVRAMs will be available. Alternatively, log-files may be moved off-site to ensure a proper system trace, as access may not be possible after a system failure. When writing logs to a non-volatile media like a flash-card locally one needs to consider the read/write cycle limitations of these devices, as letting syslogd/klogd write at full-speed to a log-file on such a media can render it useless within a few months of operations, making in hardly better than off-site logging.

A clear advantage of RAMDISK based systems is that although the filesystem modifications are volatile as — is the entire system — a 'hack' would be eliminated by the next reboot, giving a safe although invasive possibility to relatively quickly put the system into a sane-state of operations. To enhance this feature, access to the boot-media can be prevented by removing the appropriate kernel module from the kernel and deleting it on the filesystem. In case the boot-media needs to be accessed for updates, the required filesystem/media kernel-modules simply can be uploaded to the target and inserted into the kernel. This strategy makes it very hard for an unauthorized user to access the systems boot-media unnoticed. A reboot puts the system in a sane-state, as noted above — a system can also be configured to boot into a maintenance mode over the network, allowing for an update of the system. These methods are quite easy to implement. For example, such a dual-boot setup RAM-disk or Network, requires no more than a second kernel on the boot-media ( <= 400K) and a boot-selection that is configurable (syslinux, grub, lilo etc.) on the system. RAMDISK based systems can be a security enhancement, if setup is done carefully. Flash and Hard disk

Embedded systems need not always be specialized hardware — even if many people will not recognize an old i386 in a midi-tower as being an embedded controller — this can be a very attractive solution for small numbers of systems, development platforms and for inexpensive non-mobile devices. The processing power of a 386 at 16 MHz is not very satisfactory for interactive work, but more than enough for a simple control tasks or machine monitoring system. The ability to utilize the vast amount of commodity components for personal computers in embedded systems is not unique to embedded GNU/Linux, but Linux systems definitely have the most complete support for such systems, aside from being simple to install and maintain.

Hard disk based systems

Obviously the last mentioned method is only acceptable for systems that don't have low power requirements and can tolerate rotating devices, that is, are not to operate under too rough conditions. In these cases, the advantage of Linux supporting commodity PC components may be a relevant cost-factor, as especially for prototype devices and those built in very low numbers, these components simplify system integration substantially (no special drivers, no non-standards system setups required). Aside from these specialized systems, hard-disk based systems are also interesting for development platforms, as they eliminate the storage constraints that are imposed on most embedded systems. And, with there ability to use swap-partitions on such a setup offer an almost arbitrary amount of virtual-RAM (although slow) for test and development purposes.

Flash/solid-state disks

Solid state 'disks' have already been available for Linux in the 2.2.X kernel series. Obviously the IDE compatible flash-disks were no problem; other variants like (CFI-compatible,NAND-flash,JEDEC etc.) were more of a problem, but the MTD project now has incorporated these devices into the Linux kernel with the 2.4.X series in production quality. The restrictions for some of these media do stay in place, that is, that they have a limited number read/write cycles available (typically in the range of 1 to 5 million write cycles — depending on the technology used and environment conditions as well as operational parameters). This can be a problem if systems are not correctly designed. A file system and the underlying storage-media tend to erase/write some areas more often than others (e.g. data and log files will be written more often than applications or configuration files, naturally the load can be very high in all temporary storage areas so the storage media may wear out faster depending on the systems layout. wear leveling strategies have been design to reduce this “hot-spot burnout” but this generally means data around to level out the wearing and thus reducing read/write performance of the media.

Imagine a swap-partition on flash or the system logfiles with syslogs parameters not adopted; such a flash device could run into problems within as little as three months! When using a solid-state media with limited read/write cycles, filesystem activity should be reduced, write logfiles at large intervals, write data to disk in large blocks, make sure temporary files are not created and deleted at high frequency by applications. Taking the read/write limit into account, the effective life-span of such a system easily can be extended to years. If high frequency writes are an absolute must, then the usage of RAMDISKs for these purposes is preferable.

Since solid-state based systems generally don't loose their data at reboot, one must also take care of data accumulated in temporary files and especially in logfiles. For this purpose some sort of cron-daemon will be required on such a system, allowing for periodic cleanup. Also, in general, a non-volatile root-filesystem will be 30-40% larger than a volatile RAMDISK based system — if file integrity checks are necessary (as a reboot will not put the system back into a sane state after file corruption or an attack on the system) the filesystem can be double as compared to a RAMDISK based system.

Alternatives to delayed read/writes to devices with limited read-write cycles, are to use filesystems that implement wear leveling like jffs and jffs2 (or use devices that implement wear leveling in hardware like DOC or some PCMCIA cards). Generally this should be taken into account for any devices that don't implement wear leveling on the hardware level (like Compact Flash and Smart Media . . . correct me if I'm wrong on this . . . ). And no — journaling filesystems don't automatically guarantee wear leveling. They will protect the filesystem against power-fail situations which older filesystems like minix or ext2 don't handle very well — especially if the failures occurs during write cycles — but journaling filesystems will also show hot-spots with respect to read/write cycles that can reduce the life span of some devices.

One characteristic of solid-state devices that must be taken into account is that they are relatively slow (although faster devices are popping up lately). This has implications on the overall system performance as well as on the data-security of items written to disk. Solid-state disks will often exhibit a data-loss on the items being processed at the time of power-loss, though this does not though influence the integrity and stability of the filesystem itself. So in a solid-state disk based system, critical data will have to be written to a fast media if it is to be preserved during a power-loss.

The generally low performance of solid state disks with respect to read/write bandwidth can be overcome in some setups by having a “swap-disk” located in RAM. This might seem surprising that reducing system RAM and putting some into a swap-partition can improve performance, but this is the case due to the different strategies that Linux uses to optimize memory usage — swapping to a slow media would hurt performance greatly — swapping to a fast media will improve swap performance and at the same time the Linux kernel will modify its optimization strategy to use the reduced RAM as good as possible. The implementation of such RAM-swap-DISKS can be done with current MTD drivers using slram on top of mtdblock. Slram provide access to the memory area reserved (by passing a mem= argument to the kernel, limiting the kernels memory to less than physically available); mtdblock provide the block devices interface so that this memory area can then be formated as a swap partition on system boot.

— Continued —

---

Story navigation . . .

• Part 1: The main challenges in High-end Embedded OSes
  
• Part 2: Resource Allocation
  
• Part 3: Operational Concepts
  
• Part 4: Compatibility and Standards Issues

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.