Book review: Craig Hollabaugh’s “Embedded Linux”
Jun 1, 2002 — by LinuxDevices Staff — from the LinuxDevices Archive — viewsFirst impressions
A system developer planning to use Linux for an embedded design is faced with a number of decisions, not the least of which is whether to use a packaged commercial Embedded Linux distribution or to devise a homebrew solution from the available free tools and components. The custom approach has much appeal because of its low cost and radical flexibility, allowing one to choose any approach or tool rather than those chosen by the toolkit vendor. But with this flexibility and low cost comes the chaotic documentation typical of Linux. Thus, books like Embedded Linux: Hardware, Software, and Interfacing, by Craig Hollabaugh, fill a significant void.
But given the book's ambitious scope, Hollabaugh has done an admirable job covering the material in a clear and focused way. Experienced Linux users should have few problems constructing embedded Linux systems using the explanations and tools provided. Even Linux novices will find the book useful, though beginners should supplement Hollabaugh's Embedded Linux with one of the many introductory books on Linux.
Hollabaugh and Addison-Wesley are to be commended for the generally high quality of the book, especially in light of the somewhat poor writing, editing, and layout that are typical of many technical books. I found very few typographical errors and noticed no grammatical errors. The book is beautifully laid out and clearly organized. This attention to detail is refreshing at a time when many publishers seem to have a “you know what I mean” standard for writing and editing.
Blazing trails with Embedded Linux
Hollabaugh's Embedded Linux is structured as an extended development and system integration project for the fictional Silverjack ski resort. “Project Trailblazer” calls for monitoring air temperature and providing live images at each of the ski runs, recording lift traffic and authenticating users before allowing access to the lifts, controlling snow making equipment, timing races, and playing music. The book details how each of these requirements is to be met.
Some may find this extended conceit somewhat distracting, and occasionally maintaining the Trailblazer fiction while focusing on the book's embedded Linux subject leads to some improbable results. But the consistent emphasis on meeting real needs (even if the needs are fictional) keeps the book — and the reader — focused.
Chapter 1 is a short introduction to Linux, open source development, and embedded systems in general — subjects quite familiar to LinuxDevices.com readers. Chapter 2 introduces Project Trailblazer, describing the resort with its physical and network layout, and lists the project's requirements. This lays the groundwork for meeting the requirements in the rest of the book.
In 44 pages, Chapter 3 covers the crucial issues of platform selection and setup of the development environment, thus showing how ambitious this book is. I don't think it is quite possible to cover these issues adequately in 44 pages, but Hollabaugh has done remarkable things with the space allotted. For platforms, he quickly chooses an x86 development workstation and SBCs with x86, PowerPC, and StrongARM processors. Here, it becomes immediately apparent how the Trailblazer conceit can be distracting: for the purposes of the book multiple architectures had to be chosen, but a real project is unlikely to choose SBCs with three different processors.
In any case, Hollabaugh quickly goes into cross-development platform selection and setup. With no debate or discussion, he sends the reader off to download the Debian distribution and install it. X will not be installed — no time for that; only 44 pages to cover everything! The entire operating system is installed in six and one half pages in ten steps (counting step 10 — “Have Fun!”). I hope there is nothing unusual about your hard drive or your network card, and that you're on a 192.168.1 Class C subnet without DHCP, and that you're comfortable working without X, because we've only got 31 pages to go and we still have to cover building the toolchains.
Hollabaugh's coverage of building and configuring the toolchains in these 31 pages is truly remarkable in its usefulness and clarity. He has written a series of scripts to automate the process, and reproduces them in full in the text along with short clarifications in between. The scrips are written in refreshingly simple and clear bash, free of esoteric and obscure script functionality; even an occasional or novice script writer could read this section and emerge with a good working understanding of the process. The PowerPC and ARM toolchains are built with the same set of scripts, with architecture-specific configuration localized to one file, so they could be customized for other architectures quite easily.
The book is worth having for these 31 pages alone. I have not seen another how-to guide for toolchain construction that is quite as focused on the goal as this one. If you just want the steps to achieve the goal of a usable toolchain, this is the document for you. You might also see the GNU Toolchain for ARM Targets HOWTO for another explanation.
Chapter 4, which covers booting Linux on the target boards, is similarly dense. For building the root filesystem, Hollabaugh recommends using MontaVista's Hard Hat Linux Journeyman Edition. This provides prebuilt cross-compiled versions of many needed packages, freeing you from the necessity of finding, downloading, and building open source packages from a variety of sources. As with the toolchains, he provides a clear, well-written script to download the appropriate packages from MontaVista and build the root filesystem.
Sadly, the Journeyman edition is no longer available from MontaVista. Hollabaugh still has the last version (binaries only) of it on his web site, and his buildrootfilesystem script gets it from there, rather than from MontaVista. This is not a long-term solution, as the packages are quickly becoming obsolete. Ultimately it may be that the wisest approach is still to get the packages from their original sources and build them rather than relying on any one repository of cross-compiled packages. Such is the life of the embedded software designer seeking “free beer”.
The general approach to booting is to build the kernel and root filesystem on the cross-development host and boot the targets over the network using TFTP, meanwhile using minicom to issue commands to the target's bootloader over a serial port.
As with Chapter 3, this chapter provides excellent information on building an embedded Linux kernel and root filesystem, and booting these on the targets. This is great if Hollabaugh's approach meets your needs, or if you are an experienced Linux user with the background enabling you to substitute alternative approaches where needed. But a beginner with unusual needs would not find sufficient help here. This may be inevitable given the space constraints Hollabaugh imposed on himself. But even if alternatives could not be described in detail, he could have provided more extensive lists of references. A link to the excellent Linux From Scratch site, for example, would be a help for those needing more background.
A more specific criticism of this chapter concerns Hollabaugh's neglect of BusyBox, by now virtually a standard tool for building compact embedded Linux systems. BusyBox provides a variety of simple Unix-like utilities in one executable, saving considerable space. Not only does Hollabaugh not mention BusyBox as an alternative to installing numerous GNU utilities, in his only mention of it he seems to mistake it for C library alternatives to glibc, like uClibc and newlib. This omission is a serious one — by now anyone building a Linux-based device must at least consider BusyBox as one approach.
To wrap up the book's coverage of setting up the operating system and development environment, Chapter 5 covers debugging. As one would expect, this discussion emphasizes using GDB in its client-server mode using the gdb and gdbserver programs. The discussion is brief, simply running through the debugging of a short “hello, world” program. No coverage of X-based wrappers to GDB such as DDD is provided — not surprising since the cross-development host for Trailblazer does not run X. The brevity of this chapter makes sense; cross-debugging using GDB is covered well in the available GNU documentation.
Having covered the basics of embedded Linux development in a mere 138 pages, in Part 2 Hollabaugh covers hardware interfacing, timing, and system integration. In the hardware interfacing chapters he explores asynchronous serial communication, parallel ports, USB, memory-mapped hardware, and board-level synchronous serial communication. These are some of the strongest chapters of the book, reflecting the author's apparent expertise in the area.
In the Chapter 6, on asynchronous serial interfacing, Hollabaugh provides some excellent C language utilities which are good for manipulating serial ports from shell scripts. Especially useful are setSerialSignal, which sets the hardware handshake signals DTR and RTS; and getSerialSignal, which read DSR, CTS, DCD, and RI. Both utilities use the standard ioctl() call for port manipulation, a sensible choice as this interface provides the functionality most applications need. These utilities let you use the handshake signals as digital I/O points, making the most of your built-in hardware.
Chapter 7, on parallel port programming is somewhat more complex, offering more approaches for port control than were discussed for serial ports. Three approaches are taken: ioperm(), which allows for direct port manipulation from user space; use of ppdev, the standard Linux parallel port driver; and writing a custom parallel port driver. All three are covered in the author's typically clear and straightforward manner, with helpful examples provided. The reader should be equipped to use any of the three approaches after reading this chapter.
The chapter is marred slightly by an unsubstantiated and unfair offhand remark. In the context of describing the fictional Trailblazer project's progress, Hollabaugh writes that the engineers started by reading the classic Linux Device Drivers by Rubini and Corbet. He writes, “Unfortunately, after reading this book, the engineers still didn't know where to start”, then immediately suggests other sources for information on device driver development. The implication is that the Rubini and Corbet book contains gaps in its coverage of the subject. Many readers will find this implication surprising; the book is considered almost a must for learning how to write device drivers. In any case, it would have been fair for the author either to have provided a complete review of the book, describing its alleged deficiencies, or to leave the matter alone.
In Chapter 8 Hollabaugh leads us through the still rather arcane art of USB interfacing with his usual clarity. This chapter (and indeed, all of Part 2) demonstrates one reason why Linux has become so useful in embedded systems: it puts warehouses full of cheap and useful PC peripherals within the reach of embedded system designers. The author demonstrates how speakers, PC cameras, and SmartMedia card readers can be effectively and cheaply interfaced for use by embedded systems. He uses Yamaha speakers, a Kensington camera, and a SanDisk card reader and integrates them nicely into Project Trailblazer; all of these consumer devices are dirt cheap and easily obtained, in contrast to many peripherals more traditionally associated with embedded systems.
Recognizing that USB device driver development is still quite difficult, Hollabaugh does not provide any examples. Instead he focuses on using already available drivers for the peripherals, an art that itself involves several steps. Using the Kensington camera, for example, requires first plugging it in, viewing the system log file to get the “vendor” and “product” numbers, looking these up on the Linux-USB device overview web site to determine which driver to use, configuring the kernel to use this driver, building the kernel, booting it, mounting the usbdevfs, running mknod to create the device, downloading and building the appropriate application software — and, finally, running it.
This process is, of course, not unique to embedded systems, and Hollabaugh leads us nicely through it. The only complaint I have is that he treats each of the three devices as separate, walking the user through kernel configuration as if the device currently being interfaced is the only one in the system. An experienced Linux user will know that if you want all three devices running you'll have to build the kernel once to support all three, otherwise only the last one built in will be present. But a novice is sure to find this out the hard way.
Chapters 9 and 10 cover memory-mapped I/O design and synchronous serial I/O design respectively. Depending on your perspective, these chapters might either be a refreshing break from a world in which software is becoming too important in embedded systems, or hardware-oriented irrelevance that doesn't belong here. The real work in these chapters is in the hardware designs, with simple device drivers mostly on ground covered previously in the book. My opinion is that it is indeed refreshing to find a software-oriented book that doesn't assume the reader is afraid to look at a schematic, but that the pages occupied by this material might have been better invested in more detailed descriptions of the embedded Linux development environment.
Chapter 11, “Using Interrupts For Timing”, is the weakest, technically, in the book. The Project Trailblazer premise for the chapter is that a race timer is needed for one of the ski runs. Thus inputs for the start gate and finish line are brought into the controller, and the software is supposed to time the interval between their occurrences. To determine how much latency can be expected, Hollabaugh wrote interrupt latency tests for the three SBCs. These tests repeatedly record the time, generate an interrupt, and when the interrupt is serviced again record the time. The tests report the average latency.
Not surprisingly, these tests show results in the low tens of microseconds. Then Hollabaugh uses these results to conclude that the stock Linux kernel is adequate for timing needs. But surely the most important metric is the maximum latency, not the average. Silverjack racers probably don't want to cross the finish line, nor have one of their competitors cross the start gate, when Linux is beginning one of its hundred-millisecond non-preemptible periods. Knowing that these are rare and that on average stock Linux timing is accurate is probably not reassuring.
The implementation of the average timing tests also could be improved. On all platforms, a digital output signal is routed to an interrupt-generating input. In the device driver, the time is recorded and the output is asserted in a function called proc_read_interrupt_latency() when a /proc entry is read. The interrupt service routine records the time again and deasserts the signal. proc_read_interrupt_latency() then calculates the time difference and records it. But no synchronization is performed between these two routines. proc_read_interrupt_latency() asserts the signal and immediately starts to record the time difference, assuming that the interrupt will occur immediately and be processed before any more of its own code is executed. Thus the assumption is that the interrupt will have an insignificant propagation delay. This is probably valid for the simple lab setup described, but it's still rather bad form to make this assumption, especially in a book intended as a learning tool.
Finally, the sample code is structured as three separate programs, one for each SBC, although the design of each is identical, with the only differences being the means to control the interrupt and output signal. Far better, would have been to use one program and isolate these differences with conditional compilation.
Chapter 12, “System Integration”, pulls everything together into a usable Project Trailblazer. This chapter again demonstrates why Linux is such a useful tool for embedded systems, as it uses bash, MySQL, and Apache to provide the high-level functionality that embedded systems increasingly need. Thus the SBCs are queried for information like ski run temperature, lift accesses, and images; and this information is made available via a web interface and/or saved in a database for later analysis. All of this is accomplished in a remarkably clean and efficient way with freely available tools. Hollabaugh makes it clear that a good knowledge of open source tools can allow you to implement powerful systems with minimal effort.
Finally, Chapter 13 provides a wrapup, touching upon some issues not fully covered in the book. The chapter actually serves as a good list of important embedded Linux issues not covered: GUI development, using tools like Microwindows and Qt/Embedded; real-time issues, which were only touched upon in Chapter 11; and the commercial embedded Linux toolkits. A complete understanding of Linux for embedded systems requires familiarity with these issues, and it is to the author's credit that he lists these as important but beyond the scope of his book.
The only disagreement I have with this wrapup chapter is Hollabaugh's philosophy regarding non-x86 architectures. He contends, “This book's examples show that working with the [ARM-based] MediaEngine and the [PowerPC-based] RPX-CLLF was as easy as working with the [x86-based] MZ104.” This assertion is hard to credit — even with the author's well-written scripts, acquiring and building cross-development GNU toolchains certainly takes more time than simply using those already provided on your desktop computer. And examples like that from page 372, in which Hollabaugh had to modify the netcat source in order to compile it for ARM, show that even the author himself has had his share of trouble working with non-x86 architectures. The lesson is certainly not that x86 should be the only architecture considered, but it is fair to say that a slight bias toward x86-based devices is prudent when choosing embedded Linux target platforms because of the maturity of Linux on that architecture.
Conclusion
Don't let my quibbling with details in Embedded Linux mislead you — this is an extraordinarily worthwhile book that anyone working with Linux in embedded systems should have. My comments merely reflect my opinion that a book this important should have everything right.
Embedded Linux is published by Addison Wesley Professional (ISBN: 0672322269) and is available at Amazon.com, Barnes&Noble.com, and fatbrain.com. 43 sample pages of the book are available for reading online at Amazon.com, and an introduction to the book, by the author, is here.
About the author: Jerry Epplin has written embedded software for the past fifteen years, primarily for medical devices. He can be reached at [email protected].
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.