Mr. Dion gives a step-by-step on putting this tiny ticker to use.

Last month in the first of four articles illustrating the use of the MachZ chip, Doug Stead introduced the MZ104 and how it can be used in a number of embedded systems designs. In this article, we will show how to embed Linux on an MZ104 and the steps involved in developing a Linux-based embedded application.

In conjunction with these articles, Embedded Linux Journal is running a design contest based on the MachZ chip, the PC/104 MZ104 board, the M-systems DiskOnChip and the BlueCat Linux operating system. One hundred complete kits of these products will be awarded to the ideas judged best. Out of these winners additional prizes will be given to those who can successfully develop a working proof of concept for each category entered. The free kits will include:

• BlueCat Linux Development Kit
• Tri-M Systems MZ104 PC/104 Board
• On Board ZF Linux Devices' MachZ Chip and Phoenix BIOS in Flash
• M-Systems 4MB DiskOnChip Millennium (Flash Disk)
• MZ104 Cable Kit for connecting to peripherals
• 32MB SO DIMM (DRAM)

Figure 1. The MachZ Chip

As you may recall from last month's article, ZF Linux Devices Inc. released their new single device PC-on-a-Chip, the MachZ, in June 2000. This device heralds a new era in ultralow power consumption utilizing standard x86 technology. Specifically designed for embedded applications, the MachZ comes bundled with a Phoenix BIOS.

Figure 2. Annotated MZ104 Board

The MachZ chip has taken the embedded x86 hardware to a new level by combining the full x86 PC functionality together with ZF's patented FailSafe technology into a single chip, the MachZ “PC-on-a-Chip”. This device, just 1.4″ square, represents the next level of integration of PC functionality. MachZ is the only chip with a built-in FailSafe mechanism for restoring operation after a system failure.

From the viewpoint of porting Linux and developing embedded Linux applications, the MachZ and MZ104 provide a very comfortable environment. In essence, a standard off-the-shelf embedded Linux for the PC platform should run on the MachZ with minimum effort. Of course, actually getting an embedded Linux to run on the MachZ or MZ104 is a little more involved than hitting the power switch.

There are four major steps involved in embedding Linux and developing a Linux-based application. Assuming that you've selected the MachZ and the MZ104, and that you've done a top-level system design, here's what you need to do:

1. Set up and test your development environment. This includes both a development host for cross-development as well as a suitable embedded development platform incorporating the MZ104.
2. Develop and debug any required Linux kernel features. This includes mostly new drivers or other board-specific changes (for caching or local set up and initialization). For the MZ104, most of the default settings will be fine.
3. Develop and debug Linux application software. This step is the one that many embedded systems designers think most about. It's where your embedded system really differentiates itself from other embedded systems.
4. Package and test the final application in a production configuration.

The rest of this article will go through each of these steps and describe what needs to be done. We will use LynuxWorks' BlueCat Linux to illustrate each step. Figure 3 illustrates an overview of the total development process using BlueCat Linux.

BlueCat Linux comes with a number of handy tools and capabilities that are only found in BlueCat Linux and not in “standard” Linux distributions. The most significant include:

• Sample or demo systems. These are pre-configured and prebuilt Linux kernels for embedded systems. There are several sample systems that illustrate the use of selected features and capabilities of Linux.
• mkrootfs. This is a tool designed to automate the creation of root file systems for Linux.
• mkboot. This is a tool designed to automate the creation of bootable kernel images suitable for loading onto floppy or hard disks, Flash memory, ROM, etc.
• BLOSH. This is the BlueCat Linux OS Loader. It's a tool that runs in a special sample system and supports booting BlueCat Linux kernels over an Ethernet (very useful during development).
• Cross GNU tools. Special versions of the GNU compilers and debuggers are provided to support cross-development (developing on a host system to download and debug on a target system).

Figure 3. Embedded Linux Development Environment

Embedded Linux Development Environment

Setting up an embedded Linux development environment involves two major parts. The first is the development host. The BlueCat Linux distribution from LynuxWorks supports workstation PCs running either Linux or Microsoft Windows as development hosts. It's best to equip these machines with high-end, powerful processors, lots of memory, and most importantly, LAN and serial I/O support. Thankfully, all high-end PCs these days come with LAN and available serial I/O ports.

Installing the BlueCat Linux development tools on your host system is simple. The BlueCat Linux development tools (described in more detail later in this article) simply load into an existing file system. Unlike other Linux distributions, BlueCat Linux does not need to take over your desktop PC system and replace the existing OS running there.

Figure 4. An embedded Linux development environment involves both host and target.

Once installed, BlueCat Linux provides a set of tools that may be used to develop embedded Linux applications. The most important are specially modified versions of the very popular GNU compiler and debugger. These versions of gcc and gdb are designed to support cross development from your desktop PC to the MZ104 and other embedded systems. In addition, BlueCat Linux provides tools for embedding the Linux kernel and other software into Flash or ROM.

The second part of setting up your embedded Linux development environment is configuring a target system to be used during your development. Many embedded systems do not have hard disks, floppy disks or Ethernet connections, but these devices substantially simplify the task of developing and debugging embedded systems. Fortunately, this is easy with the MachZ and the MZ104.

The first step is to set up your MZ104 system and attach a keyboard, a standard floppy disk drive, and through the ISA bus, a video card and monitor. Most video cards and monitors available are supported by Linux, so finding compatible hardware should be easy. Once the target hardware is set up, the next step is creating a bootable floppy disk with embedded Linux on it. With BlueCat Linux, this is very easy. BlueCat Linux comes with several prebuilt sample or demo systems that let the user easily configure a BlueCat Linux embedded OS for their hardware. These sample systems include prebuilt, compressed kernel images suitable for installing on a floppy or hard disk.

Figure 5. An MZ104 plugged into a three-slot standard 16-bit ISA bus passive backplane.

To test the MZ104, take the following steps:

1. On the development host, locate the BlueCat Linux sample or demo systems. Find the Hello sample system. This sample system provides a simple Hello World demonstration.
2. Using the BlueCat Linux mkboot command, create a bootable Linux floppy disk with the Hello World sample system.
3. Go to the MZ104, insert the bootable floppy disk and power up. If the hardware is properly configured, the Hello system should boot and begin to print the words Hello World on the monitor.

While this process is pretty simple, it's because a lot of the work is being performed behind the scenes (see Table 1).

Table 1. A Typical BlueCat Linux Sample or Demo Subdirectory

To build each of the configurations provided with BlueCat Linux from scratch would require developing and testing kernel configuration files, as well as mkrootfs .spec files, and then building and testing the resulting kernel configurations.

The BlueCat Linux mkrootfs tool is used to create a root file system for the Linux kernel to use. Every time a Linux kernel boots it looks for a root file system where it will find its initialization information and application software. The root file system can be located on a disk (floppy or hard disk), a network mounted file system (such as NFS), in memory (using a RAM disk) or Flash memory file system. Based on a .spec file, the mkrootfs tool generates a Linux file system including all of the required files needed by the Linux kernel to boot and run. The mkrootfs tool only includes those files that are needed, so the root file system that is created is as small as possible. In those cases where there is no disk, network connection or Flash file system, the root file system can be placed in a compressed form in a ROM and then decompressed into a RAM disk upon system boot.

Without the mkrootfs tool, you would have to construct your desired root file system by hand. This can be easy if you simply include all of the files found on a desktop Linux distribution. Of course, you'd only do this if you don't mind the root file system containing hundreds of megabytes of programs and data. In most embedded systems you will want to minimize the size of the root file system (especially if it's going to be placed in flash memory or on a RAM disk). The mkrootfs tool simply automates this process for you.

The BlueCat Linux mkboot tool similarly automates the process of creating a bootable image for hard or floppy disk. While a bootable disk can be built by hand, having a tool to automate the process makes it simpler and more efficient.

So now that Hello is working, you know that the basic hardware and BIOS are operational. The next step is to set up a development and debugging connection to the host development system. First, through the ISA bus on the MZ104, insert an off-the-shelf Ethernet card. Again, most commercially available Ethernet NICs are supported by Linux, so finding one for your development should be relatively easy. You should check to see which Ethernet NICs are configured with the standard sample systems distributed with BlueCat Linux. If you use an Ethernet NIC that is not in the standard sample systems, then you will need to reconfigure the sample systems to add the Ethernet NIC you're using.

Figure 6. PC/104 stack plugged into a three-slot 16-bit ISA bus passive backplane. An ISA Ethernet card is plugged into the backplane, and a PC/104 video card is connected directly to the MZ104. The keyborad and mouse are connected using the contest cable kit, which allows you to connect an IDE hard drive, CD ROM Drive, and serial and parallel devices to the MZ104.

Next, go back on your development host look for the BlueCat Linux sample system called Ping. This is a Linux sample system that boots up on your target system and lets you ping your development host over the Ethernet. To run this sample system you will need to know the IP address to your host system and to assign an IP address for your target system. Using mkboot, place the Ping sample system on your floppy disk and boot it on your target system. Once the target system is booted, it will run a Linux shell and allow you to type in the networking configuration information and ping your development host. If your hardware is set up properly, this should work without any changes to the Ping sample system.

Of course, even if the BlueCat Linux sample system wasn't available, you could build your own Ping sample system by carefully analyzing what minimal set of utilities and files are needed to enable an embedded system to ping another system on a network.

Once you're able to demonstrate that the Ethernet connection is working, the next step depends a great deal upon your embedded system design. For the sake of this article, let's assume that your final application will use the MZ104 without any IDE disks (hard disk or floppy disk) and without an Ethernet connection. So, your final application will use the DiskOnChip for file system storage purposes.

For this system design, the main development problems that need to be solved are:

1. How to download and debug kernel and application software
2. How to copy files to the DiskOnChip file system

The simplest way to download and debug kernel and application software is to use the BlueCat Loader Shell (BLOSH). This program is found under the BlueCat Linux Osloader sample system. The BLOSH sample system boots from floppy disk, hard disk or Flash and allows you to download and boot kernels with optional root file systems. To use BLOSH, you must set up your host system to support the Trivial File Transport Protocol (TFTP). Configuring a TFTP server on your host system is outside the scope of this article, but doing this is straightforward on most desktop Linux systems.

Using the mkboot utility, create a bootable floppy with the Osloader sample system that includes BLOSH. Now whenever the target system is booted, it will automatically execute BLOSH to allow you to download and boot newly built test kernels (and root file systems created by mkrootfs) from your development host.

BlueCat Linux supports the DiskOnChip and provides detailed instructions on how to configure and build kernels that support it. Once you have built a kernel including the DiskOnChip driver, this kernel can be downloaded to the target system using BLOSH and then used to initialize and load the DiskOnChip file system.

Similarly, BlueCat Linux supports several Flash memory devices. Simply configure a system with the appropriate Flash memory driver, download it using BLOSH and then initialize your Flash memory.

At this point, your development system is installed and operational. You're now ready to start enhancing or tuning the embedded Linux kernel and to develop your application software.

Configuring and Debugging Embedded Linux Kernels

While BlueCat Linux comes with preconfigured and prebuilt sample Linux kernels, there are several reasons why you might want to reconfigure the kernel for your own use. For example, you may wish to

• Customize for functionality: modify a kernel option to add or remove a kernel feature (for instance, networking support) or modify the default feature behavior.
• Customize for hardware devices: modify a kernel option to add or remove support for a particular device.
• Customize for size: remove kernel features not required in the embedded system to reduce the kernel size.
• Customize for performance: modify a kernel option to improve the runtime performance of the kernel.

There are two major steps in reconfiguring a Linux kernel. The first step is to update the Linux .config file. The second step is to rebuild the kernel using the new configuration.

BlueCat Linux can be configured in the same way that the “standard” Linux kernel is configured. The configuration procedure is composed of the following steps:

• On the development host, locate the directory with the latest Linux .config file. For sample systems, the .config file is located in the same directory as the prebuilt kernels.
• Run the command make config. In BlueCat Linux, there is also a make xconfig that provides an X Windows-based configuration tool. This program leads you through a series of kernel configuration options and allows you to change settings.
• Upon completion of the configuration scripts, the updated kernel settings are placed in the .config file and a number of kernel header files are updated to reflect the changes.

The next step is to rebuild the Linux kernel based on the configuration just created. To rebuild, simply go to the kernel build directory and execute make bzImage. This makefile automatically builds two kernel configurations:

• bzImage: this file is used to boot from floppy or hard disk. The file contains a compressed image that automatically decompresses after being booted from the disk.
• bvmlinux: this file is used to boot using the BlueCat OS Loader. This file contains a compressed kernel without a boot sector. The bvmlinux file can also be used to create an bootable kernel image programmable into ROM or Flash memory.

Because the MZ104 is basically a PC-like environment, the main reasons that you would reconfigure and rebuild your Linux kernel is to add drivers for custom hardware or to remove drivers for devices that you're not planning to use in your application (to save memory space). In the latter case, the amount of memory that's saved would be very small and may not justify the loss of flexibility in the target system (you never know when it will come in handy to have the printer port driver installed).

Debugging the Linux kernel is a little more complex. BlueCat Linux provides the capability of using the GNU debugger gdb to debug the Linux kernel over a serial port. To set this up, you will need to configure the BlueCat Linux (as discussed above) to enable the BlueCat Linux kernel debugger. The kernel debugger is a small debug monitor that is loaded with the BlueCat Linux kernel. You will also need to connect a null modem serial cable between the debug serial port on the MZ104 and your host workstation.

The BlueCat Linux kernel debugger uses the second serial port by default. This is COM2, or /dev/ttyS1 in Linux terminology. You should use COM2 for kernel debugging. It's possible to reconfigure which port is used for kernel debugging, but for the MZ104, using COM2 is the correct choice because the COM1 (/dev/ttyS0) will be used for the default console device.

To take full advantage of the debugging capabilities of gdb, you will want to rebuild selected parts of the Linux kernel using the GNU compiler's -g option (to enable debugging information). When a module is compiled with the -g option, full source level debugging of that module is possible. Otherwise, all debugging will be done at the assembly language level.

The BlueCat Linux kernel debugger works as a server and performs the following basic operations to requests made by gdb:

• Memory read
• Memory write
• Register examination
• Execution resumption
• Single-stepping

Breakpoints are set and cleared using the memory read and memory write operations. Once enabled and operating, it's possible to use gdb to execute the kernel, set breakpoints, examine registers and memory, single-step the kernel one instruction at a time, or a line of code at a time, print stack tracebacks, etc. In addition to normal gdb functions, the BlueCat Linux kernel debugger allows you to display the contents of certain important kernel data structures. Table 2 describes the extended gdb functions supported by the BlueCat Linux kernel debugger.

Table 2. gdb Functions Supported by the BlueCat Linux Kernel Debugger

Embedded Linux Application Development

Developing applications for embedded Linux is very similar to developing applications for desktop Linux systems, and this process is very different from developing applications for traditional embedded real-time operating systems. Linux, like most POSIX-compatible operating systems, provides processes and threads that are protected from each other and from the underlying operating system kernel. In Linux, under normal circumstances an application cannot crash the kernel or cause other applications to crash. Many traditional embedded operating systems require the applications and the kernel to reside in the same memory space. A bug in one application can easily cause the kernel or other applications to crash. Even worse, recovery is almost always impossible, forcing at least a system reset and reboot. For this reason, and others, using a POSIX operating system like Linux is very smart for embedded system designers.

Assuming that your Linux kernel is up and running on your MZ104, developing and debugging your embedded application software is very easy. In fact, since the embedded Linux kernel is compatible with desktop Linux systems, you can actually start your application development on your Linux-based desktop system. Or, you can develop your embedded application software on the MZ104. In either case, you don't need complicated, hard-to-use development tools to get your work done. Many traditional embedded real-time operating systems require you to use such complicated tools to develop your applications because they don't support the capabilities and power of Linux.

Whether you're developing and debugging on your desktop or MZ104, the overall process is the same. In both cases, you will use the popular GNU compilers and debuggers. Simply create your application code, compile with the GNU compilers and invoke the GNU debugger gdb to debug the code. To set up and debug using your MZ104, you need only take the following steps:

• Set up the embedded Linux development environment described in the beginning of the article, including the Ethernet connection.
• Configure your Linux kernel using the available tools described in the section titled “Embedded Linux Development Environment”, including the ability to log in to the MZ104 target system. In BlueCat Linux, there are several sample or demo systems that are prebuilt to allow you to do this.
• Configure your root file system as described in the section titled “Embedded Linux Development Environment” to include the program you're debugging and the gdbserver program. The gdbserver program is executed on the MZ104 to support gdb, which is running on the host system.
• Boot your MZ104 with the Ethernet-enabled kernel and with gdbserver available. On the MZ104 system, start the gdbserver. This can be done by using Telnet or other method to log in to the MZ104 system and execute commands.
• Start gdb and configure it to execute and debug an application on the target system.

At this point, gdb is capable of debugging the application on the MZ104 in exactly the same way it is capable of debugging applications on the host system. The same commands and capabilities exist in both environments.

Since the debugging environment and debugging tools are easy to set up and use, it's possible to focus your attention on your embedded application software. Usually this is the part of your embedded system that differentiates your product from other competing products. One of the main advantages of using Linux in your MZ104 system is that there are so many software packages, both open-source and royalty-bearing, that are available for Linux. Since the MachZ is essentially a PC-compatible device, many of these packages are available without any porting or tuning required. For time to market reasons, it's almost always better to leverage available software than trying to build your own.

Figure 7. Three PC/104 boards, an Ethernet card, a video card and the MZ104 with the cable set, included as part of the prize hit. The LAN cable connected to the PC/104 Ethernet board is also included.

Packaging and Testing an Embedded Linux-Based MZ104

If you've made it this far in your development, you've successfully built an embedded Linux development platform using your MZ104, an ISA bus, a keyboard, a video card and monitor, an Ethernet card and cable and a serial cable. You've also configured and built a Linux kernel for the MZ104. You've run some tests and used the BlueCat Linux kernel debugger to troubleshoot any problems. Lastly, you've developed your application software and tested and debugged it either on the host system or the MZ104, or both.

Now you're ready to configure and test your final application. During the development phase you would have created a file system on the DiskOnChip. Configuring BlueCat Linux to boot from the DiskOnChip is straightforward and documented in the BlueCat Linux User's Guide. Remember, the DiskOnChip presents itself to the system just like a hard disk.

Technically it's possible to place a full file system into the MZ104 Flash memory. In fact, BlueCat Linux comes with a journaling Flash file system that provides full-featured Flash support including wear leveling (so the Flash doesn't wear out unnecessarily) and fault recovery. However, the DiskOnChip hides all of this functionality from the Linux kernel and provides an efficient, clean interface to the file system with a minimum of system overhead.

The first step to packaging your final product is to create a final version of your application software and a near-final kernel. The near-final kernel should be bootable from the DiskOnChip. But, it should also include all of the debugging device drivers and software (Ethernet drvier and kernel debugger) just in case. The application software and required system software should be assembled into a default root file system for the DiskOnChip. You should use the mkrootfs program to create this root file system and use the BlueCat Linux sample systems, configured to use the DiskOnChip to download this root file system and load it into the DiskOnChip. Lastly, the near-final kernel should be loaded into the DiskOnChip file system and the system rebooted.

At this point it should be possible to run any final product tests against your MZ104-based system. This may include external text harnesses or simply using the product as it is intended to be used. It's generally a good idea to send the output of the tests over the Ethernet or to enable system logging to send status and information over the Ethernet, to your development host. For example, it's easy to configure the Linux syslog capability to post all of the system logs to a remote system over the Ethernet.

Once you're convinced that your embedded system is operating correctly you can systematically remove all of the unneeded devices and device drivers. Initially, keep the hardware intact while only removing software support.

The first step is to reconfigure BlueCat Linux to remove the networking subsystem. Keep the keyboard, monitor and floppy disk enabled for now. Reconfigure the system and download new DiskOnChip file system images. Before doing this, however, be sure to build a recovery floppy disk to be able to reload the Ethernet enabled version in case of failure. You can log in to the MZ104 via the keyboard and monitor to view the system log and insure that the system is operating normally.

Once you're sure that the system is operating properly without the networking software, reconfigure the kernel to remove the keyboard and monitor support (so-called “headless” operation). Leave the kernel debugger and floppy disk enabled. At this time, you will also want to reconfigure the console device to be COM1 instead of the keyboard and monitor. Reload the system (using a kernel booted from the floppy disk) and reboot the system. Use the kernel debugger over COM2 to insure that the system is operating properly. You can also use a desktop PC and connect it to the MZ104's console port (COM1). By opening a terminal emulator on your desktop system, you can log in to the MZ104 over the console port and check system status.

At this point, you can dismantle the hardware system and remove the ISA bus (along with the Ethernet NIC and video card), and disconnect the floppy disk and keyboard. The only remaining connection to the MZ104 is the serial cable for the kernel debugger and any other I/O devices you're using in your application. You may also want to keep the console port connected to a terminal emulator on your desktop system. Reboot the system and use the kernel debugger to insure that the system is operating properly.

The last step is to disconnect the kernel debugger serial connection and disable the kernel debugger. However, many system designers like to leave the kernel debugger enabled and reserve a serial port (such as COM2) for system testing and diagnostics in the field. One caution here: if you leave the kernel debugger enabled in your final system, there is potentially a security hole in your system if someone can access COM2 directly. After the kernel debugger connection is removed, test the system externally in a so-called black box mode. If everything is still working as planned, you're done!

Well, not exactly. There's still the problem of how to get the final DiskOnChip image into a production MZ104. There are several methods of solving this problem. In some cases, when the DiskOnChip is socketed, it can be programmed off-line and inserted into production units prior to final test.

Another approach is to program the DiskOnChip after the final production units are assembled. A special BUR (boot up ROM) included inside the MachZ chip allows programming of the Flash and the DiskOnChip even after the parts are soldered into the final board. The BUR will read from a high speed serial port on the MachZ (called the Z-tag port). Generally you would download a FlashProgrammer BUR Extension into the MachZ on-chip RAM through the Z-tag port and under control of the BUR. This FlashProgrammer (or BUR Extension of your choice) will then read the flash image at up to 1.2 megabits/second through the Z-tag port, and while reading the data, transfer it into the Flash or the DiskOnChip.

Conclusion

As you have seen, developing, debugging and packaging embedded BlueCat Linux applications on the MachZ and MZ104 is fairly straightforward. But, for truth in advertising reasons, it's important to note that lots of details have been skipped over in this article. These details are available in the BlueCat Linux documentation or other Linux documentation. Before attempting to embed Linux, it's very important to learn as much as you can about Linux and to get help from experts. Most, if not all, of the embedded Linux distributors (including LynuxWorks) will provide expert assistance through their professional services teams. If time to market is a concern, then taking advantage of this expert assistance is a must.

In addition, there are several vendors, including LynuxWorks, who offer advanced development tools to help you get your development done more quicklyand minimize time to market. For example, LynuxWorks offers the VisualLynux integrated development environment for developing Linux applications from Microsoft Windows.

In the third article in this series, we will further explore the technical aspects of installing BlueCat Linux on the MZ104 board. A CD will be prepared with all configuration options (booting from Flash, booting from DiskOnChip, booting from floppy and booting from hard disk). The article will describe in detail the specific configuration files prepared for the MZ104, will review memory maps and resource utilization for various Linux Configurations supported by the MZ104 and will discuss the technology of the BUR and Z-tag features built into the MachZ chip.

In a fourth article, we will look at high reliability embedded solutions highlighting FailSafe and the Lynx OS, a real-time adaption of Linux available from LynuxWorks. FailSafe is a patented concept included in every MachZ chip that allows the solution provider to recover BIOS, O/S and application software and re-establish full operation after an unimagined catastrophic failure occurs. Lynx OS is a high-reliability, multi-tasking embedded Linux.

---

About the author: Luke C. Dion ([email protected]) is Vice President of Technology and Strategic Planning at LynuxWorks. Luke has over 20 years experience in embedded systems and UNIX-like operating systems. In addition to LynuxWorks, Luke has worked with embedded software at Itron, Microtec Research, and Motorola Computer Group.

---

Copyright © 2001 Specialized Systems Consultants, Inc. All rights reserved. Embedded Linux Journal Online is a cooperative project of Embedded Linux Journal and LinuxDevices.com.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.