Introduction

This whitepaper provides a discussion of theory and methods for building uClinux systems that boot, run, and operate using Flash memory. The paper begins with a brief introduction to Flash memory technology, followed by a discussion of the different ways uClinux system components can be laid out in Flash. After that, there is a description of the process of kernel driver configuration and setup, as well as information regarding root filesystem choices. The paper wraps up with a real life example that should crystallize some of the finer details.

Hopefully by the end you'll have an understanding of how to build uClinux systems that take advantage of Flash Memory, and will be aware of some of the choices you can make, and the implications and trade-offs involved. Although aimed at using Flash memory, most of the layout ideas apply equally to Flash and ROM (you just won't have the ability to update it in circuit, or to run a read/write filesystem with standard ROM devices).

Flash memory

For most of the previous 20 years, ROM (and EPROM) have been the mainstay of non-volatile storage for embedded systems. But today the majority of modern deeply embedded systems rely on Flash memory.

Flash memory primarily comes in two flavors, NOR and NAND (there are some other more exotic varieties, AND for example, that I won't discuss here).

Reading NOR flash is essentially like reading SRAM. You can read values from random addresses, with the whole of its address space visible. You can execute code directly from NOR Flash, since it looks like SRAM (this is often referred to as Execute-In-Place or XIP). Because of this feature I think it is fair to say that NOR Flash has been a more popular choice in small embedded systems. All your code can potentially run directly from the Flash, thus reducing your RAM requirements. In this respect NOR Flash can be used as a replacement for standard ROM. There are quite a few manufacturers of NOR Flash devices, including Intel, AMD, Fujitsu and Toshiba. NOR Flash memory chip capacities typically range from a few kilobytes up to 64MB.

NAND Flash is generally read a block at a time, and so is not like a typical random access memory technology. It is more like a hard drive in nature with block sizes usually of the order of 512 bytes in size. NAND Flash tends to be cheaper per bit, but is prone to more errors at the bit level, so you need software to handle bad blocks. You cannot execute code directly from NAND Flash. One interesting variation using NAND Flash is M-System's DiskOnChip devices. They use NAND Flash internally and combine it with some logic to handle error detection and correction, and also to simplify external CPU access. Don't be fooled by the name, the interface to these devices is nothing like a hard disk drive. Manufacturers of NAND Flash devices include Samsung and Toshiba — both of which supply NAND Flash technology to M-Systems for DiskOnChip as well. NAND Flash device capacity typically ranges from 8MB up to 128MB, with DiskOnChip going up to 1024MB.

Writing to Flash Memory is not like writing to conventional RAM. For both NOR and NAND Flash there is a sequence of steps needed to initiate a write of data. A write will almost always involve an erase cycle on some part of the Flash first.

All Flash memory types have a notion of segments or blocks (or more precisely erase segments). This is the minimum size unit of the Flash device that can be erased. You cannot erase a single byte or just a few bytes. The segment size is very much device dependent. They tend to range from 8KB up to 128KB. An erase operation will leave all Flash memory bits in the erase segment with a logical value of “1”.

Writing of data is actually a process of either flipping a bit to “0” or leaving it as a “1”. So although you can essentially write a value to any address you cannot flip a bit that is “0” back to a “1” without an erase cycle.

Thus generally when you want to change the contents of the Flash you need to do an erase cycle followed by a write cycle. This has some interesting implications, as we see when we look at read/write filesystems for Flash.

The life span of Flash memory is generally measured in terms of the number of erase cycles that can be done. The exact number varies from device to device, but typical ranges are from 10,000 to 1,000,000 erase cycles. As a rule of thumb, you do need to be careful about how often you erase/write a Flash memory segment. This is particularly true when using a read/write filesystem on top of your Flash.

I am largely going to concentrate on NOR Flash for most of this discussion. It is far more common in smaller systems, and it is by far the most used type of Flash memory in uClinux systems.

System

The amount of Flash and RAM contained in any embedded system will always be a tradeoff. Often cost is the primary consideration, with Flash memory generally being more expensive per Megabyte than RAM. Ultimately the sizing will depend on your requirements and exact system details.

The Flash memory is obviously the device that holds the system code and data and where execution will start on power up. It is the primary file store for a uClinux device.

In its simplest form you can just put the uClinux kernel start code at the processor start address in the Flash memory and just use the Flash as a single large chunk of storage.

Alternatively, it is often convenient to partition the Flash into separate regions. This notion is not unlike the partitions of a disk drive, although on a simpler and smaller scale. A typical Flash partition arrangement might be something like:

    SEGMENT      PURPOSE

       0         boot loader
       1         factory configuration
       2
       .
       .         kernel
       .
       X 
       .
       .         root filesystem
       .
       Y

This arrangement has allocated segment 0 to the boot loader code, segment 1 to store factory configuration (say ethernet MAC address, or kernel command line arguments), segments 2 through X to the kernel, and finally segments X through Y to the root filesystem.

This is a simple example. Clearly you could build a arbitrarily complex set of partitions if that is what your system needs. As we will discuss later on, the uClinux block devices that support Flash memory handle this notion of partitioning. Note that the partitions are always a whole number of Flash segments. You really want to do it this way since you can only erase an entire segment at a time.

This example also separates the kernel and root filesystem into individual partitions. This is not the way a normal Linux desktop or server system would be set up. Typically they would have the kernel binary exist as a normal file within the root filesystem. The problem with this is that you need a very clever boot loader (something like LILO or GRUB) that is capable of figuring out which blocks (that contain the kernel binary) of the hard drive need to be loaded into RAM. There are two distinct advantages to placing the kernel image into Flash as a contiguous blob. One is that you can now run it directly from Flash (XIP.) The other is that it is trivial for a boot loader to figure out where the kernel code is in Flash to execute it (you may not even need a boot loader.)

There are several options for placement of the kernel and root filesystem. The best choice very much depends on a number of trade-offs. I'll list some typical configurations here, and we can look at the advantages and disadvantages of each.

• kernel at fixed offset, root filesystem at fixed offset — this option has the benefit of the key system components being at fixed addresses. Easy for the boot loader to find the kernel, and easy for the kernel to find the root filesystem. It will also be simple to upgrade the kernel and filesystem independently of each other. The disadvantage is that you will inevitably have some wasted Flash space between the end of the kernel binary and the start of the root filesystem.
• kernel followed by root filesystem — this option saves some space by not partitioning the root filesystem separately. It is also common with uClinux systems to combine the kernel binary and the root filesystem into a single binary file. This implies that you have to update the kernel and filesystem in a single pass (although often this is a good thing).
• compressed kernel and root filesystem — this option saves a lot of Flash space by compressing the kernel and root filesystem. This option would require a boot loader that is capable of uncompressing the image into RAM. So it will use more RAM, but has a significantly smaller Flash footprint than of the other options. You could also compromise here by just compressing the kernel or root filesystem if that made sense.

There is certainly no reason you could not have multiple filesystem partitions too. This may make sense if you anticipate only wanting to update a section of the filesystem, or perhaps if you have a portion of your filesystem that is read-only, and a partition that you want read/write.

To boot loader, or not to boot loader

The first consideration when booting is CPU dependent. Where does the processor start execution on power up? Many CPU's start execution at a fixed address (for example ARM, x86). Others read a fixed location in the address space and use that value as the start address (M68k, ColdFire). This has a direct impact on where you must put your system startup code in the Flash memory.

At this stage it is worth considering whether you want to use a boot loader in your system, or just start running the uClinux kernel.

There is no problem with the CPU startup just starting execution of the kernel. Your uClinux startup code will have to do all hardware setup (this usually includes setting up chip selects and RAM setup) and it will have to load the kernel data segment in RAM and clear the bss segment. But this is all very straightforward. The only difficulty with this scheme is arranging the kernel code to be at the correct offset in the Flash memory so that the CPU will start executing it on reset. For most modern CPU's that start executing at offset 0 or hold the start address near offset 0, this is quite simple.

A boot loader is a small stand alone piece of code that usually does basic hardware setup (again chip selects, RAM, etc) and then loads or starts execution of the uClinux kernel proper. You can do some useful things in a boot loader. It can prompt the user to load one of multiple kernels present in the Flash, or it can load a kernel and system image through some other I/O device (for example serial port or Ethernet port).

A boot loader can provide some level of protection against a broken or corrupted kernel image. This can be important when used with Flash, for it is possible that a Flash update of the kernel, or other critical data, is incomplete (power failure in the middle of update, or worse, accidental loading of bugged code). The boot loader, which can be locked permanently into Flash Memory, can provide a recovery mechanism to fix this situation.

There are a number of boot loaders in use with uClinux today. Examples of freely available loaders include CoLilo, My Right Boot (MRB), PPCboot and Motorola dBUG. A number of companies have proprietary boot loaders for specialized tasks, for example Snap Gear and Arcturus Networks.

uClinux kernel block drivers

There are currently three choices for the block device that will contain the root filesystem in uClinux:

• Blkmem driver — The blkmem driver is the oldest and may well still be the most common choice for use with uClinux. It was specifically designed for uClinux, but it is relatively simple and only supports a handful of common NOR Flash memory types, as well as root filesystems in RAM. It is also difficult to configure, requiring code modification of tables to set up your Flash partitioning. However, it does provide basic support for erasing and writing Flash regions.
• MTD driver — The Linux MTD drivers are the standard Flash memory drivers for Linux. They support a huge variety of Flash devices, and offer powerful mechanisms for defining partitions and mappings. For anything other than trivial setups you create a map driver that defines your exact Flash layout. It can span multiple Flash devices, with interleaving, and even different Flash device types in the one system. There are a number of configuration options in the Linux kernel config devoted to MTD setup. Use the on-line help to decide on which options you need. It will also definitely help to look at an example device that uses the MTD drivers to get started.
• RAM disk driver — Thirdly, you can use the Linux RAM disk device. This is commonly used in standard Linux for diskless booting. The RAM disk driver does not offer any direct support for any underlying Flash memory, so it is only useful for the purpose of holding the root filesystem. This may make sense on a system where you store the root filesystem compressed in Flash.

The MTD drivers clearly provide the most powerful support for Flash. They also allow you to run real read/write filesystems specifically designed for Flash memory, such as JFFS and JFFS2. You cannot do this with the blkmem driver.

Root filesystem

There are a number of choices for root filesystem in uClinux.

Traditionally the ROMfs type has been the most commonly used. It is a simple, compact, read-only filesystem. It stores all data of a file sequentially, so it allows for application programs to be executed in place (XIP) in the filesystem on uClinux targets that support this (which is true for m68k, ColdFire and ARM). This can can make for a considerable reduction in memory footprint for a running system.

Cramfs is a new filesystem for 2.4 series Linux kernels. It is designed to be a compact read only filesystem. Its primary advantage is that it stores all files compressed and decompresses them on the fly. Because it store files compressed, you cannot run applications in place (no XIP). It is quite space efficient in terms of Flash usage, but more RAM will be required since all application code needs to be copied into RAM for execution.

Some systems will need a read/write root filesystem. By using the Linux MTD drivers it is possible to run a journaled Flash filesystem like JFFS or JFFS2 on top of Flash memory. Journaled filesystems are safe from sudden power loss (that is an unclean shutdown condition), and don't require a filesystem check on the next boot up. Since the JFFS and JFFS2 filesystems are specifically designed for use with Flash memory, they also provide a feature called wear leveling. This is where the filesystem code lays out data and updates it in such a way that all parts of the Flash are erased a similar number of times. This can dramatically increase the useful lifetime of Flash memory devices. JFFS2 has the distinct advantage of storing files compressed, so uses much less Flash space. It should be used in preference to the older JFFS. Something else to be mindful of when using a journaled filesystem is that some small amount of Flash will be wasted for the journal overhead and garbage collection system. This wasted space is typically of the order of 2 Flash segments in size.

If you are using a RAM disk then it is common to use ext2 as the filesystem. You can certainly do this with uClinux too. Ext2 is not particularly space efficient, and being on a RAM disk any changes you make to it will be lost on the next reboot (some consider this an advantage in the embedded space, since you always start your system from a known filesystem state).

There are a number of other filesystems that you could use. Linux has a large number to choose from! But those listed above are the most commonly used with uClinux. There is no reason you couldn't use an MS-DOS FAT type filesystem if you really wanted to.

One more thing worth noting is the way in which a root filesystem is constructed for use with uClinux. Usually the root filesystem for an embedded device is constructed on the development host and then loaded into the target. Typically you create a directory tree in your development environment that is a mirror of what you want the final root filesystem to be, then use a host based tool to construct a binary filesystem image from this directory tree. The genromfs utility is a prime example. Given a directory tree on a host it will construct a file that is a ROMfs binary image. Similar tools exist for many filesystems types (such as mkfs.jffs2 for JFFS, mkisofs for ISO9660).

Flash tools

There are a number of tools available at the application level for use with Flash Memory on uClinux. Some are specific to the underlying block device driver in use.

When using the MTD drivers the primary tools available are:

• erase — erase some flash segments
  
• eraseall — erase all of flash device
  
• lock — write lock flash segments
  
• unlock — write unlock flash segments
  
• mkfs.jffs — construct a JFFS filesystem from a directory structure
  
• mkfs.jffs2 — construct a JFFS2 filesystem image from a directory structure

All of erase, eraseall, lock and unlock are used on the target device in mtd partition devices. The mkfs.jffs and mkfs.jffs2 tools are generally used on a host system to build filesystem images that can be loaded into Flash on a target device. Since the MTD drivers provide standard char and block devices on the target you can use system tools, like dd, to write contents into Flash devices.

Netflash is a tool developed specifically for uClinux that is a nice way to update either MTD or blkmem devices. It takes a file and programs it into a Flash device (also doing the erase step for you). It can load files over the network (via tftp, httpd or NFS) or even use local files to program Flash.

A real example

It is worthwhile looking at a real life example, just to demonstrate some of the details we have looked at. The example system is based around a ColdFire 5272 processor with 2MB AMD Flash and 4MB of SDRAM. (For reference this is the Snap Gear LITE VPN router product).

The system is running a uClinux 2.4.x kernel, and we will use the MTD drivers for Flash support. We won't use a read/write filesystem, but instead use a ROMfs filesystem type.

The AMD Flash selected is a “bottom boot” type. It has a number of small erase segments at the bottom of the address space with sizes 16KB, 8KB, 8KB and 32KB. All the rest of segments are a uniform 64KB in size.

The Flash memory partition map will be:

SEGMENT  SIZE  MTD-DEVICE  DESCRIPTION

  0       16K    mtd0      boot loader
  1        8K    mtd1      kernel boot arguments
  2        8K    mtd2      factory configuration info
  3       32K    mtd3      spare
  4       64K    mtd4      runtime non-volatile config
  5
  .
  .     1984K    mtd5      kernel + root filesystem
  .
  35
 0-35   2048K    mtd6      all of Flash memory

There are a few interesting things to look at here. Firstly notice how we try to take advantage of the smaller segments at the bottom (some Flash memory devices have a set of smaller segments at the top or bottom of the device). Also you can see that it is possible to overlap areas of the Flash covered by different mtd partitions (although you would want to be very careful when doing this. In this example it is used to create a single mtd device that allows reprogramming the entire Flash in one go).

Although not shown in this example, it is possible to order partitions in an arbitrary fashion. It is not necessary to sequentially partition Flash segments.

The kernel is stored compressed. The boot loader sets up SDRAM and then decompresses the kernel into it for execution. The root filesystem is stored immediately after the compressed kernel image. It is used in place in the Flash. Typically the root filesystem is around 1.5MB in size, after removing the space used by the compressed kernel image.

One distinct advantage of having the kernel and filesystem in a single combined image is that a firmware upgrade is a one step reprogramming of the /dev/mtd4 device. For many devices this single firmware upgrade is the best. Version control is much simpler when there is an all-or-nothing upgrade.

The key to this setup is an MTD map driver. You can find the source for this example at uClinux-2.4.x/drivers/mtd/maps/nettel-uc.c. It defines the mapping layout above, and it also configures the root filesystem to be on device mtd4 (it cleverly avoids the kernel binary to find the ROMfs filesystem.)

During boot the MTD drivers print some diagnostics to let you know what devices it has found, and what the partition setup is. For example, in the boot log we will see:

SnapGear flash probe(0xf0000000,2097152,2): 200000 at f0000000
CFI: Found no Flash device at location zero
Found: Toshiba TC58FVB160
number of JEDEC chips: 1
Creating 7 MTD partitions on "Flash":
0x00000000-0x00004000 : "Bootloader"
0x00004000-0x00006000 : "Bootargs"
0x00006000-0x00008000 : "MAC"
0x00010000-0x00020000 : "Config"
0x00008000-0x00010000 : "Spare"
0x00020000-0x00200000 : "Image"
0x00000000-0x00200000 : "Flash"

So the MTD drivers found a Toshiba Flash part, and divided it up into our desired mapping. The MTD drivers have a debug verbosity level that you can configure in the kernel configuration. Increasing this setting can provide a lot more feedback about the probing process, and more detailed information on the Flash devices found.

You can see from the initial probe message that the Flash memory device is mapped into the CPU address space at address 0xf0000000. This address was set in the mapping driver (nettel-uc.c).

In the field the kernel and filesystem image is updated using the netflash utility. The command line is as simple as:

   netflash  imagez.bin

Where you supply the address of a local tftp server to fetch the image from. Netflash does the rest, rebooting the unit when done (a necessary step when your root filesystem is in-place in the Flash).

References

The following URL's will be helpful:

• uClinux.org
• Linux MTD drivers
• ucdot.org — uClinux news and current events

---

About the author: Greg Ungerer is currently “Chief Software Wizard” at SnapGear, a leading vendor of open source embedded technology. He has been working in the software industry for 12 years, and during that time has gathered a wide range of experience in the design, coding, testing, and maintenance of Unix (and Unix-like) operating systems, device drivers, networking, and embedded systems. Ungerer has been working with Linux for 8 years now, and developing Linux-based embedded technology for the last 4 years, and is currently one of the core maintainers of the uClinux kernels with a focus on ColdFire and SPARC.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.