Foreword: This whitepaper by Cliff Brake, an embedded systems consultant, offers practical advice about using Linux in embedded projects. Topics covered include determining Linux suitability, evaluating costs and risks, project scope calculations, scheduling and timeline estimations, and operating system and embedded product lifecycle management. Enjoy . . . !

---

Introduction

Are you planning to develop a device that requires an embedded operating system?

This paper will help you ask the right questions and plan for a successful project. With so many options, it is critical to do a little planning up front. What does “embedded” mean? In the context of this paper we are talking about Embedded Systems which generally means any computer embedded in a device that is not a desktop computer or a general purpose server. LinuxDevices.com lists a number of devices that utilize embedded Linux.

The following issues will be discussed in this paper:

• Is Linux the right OS for your device?
• The BIG WIN with embedded Linux
• Overview of the components, approaches, and skills required
• How much does it cost?
• How to manage the risks
• Do you need to maintain your system?
• Thoughts on IP and licensing issues

When to use Linux in an embedded application

There are many options for embedded operating systems and each solution has its strengths and weaknesses. The use of a Linux class operation system (OS) in your device can provide a great deal of functionality and flexibility, but it comes at a cost — complexity. Linux is generally good for systems that have one or more of the following attributes:

• Require a high level of performance (Ex. networking throughput)
• Require complex, standards-based applications (Ex. Complex GUI, Web Server, Web Application integration)
• Utilize standard, but complex interfaces (Ex. Ethernet, USB, advanced storage options)
• System is upgradeable or expandable
• System must be scalable

Each one of these attributes will be examined in more detail.

Performance

Many embedded systems today require a high level of performance. If you need to move a lot of data in a short amount of time, Linux is a good choice.

Complex applications

Would your system benefit from an integrated web server? Being able to access a device from any web browser is a significant advantage in many applications and can reduce the cost of deployment and use. There is also a significant trend where devices are required to integrate directly with web services. A significant movement is the Web 2.0 concept where different web applications communicate directly with each other. There is no reason your device can't integrate directly with web services using technologies such as SOAP. Linux is a good choice for systems that require standards based complex applications.

Complex interfaces

Linux provides drivers for many standard interfaces such as networking and USB. Standard interfaces give you many options for adapting and expanding your system in the future. System must be upgradeable or expandable

Many devices can benefit from the ability to continuously add features or adapt a system throughout the product lifecycle. Embedded Linux systems offer many options for easily loading new applications and software updates that give you the flexibility to meet future customer needs.

Linux is scalable

Linux runs on most 32- and 64-bit processors available today and is often the first operating systems ported to new processors. There are also many single board computers and hardware platforms available that can be used to build an embedded Linux system. Writing your application on top of Linux means you have the option to move it to different platforms and hardware as performance and other needs change. Linux is very scalable and flexible.

If your system has some of the above characteristics, then you should seriously consider Linux. There are other solutions available as well. Microsoft Windows CE is probably the OS most compared to embedded Linux. Both OSes have their strengths. For example, if your application team wants to write the application in the .NET Compact Framework, then Windows CE is a good choice. If you need flexibility and performance, then Linux may be a good choice. Every application is different and there is no general answer to which is best. Also if your system is single purpose and does not require complex standards based interfaces and applications, then application specific solutions may best, as they cost much less to implement.

The “Big Win”

Linux provides an impressive level of device and application support for embedded systems. How is this accomplished? The answer is the server and the desktop. Many of the technologies developed for Linux on desktop and server systems can be used directly in embedded systems — this is the “Big Win.”

The re-use of production grade technology is powerful and is what gives Linux its competitive edge. For example, the networking stack used in a mission critical web server is the same one used in your embedded device the size of a credit card. Another example is USB. If you implement a USB port on your device, you can expand your system with any USB device that will work on a Linux desktop system. No other operating system scales from tiny embedded systems to mainframe computers. Because Linux functionality available on desktop systems can be used directly in embedded systems, Linux provides connectivity to many standard interfaces such as Ethernet, Bluetooth, USB, CAN, etc. In summary, Linux gives you a production quality OS with good device support.

Another aspect of the Big Win is that in many cases, it is easy to prototype critical aspects of your application on standard PC hardware. This allows you to verify that some aspects of the system will work as expected and allows you to get a head start with application development.

What makes up embedded Linux?

Embedded Linux is a general term used to refer to a range of components required to make the system function. Linux technically refers to the operating system kernel, which is only one piece of the system. If you are not talking about the following aspects of your project, you may be missing something:

• Bootloader — responsible for initializing the hardware, loading the Linux kernel, and system updates
• Linux Kernel — provides access to hardware and standard operating system services
• Root file system — basic file system and utilities required to run Linux. May also include graphical toolkits, etc.
• Applications — the part of the system the user sees, and is often customized for each device
• Tools — compiler and other tools required to build and debug the system
• Maintenance — how are you going to maintain your project going forward

Approaches and required skills

There are many ways to implement an embedded Linux system. If you don't want to design your own hardware, there are many embedded computer board manufacturers that supply off-the-shelf hardware. Using off-the-shelf hardware is also useful for the prototyping phase of the project. Where do I get embedded Linux? Several options are listed below:

Source	Advantages	Disadvantages
Open source projects (kernel.org, openembedded.org, etc.)	large selection good support easy to stay in sync with new developments	high level of expertise required to find and integrate components current state is sometimes unknown
Commercial Linux Vendors (TimeSys, MontaVista, etc.)	what they provide usually works often provide additional features	can be more difficult to synchronize with mainstream developments may be higher cost
Provided by single board computer vendor	less initial effort required	less flexible — you don't get to choose what you are starting with depending how the software is implemented, it may be very difficult to synchronize with mainstream sources to add new features

Details of the above approaches will be covered in future papers by BEC Systems. What type of skills are required to implement an embedded Linux system? The most important skill is to be able to use, integrate, and debug someone else's code. Porting Linux to a new system is 99 percent integration. Engineers who like to write everything themselves may not be very effective. Other useful skills are a good understanding of hardware, computer architecture and operating systems. Having the experience and knowledge to manage the complexity of modern embedded Linux systems is really the key to success.

At some point during your development, you will probably run into some type of problem and need support. With a system as large and complex as Linux, no one is an expert at all of it. Being able to ask the right person a question can save you days of debugging. Fortunately, the right people are often accessible on email lists. It is rare to not have a properly framed question answered within a day. This is another area where a few hours from an experienced consultant can save a lot of time when tracking down solutions to problems.

Tools are an important issue for many developers. The Linux community in general does not put a lot of emphasis on fancy IDEs and debuggers. However, there are a number of commercial Linux vendors who provide well integrated tools.

One common request is to be able to do embedded Linux development on a Windows computer. This is very possible and commercial vendors support it, but in my opinion, there is a lot of value in using a Linux host system as this is a good way to improve general Linux skills. Linux skills on the desktop are directly applicable to embedded Linux systems. Someone who cannot (or will not) use Linux on a desktop computer may have trouble working with Linux on an embedded system. PC's are cheap and there is no reason a developer cannot have two PC's — one for windows and one for Linux. It is fairly easy to access a Linux development system remotely to avoid cluttering your workspace with two displays, keyboards, etc. It is also possible for multiple people to use one Linux workstation. This is another area where a small amount of training and setup from an experienced consultant can go a long ways toward making your development run smoothly.

Count the cost

Although Linux and supporting technologies are free to use in your device, the cost to implement and maintain a solution needs to be considered. There are many ways to implement a system. As with most other projects, you can buy off-the-shelf solutions and lower your up front costs, but with higher volumes it often makes sense to implement a custom solution. Due to the complexity, implementing an embedded Linux system on custom hardware is a significant undertaking.

The following is sample schedule for implementing Linux on custom, non-x86 hardware. This schedule is only presented to provide a rough idea of the amount of work required to bring up Linux on a new device. This schedule can be reduced significantly by using well supported off-the-shelf hardware, but we'll assume for a moment you need a custom solution for a number of good reasons.

Task/phase	Description	Time estimate (man weeks)
Technology selection	Evaluate various hardware architectures, embedded Linux sources. This phase involves prototyping aspects of the system.	3 weeks
Bootloader and Linux kernel porting	Adapt the bootloader and Linux kernel to custom hardware design, assuming not a significant amount of new code needs to be written	2 weeks
System bringup	Debug bootloader, kernel, and minor hardware issues	2 weeks
New device drivers	Device drivers for one new device	2 weeks
Ongoing system issues and support	Resolve ongoing system issues. Typically some amount of support at the system software level is required during most of the project. This time is spread out over the duration of the project.	3 weeks
Total:		12 weeks

The above does not include time for training, application development, power management, diagnostics, maintenance or other requirements often found in real world systems.

Where are the risks?

As mentioned before, embedded Linux systems are complex. Linux provides developers a tremendous amount of functionality, but it is easy to underestimate this complexity. We'll use the common iceberg analogy. Let's call your application software (the part you write) the tip of the iceberg. But, the vast majority of the iceberg is under water — this is the part you don't write and obtain from other sources (bootloader, kernel, root file system). It is easy to get focused on the part you see as this is where you are spending your time and resources, but what happens under the water can have a big impact on your project.

To further emphasize this point, lets assume your application will require one man-year of development. This might result in 3000 lines of code (LOC). A fairly minimal embedded Linux system might include a bootloader, the kernel, and a basic root file system. These projects might include roughly two million lines of code (rough estimate of code actually compiled for the project). Figure 1 shows the amount of source code written by your team (lets call this the known) versus the amount of source code you did not write (called the unknown). Although this figure is somewhat ridiculous, it helps to illustrate the point. This leads us to main point #1…

Main point #1: The most important part of an embedded Linux project is starting with the right stuff.

If you start with the wrong stuff, you end up spending a lot of time doing work in the unknown you did not plan for. Problems in the unknown can manifest themselves in a number of ways:

• a driver is not available for some of your hardware
• the board support package (BSP) provided for the processor or single board computer you are using is low quality, or not well integrated with main stream sources — there is no way to maintain it going forward
• you run into “problems” that are difficult to debug

Figure 1: Linux vs. application code size

How do you manage this risk? One of the best ways is to prototype your application before committing to a design and processor selection.

As a first step, you can prototype your application on a PC. Once that is a success, you can verify key aspects of your system using an off-the-shelf single board computer that closely matches your target hardware. If you encounter problems, you can switch to a different processor architecture or embedded Linux source before you have too much invested in the project.

The technology selection and prototyping phase is a good time to bring in an experienced consultant who can help you quickly evaluate various options. A good consultant can help identify problem areas and provide training for your team before the project crunch time hits. All too often, consultants are called in too late in the project after uninformed decisions have been made. Again, the way you save time on an embedded Linux project is start with the right stuff.

Product life cycle and maintenance

It is important to understand the product life cycle early in the project. You might ask the following questions:

• How long will this product (or follow-on products) be sold?
• Will we ever want to add new features to the product?
• How will we fix bugs and security problems?
• Will our customers ever want to add features or customize how this product is used?
• Will we ever want to expand the functionality of our product by connecting off-the-shelf devices to standard interfaces?

As noted in the previous section, most of the software in your product is in the unknown or “under water.” This leads to the conclusion that if you ever want to add new features to your product that require support at the system software level, you need to understand and have control of the unknown.

Let's use a simple example. Suppose you design a system with a USB port. At a later point in time (say two years later) you want to support a new USB WiFi device in your system that is now available on the market. A driver for this device was just made available in the latest Linux kernel. How do you add this driver to your device? Due to the nature of Linux kernel development, you can't just drop it and and go as the new driver depends on recent kernel features that are not available in the version of the kernel used on your device. If you have kept your system up-to-date with mainstream source code, this is trivial. If not, you are faced with the large task of updating the system software in your device, or back porting the new feature to your version. Back porting tends to not be a practical option as time passes. This leads to main point #2…

Main point #2: If you ever plan to add new features to your product that depend on system software, you must keep up.

There are several ways to keep up:

• periodically integrate your code changes with the latest versions of the projects you are using. With the tools available, this is easier than you may think.
• rely on commercial vendors to provide you with updates

In general it is not a good idea to get too far off the beaten path if you plan to maintain an embedded Linux system. The ideal solution is to use the mainstream sources from kernel.org, etc. This is now a viable option as many embedded processors are now supported in the mainstream Linux kernel source. If you use a Linux kernel from someplace other than a mainstream source, you might want to ask the following questions:

• Will it be maintained? For how long?
• What are the differences from the mainstream sources? What is the quality of the changes?
• Why were the changes not pushed into the mainstream sources where they can be more easily maintained?

Intellectual Property and Licensing

Intellectual Property (IP) and licensing is a controversial issue with Open Source Software (OSS) and Linux. Much of the software used in an embedded Linux system is licensed under the GNU Public License (GPL). This license states that if you use GPL licensed software (or modifications of the software) in a product, you must offer to provide source code for the GPL licensed software with your product. Many companies provide this software on their web site. You should plan to comply with this, as companies that don't comply usually end getting quite a bit of bad publicity. Linksys (a manufacturer of wireless routers and other consumer grade networking gear) uses Linux in many of their products. They provide the GPL licensed source code directly on their web site for anyone to download and modify.

The GPL is disturbing to many who are not familiar with OSS. What about my IP? Am I giving away my IP and helping my competitors? I've spent a lot of money getting Linux running on my device and now I have to give that away? It is helpful to understand the value of work you do to port Linux to your device.

I'll use an analogy of an old upright piano that is advertised in the newspaper for free to anyone who will come and get it. The piano is analogous to Linux. You borrow a truck, get four of your friends to help, and move the piano to your house. It is a lot of work, but now the piano has value because it is now at your house where it is useful to you. Porting Linux is like moving the piano. Moving the piano does not add any value to the piano itself. Likewise, porting Linux to a new device does not significantly change or improve Linux. The fact that you ported Linux to your device does not really make Linux any more useful to use on a different device. Linux is a commodity. It is not the secret sauce of your product. The application you run on top of Linux is where you differentiate your product and you do NOT have to license your applications under the GPL. Even though Linksys provides source code to the GPL licensed source code in their products on their website, they do not provide source code to the applications. The applications and hardware design is what differentiates Linksys products.

In summary:

• You must offer to provide source to the Linux kernel and other applications licensed under the GPL with your product
• Typically, there is nothing proprietary in the modifications to GPL licensed code in a Linux system.
• You can keep your applications proprietary. This is where you differentiate your product.
• Make sure you have source code for all Open Source components used in your device.

Summary

As with many endeavors, up front planning on embedded Linux projects is important. In summary, make sure you cover the following points.

• Is Linux right for the application?
• Are you looking at all of the components required (bootloader, kernel, and root file system)?
• Am I starting with the right stuff?
• Do I need to prototype the system to make sure I am starting with the right stuff?
• Does the product need to be maintained?
• How is it going to be maintained?
• Does management have an accurate understanding of IP/licensing issues
• Do I have source code for all Open Source components?

Copyright (c) 2005 Cliff Brake. All rights reserved. Reproduced by LinuxDevices.com with permission.

---

About the author — Cliff Brake owns BEC Systems, a consulting operation that aims to help customers utilize modern computer technologies in their devices. BEC offers a range of services including technology selection, troubleshooting, development and training.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.