This interview with Dr. Inder Singh, CEO of LynuxWorks, is the second in LinuxDevices.com's CEO/CTO interview series. It addresses the emergence of Linux as a standard the embedded industry can unite around, real-time performance in the 2.6 kernel, how LynuxWorks is prepared to respond to copyright infringement… cases, and much more. Enjoy . . . !

---

Q1: How would you characterize the current state of embedded Linux, as a market and as a technology?

Linux is headed towards “world domination” in the embedded world, even though it is far from achieving this goal on the desktop, something that Linux's most ardent supporters dream about. You can see this coming when Wind River, one of the most visible detractors of embedded Linux, capitulates and decides they have no choice but to jump on the bandwagon.

It is impressive how rapidly Linux technology has advanced to cover the requirements of an ever-growing portion of the embedded applications universe. The 2.6 kernel makes Linux credible for many real-time applications and provides improvements for both low and high-end systems. The CE Linux Forum (CELF) is working on solving some of the needs of consumer products such as fast boot and power management. Open Source Development Labs (OSDL) has enabled Linux to address high availability and other requirements of the communications infrastructure with Carrier Grade Linux (CGL).

Q2: What challenges do embedded Linux OS, tools, and services vendors face?

The business model remains an issue. Small system integrators or software services businesses can do pretty well in the embedded Linux market but it is not clear if you can build a large, profitable business purely around an open source embedded Linux OS and tools. In our case, the combination of open source and commercial products with strong synergy between them does provide the basis of a robust business model.

Q3: What market opportunities do you see for Linux in the embedded devices and systems market?

Embedded Linux found initial traction in networking devices like gateways and firewalls as well as entertainment products like TiVo. Now Linux is making inroads across the board in a wide range of devices and systems, from hand-held consumer devices to industrial products to large defense programs.

The semiconductor industry is playing an important role in the emergence of embedded Linux. Software support is crucial to the success of semiconductor devices aimed at embedded markets, and Linux provides a common denominator with growing market momentum. Most new devices are being launched with Linux support already available. The “freedom” and vendor-independence offered by Linux extends to the semiconductor vendor – for example, instead of having to get a real-time operating system (RTOS) vendor to support your device you can work with one of several embedded Linux vendors or do it yourself. And you can distribute the software freely.

Q4: What challenges does embedded Linux face as a technology?

An open challenge for embedded Linux is how to address markets that require certification for safety such as DO-178B or security such as the Common Criteria certification. Reliability and security are becoming increasingly important for embedded systems as they become more pervasive, and Linux has a pretty good record in this area compared to, for example, Windows. But for the highest levels of certification in these areas, there are stringent development process and documentation requirements that include complete traceability between requirements, high level and detailed design, implementation, and verification. It is not clear how these requirements can be reconciled with the community-oriented open source development methodology.

Fragmentation remains another issue. The Open Source developer community, the open multi-vendor nature of embedded Linux, and the wide range of choices available to users are major strengths. But they also present the risk, both real and perceived, of divergence and fragmentation. The memory of the Unix wars is still fresh in our memory. Vendors of proprietary systems are threatened enough by Linux to use this threat of fragmentation in the campaign of fear, uncertainty and doubt (FUD) that has been unleashed against Linux.

All embedded Linux distributions start with the same kernel from Linus Torvalds and team and there is, in fact, a high level of compatibility between them. However, different vendors add value in their own unique enhancements and sub-set the system in different ways to reduce footprint. Developers can further modify and configure the system in different ways.

For Linux to achieve its potential as the unifying open standard for embedded applications, it is critical to provide an assurance that any embedded middleware or application software, which follows an appropriate set of guidelines, will be supported by all conforming embedded Linux products from any vendor.

The ELC Platform Specification (ELCPS) [info here. –ed] standard from the Embedded Linux Consortium (ELC), which specifies the APIs for the core OS functionality that must be available in any conforming implementation, can play an important role in addressing this challenge. Test suites and wide industry acceptance of this standard are needed. The embedded Linux community also has to address standard interfaces for user interfaces, multimedia support, and other areas which are important to embedded developers, but for which there isn't a single Linux solution. The main competing solution from Microsoft does have the benefit of providing a single, much more complete, solution.

Q5: What embedded Linux technology developments do you find exciting?

The most significant technology developments are in the exploding ecosystem growing around embedded Linux. There is hardly a software product relevant to embedded systems, commercial or open source, which is not available for Linux. Most new silicon devices have Linux driver support or a Linux port in the case of devices with processors.

With embedded Linux itself, some of the work being done by the CELF in the areas of fast boot and power management will open new opportunities.

Q6: Can you share one or two of your company's most exciting successes?

It was exciting to have VeriFone as one of our first customers to utilize our dual-OS suite. VeriFone is the market leader in the point-of-sale credit card terminal market, a very competitive, high volume business. In their new product line, they use BlueCat Linux in the cost-sensitive terminals and LynxOS in the controller for the store / gas station for which performance and real-time requirements are more demanding. Based on the success of their existing products, this could mean the deployment of millions of copies of BlueCat in credit card terminals.

Q7: How about a failure?

When we introduced LynxOS over ten years ago as the first Unix-compatible hard real-time operating system, it, like Linux, required memory management hardware to support the Unix process model. The leading CPUs used for embedded applications at that time did not have memory management units (MMUs). As a result, LynxOS could only be used in high-end applications. This left the sweet spot in the market to competitors like Wind River and Integrated Systems allowing these companies to grow much faster than LynuxWorks. Now, of course, this competitive disadvantage has turned into an advantage because most processors targeting embedded systems have MMUs and the competing flat address space RTOSes do not take advantage of this facility. In retrospect, we should have developed a uCLinux-like version of LynxOS to support MMU-less CPUs. But then again, hindsight is always 20/20.

Q8: What's your vision of the future for embedded Linux? That is, how big will the embedded Linux market get and when will it start to reach that peak?

It is clear that Linux is on its way to becoming the dominant operating system for embedded devices. The most exciting aspect of the emergence of embedded Linux is its potential to become the unifying force for this fragmented industry. Linux provides, for the first time, a common open multi-vendor platform around which a software industry can develop, similar to what has happened around DOS and Windows in the desktop world.

The software complexity of embedded systems is rapidly growing to where the old paradigm of developing each application from scratch has to give way to extensive reuse of existing software and building on off-the-shelf solutions wherever possible. A common OS platform is a key ingredient for fuelling the growth of a software base that will make this a reality. With Linux, this includes both open source and commercial software.

I believe Embedded Linux will have finally arrived when vendors start promoting “Linux inside” and customers start asking for Linux powered products. We are starting to see a few signs of that already.

Q9: How do you see the Eclipse tools platform fitting into the future of embedded Linux development?

I think it is going to be a popular framework for embedded Linux development. It appears to be quite well developed for Java. The C and C++ tools, which were lagging, now seem to be getting there as well. The strong backing of IBM, a growing base of plug-in products, and an expanding open source developer community are all pluses. We are actively working on an Eclipse based offering, which will be released soon. This will complement our VisualLynux environment, which is based on Microsoft's Visual Studio and is very popular for Windows-based cross development.

Q10: How do you see the 2.6 kernel affecting the embedded Linux market?

The 2.6 kernel provides several enhancements to Linux that are important for embedded applications, including significant improvement in the ability of Linux to provide real-time response. Although the real-time performance of the 2.6 kernel is still an order of magnitude slower than RTOSes like LynxOS, it is adequate for a large number of real-time applications. Real-time enhancements to Linux have been available from several vendors, but they have taken different incompatible approaches. Most developers who needed real-time performance have stayed with RTOSes because they did not want to use a one-off variant of Linux. Thus the 2.6 kernel, by providing real-time performance in mainstream Linux, takes Linux into the domain dominated by RTOSes and expands its market significantly. Taken along with a host of other enhancements applicable to embedded systems, I believe the 2.6 kernel will provide that additional boost for embedded Linux that will establish its dominant position in the embedded industry.

Q11: Any comments on the SCO mess?

Mess is good way to put it. It certainly has added to the FUD surrounding Linux and open source. But much of the preoccupation with this subject is in the media and the analyst community. The user community does not appear to be holding back from using Linux in existing, as well as new, embedded projects.

We are all still waiting to see the facts behind SCO's claims. Nevertheless, from everything I have read I seriously doubt that SCO will prevail, at least to the extent that all users of Linux have to pay royalties to them or that the General Public License (GPL) structure gets invalidated. By the time SCO's claims are put to the legal test Linux programmers around the world will have already generated new open source code to bypass any portions currently in question and, in effect, the debate will become a moot point. I understand much of the code where there was any perceived risk relating to its origins has been scrubbed in the 2.6 release.

At LynuxWorks, having developed a clean-room work-alike of UNIX in the form of LynxOS, we have both the experience and the code base to quickly provide non-infringing replacements for any UNIX code revealed to have been illegally included in our embedded Linux product.

Q12: What do you think about Wind River's recent public endorsement and announcement of support for embedded Linux?

I see it as a vivid demonstration of the unstoppable momentum of embedded Linux. Of course, Wind could have done this four years ago and recognized Linux as an opportunity rather a threat, as we did at the time. They may have a bit of a challenge overcoming their own position as a leading detractor of embedded Linux.

Overall, however, I believe it is good for the embedded Linux market and, most importantly, customers, who get still more choices in the tools area.

Q13: What is LynuxWorks' role in the acceptance of embedded Linux?

I believe we have significantly contributed to the acceptance of embedded Linux. We were the first established embedded OS vendor to embrace Linux and we have been a strong promoter of embedded Linux from the time we launched our Linux initiative in 1999.

By adding binary compatibility with Linux to our LynxOS RTOS and marketing it as a part of a family along with BlueCat Linux, we have added to the embedded Linux momentum. Among other things this expands the domain for the Linux platform, as defined by the application software interface of Linux and the growing base of Linux software, into the market space covered by RTOSes. It also provides a migration path and lowers risk for developers selecting BlueCat Linux for embedded systems with real-time requirements.

Q14: Can you tell us about any customers and projects that you are helping them with?

Protocom, one of our BlueCat Linux customers, has developed a system-on-chip (SOC) device, the MPEG-4 audio video (A/V) CODEC with an ARM RISC CPU core. This is a powerful building block for a wide range of consumer products including personal video recorders (PVRs), set-top boxes, digital camcorders, and hybrid digital still cameras. It is interesting that Protocom decided that they needed to provide an embedded Linux operating system to their OEM customers, the only OS bundled with the device. [more about Protocom here. –ed]

Q15: LynuxWorks offers two OSs — open source BlueCat Linux, and the company's proprietary LynxOS with binary-API Linux compatibility. How is the dual-OS strategy working out, from a business perspective? Some competitors have criticized the approach as a “bait and switch” tactic – could you respond to that concern?

The “bait and switch” characterization is merely a case of sour grapes on the part of some competitors who do not have the ability to offer this kind of choice and flexibility to their customers. The fact is if we tried to pull a “bait and switch” with customers who want to use Linux, they would simply go to a competitor or do it themselves. This is the beauty of Linux: it frees customers from the tyranny of vendors who try to lock them in.

The dual-OS offering provides our customers a broader choice so that they can select the operating system that best fits their requirements. We offer them a suite of two excellent, well tested, well supported operating systems: an open source embedded Linux and an RTOS, both with the same software interfaces and a common set of commercial grade development tools. If fact, we have added a third choice, LynxOS-178, for safety critical applications.

This strategy has been great for us from a business perspective. It allows us to cover a much broader market and to better leverage our marketing and sales investment. Actually, because of the high degree of compatibility, there is also a lot of common R&D investment in areas like development tools, test suites and device drivers. Customers like the flexibility and choice this provides. In many cases this also provides them a migration path, as well as risk mitigation. For example, some customers are developing a product with Linux, but if it turns out that Linux cannot meet their real-time requirements, remembering that requirements often evolve, they know that they can move to LynxOS, without having to do any significant porting work or switching to a new developing environment. Other customers know that they can start with LynxOS and move to Linux if they need to add a lower cost, higher volume version of their product. In either case, they get to preserve the investment in their applications software, which is growing in complexity and cost, as well as the investment in training engineers for a development environment.

We have several customers who use both BlueCat and LynxOS in different products and, in some cases, in different parts of same distributed product.

Q16: How will the enhanced real-time performance of the 2.6 Linux kernel impact LynuxWorks' LynxOS RTOS business?

The 2.6 Linux kernel will definitely enable Linux to eat into what has traditionally been RTOS territory. But at the same time, it will make LynxOS, which is compatible with Linux, even more attractive against other RTOSes with proprietary APIs. Basically, anything that adds to the momentum of embedded Linux helps LynxOS to increase its share of the RTOS market. Even with the real-time improvements to Linux, there remains a market for RTOSes for serious real-time applications — our benchmarking shows LynxOS to have an order of magnitude better real-time response than Linux 2.6. So I believe that in addition to expanding the market for BlueCat Linux, it will only help us to grow our LynxOS business even as the overall market for RTOSes is reduced.

Q17: Has LynuxWorks contributed Linux kernel development work to the source tree? If so, could you elaborate on some contributions your company has made?

Our engineers have provided feedback on several kernel bugs, including a networking bug with a significant performance impact that was discovered using our Spyker tool. We have made other contributions to open source, including our Messenger backplane messaging technology and several gdb enhancements, some of which were contributed even before LynuxWorks offered BlueCat Linux. We are currently working on Linux kernel enhancements that we will certainly contribute back to the open source community.

Q18: What other significant trends do you foresee in the upcoming year for the embedded industry?

The trend towards putting intelligence in more and more products leading towards a world of pervasive computing is continuing. Also, as we become increasingly dependent on embedded software, reliability and security will get a lot more attention, and become important competitive differentiators.

---

About the inteviewee: Dr. Inder M. Singh is the CEO and chairman of LynuxWorks and board chairman and president of the Embedded Linux Consortium (ELC). Dr. Singh holds Ph.D. and M.Phil. degrees in computer science from Yale University and an MSEE from Polytechnic Institute of New York. Recently, Dr. Singh was appointed to the San Jose Economic Leadership Team to help provide economic development strategies for the city.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.