Conversations with Lineo's Bryan Sparks and MontaVista's Jim Ready

Out of nowhere Linux has suddenly became the leading operating system for embedded devices and applications. There was no battle. No CEO surrendered his keyboard to an army of penguins. No stiff-suited guy talking to a bank of microphones. No parades.

But it happened. Linux won the embedded space. The only other OS vendor of any significance in that space, Wind River, still makes a fine living doing what it has always done, which is suppling semiconductor companies and various manufacturers with embedded and real time operating systems (RTOSes), software development tools and support, to the tune of a few hundred million dollars in revenue per year. Not bad. Also not significant.

Go back a few years and two other leaders in the space were Lynx and Ready Systems. Today Lynx is Lynuxworks, and Jim Ready – perhaps the leading personality in the history of embedded operating systems – runs MontaVista, one of the top new Linux embedded OS companies.

Meanwhile Bryan Sparks, who made Caldera the first well-branded Linux distribution back in the mid-Nineties, is charging forward with Lineo, a Caldera spin-off that is on an aggressive campaign to blow away not only its Linux competition, but Wind River as well.

It's hard to find any hard numbers about Linux adoption in the embedded world, just as it was hard to find hard numbers about the Internet when nobody seemed to notice that it was taking over. But it's happening. Just as surely, and just as rapidly.

The embedded processing market is enormous. It's everything that processes bits and isn't general purpose computing. Look at it another way: it's everything that could possibly connect to the Ne,: the sprinker system in your lawn, the conveyor belt controller at a distribution facility, all the medical instruments at your doctor's office, the navigation systems in airplanet … anything and everything that relies on digital intelligence. This is the world laying at Linux's feet. It's all Antactica now.

Why? The short answer is engineers. Hey, they like Linux. It's familiar. It's small. It's net-native. It's fast. It's reliable. And it's open to improvement by anybody who wants to bang on code. What more could you want?

Lots, apparently. A faster, “harder” real-time Linux would be good. Visual development tools. Ever-better ways for engineers to support each other. The embedded world is open for renovation and reinvention from top to bottom. Next to embedded, what's happening in the rack & box world of client/server computing is micropotatoes.

Where there's more than enough business to go around, there's room for more than one leading market personality. Such is the case in the embedded Linux world, where Lineo's Bryan Sparks and MontaVista's Jim Ready are not only pushing development envelopes, but also each other. While there is clearly much mutual respect, there are also vast differences. At the risk of oversimplification, MontaVista is all about development while Lineo is all about business, which are exactly the two thrusts, literally, of Linux development Both will surely succeed, but one suspects for largely different reasons, which is why their two perspectives give us a stereoscopic view of a marketplace that's as new as Now. Or maybe newer.

On September 7 and 8 we spoke with both gentlemen at great length, recorded the conversations and found ourselves with a 16,000-word transcript. What follows is about 4,000 of those words. We e hope our edits do justice to their unique visions for this whole new development universe.

– Doc Searls

---

Jim Ready, MontaVista Software

How did you get started in embedded Linux?

My first job out of Berkeley back in '76 was with an early mil-spec computer company: Rolm. Back then it was all about cross-development. They would have development stations just like we've had for embedded work ever since. What changed since then were form factors. These Rolm boxes were huge.

Then we were fortunate in '81, when the 68k was announced. It became clear that the minicomputer situation was going to change. You would be able to build good computers out of micropricessors. But there really wasn't a standard, off-the-shelf, constant-across-microprocessor operating system.

Each was essentially a stovepipe that went up from the silicon all the way through applications.

If it existed at all. So we thought, why not build an RTOS – a real time operating system – that would, independent of processor architectures, present the same interface to the customer whether they were on an 8086 or a 68k? Then let them program in the languages that were popular at the time – PLM or PASCAL – and let customers pick and choose without changing their devlopment environment. That was the original Hunter & Ready solution, which is still valid to this day. Hunter & Ready later became Ready Systems.

I had forgotten about Hunter.

Here's a totally weird coincidence: Colin Hunter became one of the founders of Transmeta, where Linus now does his work.

It was an interesting fact to me that when I first met and interviewed Linus, he brought the conversation around to embedded devices. He did the same in his speeches. Wtthout giving away too much about Transmeta, he kept saying that the most interesting future for Linux wasn't in beige boxes.

Obviously he's a smart guy who can figure this stuff out. We certainly agree that the most level playing field – the one that favors Linux most – is the embedded space. Microsoft owns the desktop. Servers are competitive: Linux does well there against Solaris and NT. But in embedded, there's just Wind River. They're the biggest player, but still a tiny fraction of the overall market. That market is begging for standardization.

Wind River looks pretty big.

VXWorks is fine, but it's known only to its developers, it's closed source and nobody knows how it works. So, as much as Wind River would like it to be mainstream, it isn't, nor is it going to be. Meanwhile there is this pent-up demand in the embedded area to get normal. Linux is the shining light in this dark space. The customers are like moths to a flame, in a good sense.

Let's get back to your history. How did you get into Linux?

In 1992 I brought Linux into Ready Systems, coincidentally to the building I am in right now. I brought all the engineers into the cafeteria and said hey, I just downloaded this stuff that's been produced on the Net by a bunch of guys, including this guy from Finland, and I built these two floppies, doing raw writes out to a Sun workstation. If you know anything about any of this, you know that the probability of anything working was zero. But I stuck it into a clunker 386 that I bought at a junk show and the goddam thing worked. It came up. And it was UNIX. I was blown away. It was a very sophisticated system. We got X Windows running, put it on the network, and found acres and acres of functionality were working, very stably. Relative to the embedded software of the time, this thing was so far ahead in terms of functionality that it was an amazing eye opener. We actually did an internal study about Linux and wrote a paper called VNIX – I forget the spelling, but it stood for “VRTX is not UNIX.” It was basically about how to use Linux in an embedded operating system.

What happened?

We had a company to run. We got busy and got merged and did a bunch of other things. Still, it has been on our minds ever since. Every month I got my copy of Linux Journal. I was a very early subscriber.

We started in '94.

Yeah, one of my engineers showed me the very first edition, and I immediately subscribed. I noticed that virtually every month the magazine got thicker, the paper got better, the ads got more sophisticated, the articles got better. You could tell that something was going on. So my indirect measure of the goodness of the Linux market in general was Linux Journal. Seriously. It was my benchmark. You guys should know that you reflected what was going on.

What happened to Ready Systems?

We merged with Microtec Research in '93. With Microtec we were the largest combined embedded RTOS tools company – larger than Wind. That was '93. In '94 we went public. Then in '95 Mentor Graphics, the EDA company, in a future-looking move, bought Microtec. And that was how the food chain worked.

Then what?

Without throwing rocks at Mentor, let's just say that the business did not prosper for any number of reasons. But I kept my subscription current in Linux Journal, followed the Linux distributions. But the shocker was an article about four or five years ago, by a guy from NASA Langley, about a side-looking radar system. The previous version of this thing was based on Multibus-I with two CPU boards, one of which ran VRTX – my software. The other one ran some UNIX. He wrote that they managed to replace those two systems with one running Linux. And I thought, Hey! Something is going on here! VRTX was being replaced by Linux. So, between that and knowing where the customers were going, we could see Linux as a serious player in our space.

Let's jump to the start of MontaVista.

When I left Mentor Graphics I went to the guys who had invested in Hunter & Ready in 1981, who by now were old friends of mine, and said “I have not come to you with a new idea in eighteen years.” That was in March of '99. For the first six months we were exactly what our competitors called us: seven guys in Sunnyvale. We didn't hide that. We were testing everything like crazy: customer requirements, hosts, targets, real time… We had some preliminary hypotheses. We tested those and came down to some interesting discoveries.

Such as?

We thought that real time would be a big factor, but there were many others that were in fact more important in terms of customer acceptance. Those needed to be addressed first, so we re-ordered our priorities.

Starting with what?

The fundamental one was availability. If you go to Fry's and get Linux from Red Hat or anybody else, it's a self-hosted X86 or possibly PowerPC system that works fine for installing on a PC or a Mac. But the classic paradigm for embedded systems is cross-development, where the system is hosted in one place and targetted somewhere else, because an embedded processor typically is not capable of supplying its own development. So the very first job was making Linux available and supporable as a cross-hosted environment, which means your Solaris machine or Linux box would let you compile, do a sysgen, build the kernel, and also compile applications and then download them into the embedded target.

How many hosts and targets do you support?

We support four processor architectures and three different host machines.

I'm guessing the host, the development platforms,s are PowerPC,X86 and Solaris.

Yes. But there are varieties on those as well. RedHat 6.1 and 6.2 on X86. Yellow Dog on a Macintosh. And Solaris – I forget the revision number – on Sun. Then the targets are a subtle point too. There's X86/IE32 family from Intel, and within that there are many variants. The compilers are built for four or five subsets of X86 – 486, 586 and so on, that GCC – the Gnu compiler – supports. Those are all uniquely built. So, when we say we support X86, it's a broad statement. Over on PowerPC, there are even more variants, and they are more fundamentally different than the X86 variants.

I know the 601 and the 604 are highly different.

Yeah, and then you get into the Motorola and IBM embedded controllers, which are great processors, that often don't have any floating point. So it took considerable effort to put together support for five or six different variants of PowerPC so that all the libraries were built correctly. I should add that one of the core values of Monta Vista is building all of this stuff out of common source, and a common rev. We maintain a strict discipline of building all of it every night.

If I look at all the permutations, you've got three hosts and four targets-

-And if you include variants it's more like twelve to fifteen targets.

That comes to 45 different directions that your souce can compile.

That's correct. And if you ever get on the wrong side of that you can see how easy it is to have drifting feature sets and all kinds of conditionalities to account for.

How do you stay on the right side of that?

Software engineering. One of the things that distinguishes MontaVista, if you look at the engineering team, starting with Kevin Morgan, is that it comes out of the enterprise-level, mainstream, state-of-the-art UNIX world. This is a fundamental and deep skill in the company, and it is not one that is shared with others in the Linux world. People who might be very strong in Linux – God bless them – don't necessarily have this kind of background in large-scale software engineering. That's an insight we acquired by getting it wrong. No pride here.

Are there people outside the company who have jumped in to help with the project?

Indirectly, yes. For instance on PowerPC, we have done most of the bring-ups, but not all of them. So the normal processes work there. Which means we are big contributors to the PowerPC tree and help maintain it. But we don't do it all. This is a community effort.

You're positioned as pure open source.

Yes. And we are absolutely an open source advocate, so all this work is always available on the Net immediately through FTP or on CDs.

So if somebody wants to download a .tar ball of what you're doing –

-They've got it. No tricks. No hold-backs, no passwords. It's there. We have a moral obligation here. We've benefited greatly from open source and ought to give back. But beyond that we think it's a business advantage. We fully believe in it. Anything we can do to make Linux – and especially embedded Linux – go farther, faster, benefits us.

You seem to have a strong relationship at Motorola. Do you have a cooperative development relationship with them?

They have been very helpful to us as a company, making sure that reference hardware is available for us. When a new processor comes out, FedEx appears with a board in hand. It's gracious on their part, and we do our best to do the bringups immediately and put them out on the Web.

There is a Motorola microprocessor in the Kerbango radio, running your Linux.

Yes, it happens to be an 823. It's a strong architecture. The IBM 405GP, for example, is also a very hot part, in the figurative sense. Both are PowerPC. We have a bunch of customers designing that part in. So we have a great relationship with IBM Microelectronics, too. We did the Linux bringup on the 405 with them. Right now we're the center of the universe for that.

Are there any embedded multiprocessing applications? I only ask because I was involved with Groupe Bull in the mid-90s, when they had finished their initial design work on the PowerPC, helping design SMP into the chip.

Yes. Certainly in some of the server-type systems where the distinctions between embedded and standard servers are blurred, you'll see some SMP efforts. But it turns out that the same work you need to do to make Linux SMP-compliant is the work that you need to do to improve the hard real time preëmptability of Linux.

So it can be done?

Oh yes. Our guys have done real-time UNIXes so many times that they know exactly what to do. We jumped on the SMP work and found that we could dramatically improve the response, by a factor of thirty. That's just a huge increase in the responsiveness of Linux in the real-time domain, based on enabling, on a single processor, the SMP software, and using it to basically make Linux more responsive. But it's done with standard Linux. That's the holy grail. Since our whole positioning is 100% pure Linux, it's not surprising that we're the guys who did this. Some of the other Linux embedded companies, such as LynuxWorks, have another operating system to sell you. Why would they put the effort in, if it undermines their own OS?

What about Lineo?

Lineo acquired the guys who do RTAI, Zentropics, so they think they've got a real-time solution there. But their motivation to improve Linux is very low, because they have a company trying to work a different way. Red Hat wants to sell you eCos for real time. That's not Linux.

What's with eCos?

It's a kind of VRTX small kernel. They should have done Linux, but they didn't, and now it's too late. They've invested all their money into building eCos and have tried to reposition it as some kind of Linux, but it isn't and everybody sees through that.

This is part of what they picked up from Cygnus, right?

Cygnus did it, and it was funded by Japanese customers who wanted a small kernel. They did an okay job, but it's not Linux. It's a rhetorical patch. A dead end. We're pure Linux, pure open source. We spent the money, hired the guys, and went after making Linux hard real time.

Tell me more about what it is, specifically, that you did to Linux to make it more preëmptive and SMP friendly, and somehow manage to speak to all these different instruction sets.

Good questions. Couple of things. When people think of real time, they think of “well, how fast does it respond to interrupts?” But the weakness in Linux, in terms of classic real time, is not how quickly it responds to interrupts. It actually does a pretty good job at that. The amount of time that the Linux kernel itself keeps interrupts turned off is reasonably small. On a 300 MHz Pentium it's probably 130 microseconds or something. It's not supersmall, but it's very very respectable. So the issue of losing interrupts is relatively trivial. The weakness in Linux actually shows up after an interrupt occurs and you want to have a process run because of that interrupt. The question is, how long should that take? The weakness is that, in standard form, Linux is a non-preemptable kernel. That means some low-priority process might have just started a system call that will take Linux a long time to finish because it does complex things. You're the pilot in the airplane and you push the eject button, and Linux gets the eject command and says “Oh, I'll queue that up, but first I'm going to go finish this low-priority thing – flushing the memory or something.” Hundreds of milliseconds later, you're still waiting for action. So you have relatively long response times for completing the circuit from interrupt to actual process running.

What you want to do is make the operations in the Linux kernel itself interruptable and preemptable, so you can stop internal processing on that low-priority thing and switch to the process level where you get punched out of the airplane. It turns out that the process of breaking up the Linux kernel – so it can leave where it was and come back later – requires identifying sections inside the code where this is possible. This is the same thing that makes running multiple processors possible. It turns out that the same engineering investigation and thought process we use for SMP can be utilized to make the Linux kernel preemptable and very much more responsive. So we're not hacking out the middle of the kernel on this. We're just enabling, on a single processor, the SMP facilities, but not for SMP. And it works like a charm.

How does the rest of the community take that?

We're being very careful here. First, part of the charm of Linux is that you can do whatever you want with it. However, we're being very humble. This is a prototype. As we announced, our source will be on our Web site, which is our tradition. And we are offering this as a starting point for a very good set of capabilities that we are suggesting, in the most humble way, that 2.5 should consider. Linus always says “show me the code.” Well, we're obeying that rule. We think this is a significant improvement in Linux in general, and showing the community the code, for all to see and play with, is the right and only thing to do.

So you're submitting it for peer review.

Absolutely. Out it goes, in the best of the open source tradition.

How do you make your money?

Service. We get technically selected, just like any embedded OS company would be. We sell, on a per-seat basis, subscriptions to Hard Hat Linux, for every engineer on a project, for one year. Like a magazine. This is how the RTOS business has always worked, except for where you assign the perceived value. The customer wants a functional RTOS and the company behind it. Whether the code is free isn't relevant. As an issue, it's naive. Our prices are industry standard – about the same as Wind or anybody else.

So what you're recognizing is that what you were always selling in your earlier companies, was not the binaries, but the relationships, the expertise, the guy on the phone.

Exactly. And the road map. And everything else a customer wants to know is there.

What about royalties?

We think royalties are a very uninspired way of doing business with an engineering team. We just don't do it. By the way, in the business nobody paid royalties anyway. They always did buy-outs or other deals that negotiated royalties away.

So you want to de-complicate life for the engineers.

Engineers like stability. So the subscription model de-couples the rev of your product, which you will do from time to time in any case, from your revenue stream. How many revs happen in two years isn't the issue. Known costs and current software are the issues. Engineers want constancy and good supplier relationships. For classic embedded, with engineering involved, this is a superior way to do business.

You have fewer interrupts in the conversational process between vendor and customer.

Absolutely. Our larger customers beat this into us. In effect, they threw our catalog in the waste basket and said “We want to do business with you, but with a structured, engineer-to-engineer relationship, based on your products, but much deeper than that. We want better visibility, better responsiveness, influence on your roadmap, and a closer relationship all around.” We got more money from those customers rather than less.

Your customers then, are engineers.

We have a B2E or E2E way of working. We sell to engineering teams. That's who we automate and outfit.

I have a friend who likes to substitute the letter W for the number 2 in those X to X business model acronyms. They think the problem with “to” is that it presupposes a distribution rather than a cooperation relationship. We don't do business “to” people. We do it withthem.

That actually is a lot better. I like EWE: engineer with engineer. I could buy that. It's a multifaceted business relationship, based on technology and product. Our customers get technology seamlessly from us, in the best possible form, which is EWE. We answer the phone. They know where we're going. We know where they're going.

What this also says to me is that Linux is a living operating system – a network of “W” relationships. It's organic, and companies like yours are organs in the growing body of expertise about Linux.

Yes. You have to lead and participate and follow along all at once. Given the way that Linux has evolved, without central control, is so remarkable that you've got to believe that there are some forces at play that are working so well that it would be very rash to think the process could be helped along or improved. It's fascinating to be part of, and it's certainly something I would not want to bet against or be on the wrong side of, honestly.

One of the things I've observed about the Net is that it was made by this same phenomenon. It is embodied by its creators with three virtues: nobody owns it, everybody can use it, and anybody can improve it. What you're describing here is more of that same thing, and one way to make a living in it. It's in Lineo's interest to take apart your real time Linux release and put it through that same peer review process.

Yep. You're absolutely right about the Internet, and about Linux. This thing is spreading like wildfire, and its in our interest – along with everybody else's – for us to pour gasoline on it. These things have very explosive properties. And we have to contribute to the ongoing explosion. Michael Tiemann at Red Hat tells the truth when he says there is a proven model here and it's the scientific method. You do your experiments, show your results and so on.

Eric Raymnond talks about that too.

Yes. In this unruly and wild Linux community there is something that propogates very quickly through companies like MontaVista that make it possible from a business standpoint to depend upon it. It's a force. We have to be careful not to act as a filter in front of all of this, but rather to act as an enabler, so people can tap in to this phenomenon in a sane way and still get their job done. That's our purpose in life. We want to enable the engineers to do what they want to do anyway, which is use Linux. Boy, that's a deep phenomenon.

Are you saying Linux is ubiquitous with your customers?

We go into these places and virtually everybody we talk has fooled with Linux at home, at the very least. All we have to do is give them the justification to use Linux at work and you've got all these guys working on your behalf.

One of the things I observed about the IBM announcement, when they essentially declared themselves a Linux company, was that this was clearly a company that was now in full compliance with its own engineers.

Oh, there is no question that this is a phenomenon of software engineers. Look at Sun's success in the early years. There is no question that they won the hearts and minds of software engineers. If you were an engineer sitting there at a VT100 with VMS at the other end of it, and the guy down the hall had a Sun workstation with a big monitor and a mouse and multiple windows open and running UNIX, you will do everything in your power to get one of those things.

I like your way of characterizing Linux as a fire you pour gasoline on. For years I liked to talk about the only kind of truly effective marketinn. The logic went like this: A) Markets are conversations; B) Conversations are fire; and therefore C) Marketing is arson. What you're saying is that if you're not committing arson, you're not really committed.

That's not a bad analogy at all. With good old VRTX, we were trying to practice marketing arson, trying to set fire to the world, and we did okay. But they were just our efforts alone. If we scaled it up a bit, we had the RTOS industry, alone, trying to set that fire. It's a respectable business. In fairness, Wind is a $300 million company through acquisition and everything. But it has taken forever to get there.

Now, however, we're part of a larger phenomenon called Linux, which is many, many conversations going on all at once. It is a wildfire. That's why it's so fundamentally different. It's instantaneously worldwide. It's very deep. It occupies the minds of countless engineers, naturally selecting the best and the brightest. It is all of these things. It's a whole lot of fun, I can tell you that.

Will you do acquisitions?

Sure. That's a real possibility. But I have been through two acquisitions, and I can tell you that in the best of times they are very hard to do. You get what you get, which includes good guys and bad guys, old agendas, different code bases. Lineo's acquisition of six companies has to be extremely difficult. We've grown to 100 people organically. And it shows in results. We're producing software.

Are you making money?

No. We're a start-up, and we're just trying to grow. This is a land grab. We have customers. We are doing well. But clearly when you establish European and Japanese offices, you've got a lot to eat while you ramp up. Investors don't put the money out to have it sit in the bank. They want it spent.

What does your plan show?

A turn to profitability at a future point. And we are exceeding our plan.

How many customers do you have?

Fifty.

So you have EWE going with fifty customers.

They aren't contracts. They're orders. Customers issue purchase orders. They buy subscriptions. We do consult, by which I mean that customers buy NRE (non recurring engineering). But if you were to think about embedded systems, you'd think hmm… telephony equipment, internet infrastructure equipment, etc. That's who we have. Companies who make gizmos.

Do you think one industry will turn out to be a more significant embedded Linux category than the rest?

We see three categories. One is what we call A-to-Z embedded. That's embedded controllers and instrumentation and gadgets of all kinds. But the driver of microprocessor technology is what we call “either end of the Internet.” The infrastructure equipment guys – the Ciscos and Alcatels and Nortels and fiber optic people – those are a very strong component of our business. Then the third cagegory is things like the Kerbango radio: things that attach. Appliances and all that. God bless Linux, it looks real good all over the place. You'd think, given the diversity here, that one kernel would have a hard time stretching over all these different applications and situations, but Linux is doing it. There's a communications revolution going on. The Internet is part of that and Linux is part of that, and both are driving requirements that are falling out in to general purpose embedded. It's very interesting: the two earliest customers were one infrastructure company and one appliance company: Kerbango. Both drove us to make Linux better in the same fundamental ways.

---

Bryan Sparks, Lineo

What got you guys into this business?

Two ways. The first is that we started Caldera back in '94. Back then we bought some business from Novell and started selling DR-DOS. We intended to use it as a thin client for cash registers and applications like that.

You own the original DR-DOS, then.

Caldera bought that in '96. We had a lot of interest in embedded, and DOS looked interesting to us for that. There was a need in single-board computers and non-PC-like devices.We ended up finding kind of a business there, and that's where I ended up betting my career. We spun off two companies. Caldera systems, which you know, then Lineo, then we collapsed Caldera, so there are just the two companies. It was confusing. For a while Lineo went by the awkward name of Caldera Thin Clients. We did some DOS and Linux stuff, but got more and more interest in Linux. Clearly there was something going on here. So we said “let's turn the ship here.” Which we did. You won't see anything about DR-DOS on our Web site. We just don't do it anymore. Our whole business is embedded Linux.

What was the time frame here?

This was around late '98. That's when we said this is it, and picked a new name and started moving forward seriously.

Good timing.

A lot of it was luck, and being in the right place at the right time, and having revenues from the DOS business to fuel the early stages of our embedded Linux product.

You were the first out of the gate with Linux distributions too, in a way.

We were.

And you got kinda snookered by Red Hat. I take it you're not going to let that happen again.

It's not gonna happen here. I learned a tremendous amount. It was very humbling. We were the leader. I made a lot of mistakes. We got passed in the last lap and somebody else took the checkered flag. Pick your metaphor. Boy, that's not going to happen here.

What exactly did you learn?

One was that we did not have a culture that pointed at a competitor. We were leading in the space and not looking out for our back, where everybody else's guns were pointed.

You had the pioneer's problem.

There are two strategies, of course. One says that first movers always win. The other says second movers win because they take advantage of the first mover's mistakes. The second is what happened here.

Happened with DOS, too. DR-DOS was first, but MS-DOS won that race.

Here I feel like this is my second chance. And it's not just me. There are a lot of second-generation Linux people here. People from Linuxcare, even Red Hat. They've been there, done it the first time, and made mistakes we can all learn from.

And you're applying what you learned.

Absolutely. I feel that we are executing about as well as any company can. I am so proud of the people we have and the job that they're doing. Ultimately that's what wins. You can have a great pitch – our pitch is okay – but you have to execute. You need top-line growth every quarter. You need good customers who come back because you do what you say you're going to do. We have that. It's real simple stuff.

I'd like to look at the embedded space we've had for twenty or so years, and how Linux is changing that.

It's amazing. We're growing and assimilating employees as fast as we can, yet the demand placed on us by semiconductor manufacturers, and their customers, still overwhelms us. So that tells me that, well, maybe Linux hasn't won, but it's certainly in extreme demand.

So it's still early.

Yes, and you have to look at Wind River. If you saw their last quarter's revenue… holy flippin' mackerel. They made a ton of money.

How many people do you have now?

Two hundred sixty. About half are here in Utah. The rest are outside.

A bunch would be in your acquisitions.

We did six acquisitions this year. And we have grown staff at most of those acquisitions as well. And we don't run them as separate companies. We run them as part of our company. But the offices represented by those aquisitions have grown. We bought technology, some of which was in embryo. We had to kind of beef up that technology, but in every case the opportunity is just huge.

And you've attracted a lot of attention.

Oh yes. You'd expect me to be optimistic, but I think: gosh, if one tenth of what comes to our door pans out, we're in great shape. Linux is going to make huge inroads in this space.

Tell me more about the demand from the semiconductor guys. Does it sort out to their customers categories? Or is it something that comes out of your working relationships? Or both?

We are extremely close to our customers. Our strategy from the start has been to establish the strongest possible relationships with semiconductor manufacturers, so we get into their roadmap, and target what would be most interesting for their customers to run Linux on.

How do you adapt to the verticals?

We verticalize some of our solutions. We're already doing that internally. But the main idea is to provide ubiquity on embedded platforms.

Which platforms in particular?

SH 3,4, StrongARM, ARM in several variations, almost all the Motorola chips – MCORE, ColdFire, Dragonball, PowerPC… Of course the X86 kind of comes for free. That comes to ten, I believe.

So your horizontal product is Embedix.

Embedix provides a whole list of services. Not only the Linux stuff, but hard real-time. Jim Ready and others do soft real time, but we do hard real time. Tymsys does too; but we do hard real time with extensions. We also do soft real time if people want that. We have a browser that we're converting into a more generic graphical user interface with cascading style sheets, HTML, Javascript and a whole bunch of other stuff that sits on top. This all works horizontally. So as we start verticalizing some of our solutions we'll get stronger offerings in those areas. We already have design wins in routers. DSL, VPN, IP security stuff, plus verticals around some of the devices we're working on.

You say you work closely with your customers, which are primarily the semiconductor companies. Tell me more about how that works.

We are literally part of their team. Their wins are our wins. Their customers are our customers. We want direct relaitionships with their customers, the OEM device manufacturers.

So if Motorola sells to GM or Ford, you want to be in there with them, so those companies are your customers too.

Yes. And this is the relationship we have with all the semiconductor manufacturers.

Who's showing the most interest in Linux?

It's across the board. But there are some surprises. If you asked me a year ago, I would have told you the sweet spot for Linux was Internet appliances, routers, gateways…

Stuff close to the Net.

Right. Stuff that takes advantage of the core competencies of the Linux communities. But the areas that have been surprising for us – and we're pretty much the only ones who do this – have been in consumer electronics. I'm talking about Dragonball, MCORE ColdFire, ARM and those devices, which run in digital cameras, cell phones, handheld devices and other consumables. We are working with a lot of companies in those areas. We can really run Linux on those devices. You don't have an MMU? You constrained by 200k of RAM? We can fit in that. The interest is extremely high in that space. When you consider that our competitors in that space – CE and Palm – lead with a GUI, we're looking good with Linux, which doesn't have an inherent GUI. So we manufacture one with a browser, Java, whatever.

But the demand is for Linux in this segment.

Exactly. It starts with customers wanting to run Linux down at this tiny scale. It's fascinating.

What about process control, industrial automation and stuff like that?

It's a market, but it's not growing, and it doesn't turn over very fast. For those spaces we offer VXworks and Psoft migration toolkits, to make the switching costs easy. Our hard real time plays real well there, too. In fact, we're doing big contracts with the government on that kind of thing. We're doing a big contract with NASA for flight simulators, weather monitoring for the FAA… things like that, through one of our acquisitions. But these aren't the surprises. The big surprises are at the bottom of the pyramid where the parts are small, the demand is high and the volumes are enormous. Things you see everywhere. So obviously those are big for us, and I think they're big for Linux too.

What's your key offering there?

An interesting road map. When you buy into Linux, you buy into a fast moving technological roadmap. We augment that with technologies we either acquire or build to make more interesting to those verticals. It's a fascinating thing. Previously they were either buying from Palm or CE, which are proprietary RTOSes with no road map at all.

What is your position among your competitors, in respect to customers.

I can't imagine anybody talking to more customers and partners than we are, just because we're large. We get a lot of feedback on what customers are asking for. Right now we're busy just backfilling customer requests on a road map. We have to get ahead of that. We need to lead in embedded Linux, from the customer's perspective. Right now we're just satisfying customer requests. That kind of drives our road map. But eventually we'll have to do more R as well as D. I think we'll get to the R in six months or so when we get ahead of the curve a bit.

I would imagine that the D kind of pulls the R along.

The customers are telling us “we need this,” and we have a choice to make. Is it on our road map? No? What do we do? Build? Buy? License?

What's an example of that?

Bluetooth – the hundred-meter wireless Ethernet speed system. The question was, do we build a bluetooth stack? Because we were thinking about building one. Then out of nowhere IBM announced an open source Bluetooth stack. Thank you very much.

How do you contrast what you're doing with what MontaVista is doing?

First, we're not a pure open source play. We license third party products with IP – intellectual property. We have our own IP. We don't open source our browser. It's really good, and we don't see any reason to open source it. The benefits don't outweigh the liabilities. But I guess the main differentiation is that we lead with product and backfill with service. We are highly motivated to build a Wind River-like tool chain.

What you're about, really, is tools.

You will see an ever-increasing series of tool chain enhancements coming out of Lineo. A bigger and bigger SDK.

So your orientation to open source is to cherry-pick those products and tools that are useful to you and building a suite of products, primarily tools, for shaping Linux into something useful for your customers.

Those tools we build. But what I'm talking about are design automation tools. Graphic configurators. Things that run on the device. Embedded apps. Our question constantly is, “How can we make the time-to market for our customers as short as possible?” To the degree that we're successful there, we will steal business away from Wind River.

It sounds to me like you're building a Cisco.

We're not doing hardware.

I was thinking of the kinds of customers you're working with, your approach to acquisitions, your determination to lead the market no matter what, your customer focus. I'm amazed, for example, at Cisco's ability to acquire companies and not have trouble swallowing them.

Right. You can imagine that we had a strategy for acquiring companies while we were private. We wanted to learn how to integrate. And you know what? Buying companies is hard. Signing the document is easy. Integrating personalities, cultures, ownership responsibilities, technologies, all those grungy things, is hard. We learned a lot. We didn't lose anybody in the process, but it was a lot harder than I thought. It took us two months more than I thought it would take.

It seems to me you went about your acquisition strategy based on target device size scale, from small to large.

That's exactly right. We said “We want to do that. How do we get there?” We had holes, just working with a generic embedded OS. We didn't have anybody at Lineo with real time experience. We needed to go acquire that expertise.

How do you integrate Linux and your hard real-time capability. Do you have a name for that?

It's folded into our Embedix product, and it's a feature of our tool chain. You want real time on this device? Here's how you select and configure it.

So when an engineer goes about designing for a particular processor, application and board, and real time is one of the things he wants, he implements it through Embedix tools. But it's not Linux, necessarily.

Yes. But that, by the way, we open source. That whole thing is available to everybody. We followed the RTAI – real time application interface – standard. We open sourced all of that. You'll be seeing more from us there. Actually we were ahead in real time for a while, but let others move in and define the market.

Who?

Jim (Ready). He talks about it all the time. And he doesn't even do hard real time.

They announced their preëmptive kernel yesterday, which they're vetting with the Linux development commmunity.

Yeah, he's kind of defining that space, but we're going to fix that. That's why we acquired Xentropics, because they're the ones who did our hard real time.

Where do you see Linux moving the embedded space?

I think by offering Linux on embedded devices we raise the bar on functionality that customers will now require. There is this dichotomy in the world where a lot of companies say they are customer driven, but in fact customers only ask for what they are aware of. They don't know their necessities until they become aware of what's really available. The context changes. Adding Linux to Internet devices of all types will seriously raise the bar in functionality that people will demand. An example. In our first release of Embedix early this year, we matched VXworks in functionality. In our fist release.

I'd like to go back to the power of invention. The real marketing principle isn't “necessity is the mother of invention,” but “invention is the mother of necessity.”

That's right. We want to show customers what they're missing out on. Linux does that on a huge scale.

It also seems to me that Linux has the same network effect that the Net has had, only starting a bit later. It's kind of the other shoe dropping. Same phenomenon, just a later stage.

Our mission statement is, “we unleash the imagination of our customers to innovate for their customers.” That means we want our customers to innnovate in ways they didn't even think they could before.

So your inventions lower the thresholds of their inventions.

Yes. There's another thing. We are going to provide a flexibility that these guys have never seen before. Before Linux came along, and a customer needed a feature, they went to Wind River and eighteen months later they might get it. The whole thing is way too stymied. Between Linux and our SDK we're really going to unleash the imagination of our customers. And they will only demand more and more features, which we will be in the best position to provide. We want their time to market requirements set the pace. Not ours.

You mentioned doing this surprising work with small consumable devices. Are you working with some of the Asian consumer electronics guys?

Nearly all of them. We have fifty people in Asia. A large presence in Japan, plus another fifteen or so in Taiwan. Most of the investments we did in our round of financing came from Asain manufacturers. Acer, Samsung, Mitsubishi, a bunch of others. And we are working with a bunch of people. I go to Asia a lot. Fortunately we're kind of all alone over there. It's hard to get in. We got in partly by acquiring a company in Japan that was doing embedded custom engineering. Then we organically grew a Taiwanese office with native personnel.

I'm interested in scale. A huge company like Hitachi might have a huge relationship with a customer like Sega. How is there room for guys like you in there? I guess there was room for Wind River, though.

I don't think our business model is all that different from Wind River's. They, being such a dominant supplier, were able to extract more money from the semiconductor manufacturers than we can; but that's part of our strategy – to commoditize the space a little bit. When they talk to investors, they talk about raising per-unit value. When I talk to investors, I say we're here to commoditize system software for interenet devices. The model is the same, but we're really putting price pressure on them. We not only give them real competition, but make it much harder by coming in at a lower price.

You still have enough room in there to make money on tools and services.

We do request a small royalty, adding our own third party intellectural property on the device. But again our strategy is customer acquisition through commoditization of the space, sell tools, and grow. But in terms of what we sell, we're similar to Wind River. It's just that a smaller percentage comes from royalties.

You charge on a per-workstation basis for the SDK.

Yes, and we have royalty-bearing components in there.

You pass those costs through.

That's right. And that's very standard for the industry. We may do more licensing. We may do more acquisitions in the area so we can keep the price low but still provide the functionality so we don't have to carry the royalties back.

How big do you expect this category to get?

There are a bunch of analysts who say big, big, big. We want to grow faster than the market as well. Just keeping pace with the growth of the pie, we're at a rate of 60%

Do you see The Smart Home finally coming, because of Linux's standardization properties? Or will the firewalls of ignorance that surrounds every specialty going to still prevent that from happening?

I don't know. I brought a bunch of CAT-5 cable to the drop point at my house and the phone guy made such a hairball of it that I had to redo it myself. But some of the standards are helping a bit. HPNA, a home networking protocol that runs over twisted pair, helps. Bluetooth helps. We're working with a company we're hoping to close that is doing stuff with home gateways like I never imagined. We have a TV set top solution that we've sold quite a bit. But this is something else. We're talking about something wired to appliances that does far more than turn them on and off. It does service and warranty control. Your AC unit can say its compressor is going and to send for a guy to avoid the insurance hit. That kind of stuff. I'm crossing my fingers that we get that one closed because it's just so cool.

That's part of the promise of XML. Devices can talk to devices, live, in an XML stream.

Can you imagine this little control box calling a technician without the customer knowing what's happening until the technician says “I'm here to fix your fridge.” This isn't ten years out. It's next year. We giggled a couple years ago that somebody in Japan put a browser on a fridge. Now, so what? We can put one on the microwave. There's your recipe. Your documentation. There was a piece out of Xerox PARC a long time ago that was right on. We are living that now. And Linux will only help there. Great internet infrastructure. Remote diagnostics. Network management. VPNs built in. If you're a telecommuter, Linux is great. We actually sell a little hardware box that's a VPN router. We sell it like candy. It's unbelievable.

You sell a lot of set top boxes?

Yeah. You can browse the Web on these things, watch enhanced TV, get your email, or whatever. We sell a boatload of that.

What else do you open source besides your real time component?

Go to www.opensource.lineo.com. There's quite a bit there. Utilities that run on an embedded device, like BusyBox. A bunch of drivers that we think would be interesting to others. We have a pile of open source projects.

Are your fingerprints going back on the Linux kernel?

Oh yeah. We give a lot of patches back. We want them to take more. But sometimes the patches we have, to fit on very small devices, aren't very mainstream.

What has Linus done for the embedded Linux space, by talking up mobile and other forms of development, and working with Transmeta?

I think he's done a tremendous amount. He's helped Transmeta a great deal too, giving them branding and recognition. And he really seems to enjoy it there.

Last question: What kinds of companies will survive and what kinds will not?

I think there will be tremendous revenue opportunities in this space. I cannot imagine that we'll be the only ones succeeding here. Lots of new companies seem to be rushing in, but it will be tough for a lot of those. We think we have found an interesting model that investors and investment bankers like, and we'll see if we can keep executing on it. But we won't be alone. And don't count out the larger companies – Red Hat and others – will make some moves.

Any last thing you'd want to add?

You'll like this one. The cluetrain stopped at Lineo, and by God, we got on.

---

About the author: Doc Searls ([email protected]) is senior editor of Linux Journal and coauthor of The Cluetrain Manifesto.

---

Copyright © 2001 Specialized Systems Consultants, Inc. All rights reserved. Embedded Linux Journal Online is a cooperative project of Embedded Linux Journal and LinuxDevices.com.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.