Recent events haven't been kind to the embedded world. After the bursting of the dot-com and Linux bubbles, many embedded Linux companies, prematurely expanded to nonprofitable levels by the prior year's venture capital investment excesses, were especially hard hit and went through multiple rounds of layoffs.

Earlier this week (April 24, 2002), I stood on the steps of a Utah courthouse with some former Lineo employees and an assortment of attorneys, and watched as the assets of Lineo, the first major embedded Linux company and my home for over three years, were sold off — part of a process called recapitalization. Only time will tell whether “Lineo v2” can successfully be nursed back to good health.

It was easy to feel a bit forlorn. But then an old saying popped into my head: “Don't cry because it's over — smile because it happened.” With that, I resolved to pass along some ideas to those who still have a chance to live the dream, and who have the vision to consider what they're doing right, or might do better.

While purists might argue otherwise, the simple truth is that Linux is not “free”. Whether the associated costs are incurred by hiring a team of Linux-savvy engineers, or in training existing engineers, or in hiring a company to handle the load, the undeniable fact is: anything of value, costs. And Linux is certainly valuable.

Several embedded Linux companies have struggled with that point, with varying degrees of success. There are certainly multiple opinions, differing approaches, and numerous opportunities. Hopefully, the outline provided here will help you decide how you can “succeed with embedded Linux”.

Succeeding with embedded Linux

One question you might hear is “How do you make money on free software?”. If you do, help the questioner on the basics of business, and if you fail, run! The answer, however, is simple and straightforward:

• by concentrating on customer needs
  
• by adding significant value not easily duplicated
  
• by adding significant value in addition to the OS
  
• by being willing to spend time to educate the customer on the difference between these points.
  
• avoid the temptation to ask for a hundred “free” boxes, since the sand to make the chips was free, the water to process the plastics was free, etc.

Case in point: I spoke with the development lead of a major company a while back about their Linux-based projects, just getting underway. He was in the process of hiring “at least 50” Linux engineers, and the managers and support personnel for that major undertaking. I was interested — that's a massive ramp up — and asked what he hoped to accomplish. His first answer was the most telling: “I refuse to pay royalties or any other fees just to use a free operating system”. I asked him how many units he intended to ship; he replied 1-2 million within 2 years. I asked where the engineers were being hired; “California”, was the quick reply. In a sincere voice, I said to him: “I don't get it. You'll lose 6-12 months of market opportunity, while you ramp up a team. You'll carry that burden for years to come. You'll gain nothing that hasn't already been accomplished by our company, or, frankly, by several of our competitors. And you'll spend at least $5 million a year to do it, in other words, at least two dollars a unit, per year, every year, instead of just once. Whether through royalty, buyout, or other license, you could get to market faster, have more market opportunity, probably a smaller and better system, at a once-per-unit cost of perhaps 50 cents a copy, with maintenance, support, upgrades, and license issues resolved for you. I just don't see the ROI.”

Surprisingly, he agreed with every point! “I just refuse to pay for free software”, he said again. We talked a while, and I saw him head to the MontaVista booth, where I'm quite certain (from the look on their faces) that he gave them the same story. His story checked out. This is a major company, and I knew some people there (in other product areas). Less than a year later, his project was in deep trouble; he never did hire all 50 people, but layoffs started, project delays mounted, morale was down, and he had already paid well over 10 times too much. Eventually, the project was canceled — a great example of why royalties aren't always so bad! Management ended up viewing it as a failure of Linux and the open source model — ya gotta have a scapegoat! Last I heard, he was promoted ;-)

Some simple guidelines

My guidelines for success in embedded Linux? Easy:

• Know your customer, and understand their markets
  
• Know the technology. Hire good engineers. Empower them.
  
• Hire knowledgeable sales people.
  
• Add value through knowledge; charge for it.
  
• Add value through features, preferably your own.
  
• When necessary, add value with partners.
  
• Be prepared to act as integrator, and charge for it.
  
• Use open source wisely — and aggressively.
  
• Don't get greedy.

Obvious, you say? Think again. I've seen company after company blow it on these points, again and again. Why do you think embedded Linux is on the upswing, other than that proprietary RTOS vendors have themselves lost the battle on these messages? Let's look at these in a bit more detail.

Know your customer, know the technology, hire good engineers, and empower them: Embedded systems require knowledge and expertise that is different from a product-oriented, one-size-fits-all model. Your customers will respect you and your company if you can bring forward personnel (engineers, managers, and sales people alike) who understand the products and markets that they themselves are trying to win.

If you decide to pursue vertical markets such as automotive devices, industrial control, or telecom, smart sales people will be a tremendous asset. To capitalize on such a strategy, you must be prepared to hire personnel who are knowledgeable in that area, and then “fill-in” around them with an empowered team that can react to customer needs and market changes. One of the most successful approaches is to make use of cross-functional “tiger teams” comprised of market-savvy personnel — generally including a lead engineer, business development manager, and marketing lead — who set their own goals, control their own personnel and budgets, but are accountable to upper management. Be sure to give these teams both the responsibility and authority to get their jobs done.

Next, add value through knowledge, features, or partners when needed; charge for that added value. This is another one of those oh-so-obvious points that a surprising number of people just don't get. Despite hundreds of contacts over the past few years, I cannot recall a single company that just wants to build a portable device with Linux on it! None. Every single company, every single tinkerer and engineer that I've met, wants features. “Obvious”, you say again? No, it's not. And unless you have the people to either assist the customer in those areas, or the add-ons they want today, or the partners that can help them get products to market, you're not going to be much more than the kid who pumps the gas — a task that almost everybody does themselves these days. No wonder companies such as the one I referred to before, feel the need to hire their own “kids”. Was it possible that the senior manager really was hiring people just to avoid paying fees? Unlikely. It's more likely the manager felt “the company's” team could add value for their products, and that getting Linux into the box wasn't much different from pumping gas. What wasn't realized was that it's sometimes closer to pumping liquid oxygen through a straw!

To the embedded Linux wannabe, value-added features and functions represent a sustainable, multiplicative revenue stream. That's the key to a healthy business. It doesn't have to be a gouging business — nobody appreciates that. But if you can help a company build a better product, faster, and if they can make extra income because of better quality and being first to market, then you've done everybody a favor: them, you, and the end user of your customer's product.

I've heard a few people (who obviously weren't trying to make a living) say that the only valid way to make money with Linux is to hire themselves out on an hourly rate, and eke out a meager living (while their customers laugh to the bank). A “services-only” model is not scalable. For example, if you have 4 engineers and one manager in a team, and that's the whole company, then no big deal. You make a little money, you worry about the future, and you hope like hell that “the other guys” like you don't go out and shoot holes in the bottom of the boat, to pick up the same business you sweated long and hard to get. As the company grows, you still need those same teams, but now you need managers just to manage the teams, and managers to manage those managers, resulting in constant degradation of the per-person income. Conversely, you have to raise your prices just to break even. With a multiplicative model, you have material you can make money from, and your teams can focus on the value-added integration efforts and market knowledge that they can bring to bear. Smart companies have always been willing to pay for those kinds of services, and advantages. Those are the companies you want as customers.

Don't be afraid to capitalize on areas that open source simply does not address, such as product integration, testing, and certification. Especially for embedded devices, the opportunity often doesn't exist to “fix it later”. The test and certification costs can be large, even staggering. Would you want your Linux-based anti-lock brake system controller to go untested, even if it is running on Linux? Of course not! OK, now how many people do you know who are willing to spend the time and money to build, test, and certify such a device, at no cost? Ahh, you're starting to understand.

You can't build everything yourself. Find and work with good partners. Learn about their offerings, and teach them about yours. You'll effectively double (or more) your market and sales presence, and your ability to deliver new, effective, and differentiating services. Then, be prepared to act as the integrator. Chances are, your customer doesn't know much about Linux, about operating systems in general, or about the efforts required to use them with the components offered by you, or your partners.

Use open source code wisely. What does that mean? In the first place, it means recognize when open source is the best solution, but be willing to recognize that open source doesn't work for everything. There are situations where there just isn't enough community interest to sustain a quality project. This is another one of those simple points that zealots don't quite understand. There are people and companies who will not work for free, and will not release their hard-won (or otherwise acquired) knowledge. If you choose to ignore them, they'll go elsewhere, maybe even to a proprietary OS. You lose, they lose, and the consumer loses (due to losing the flexibility and standards-conforming benefits of Linux, if nothing else). Ready, aim, shoot-off-yer-toes.

Use open source aggressively. Though some may disagree with this next tactic, it is both good business and good open source policy. It is: as soon as you see a competitor offer something of substantially the same type and quality as what you've been charging for, open source your code. This has the effect of “increasing the open source gene pool” — and frankly, it forces your competitors to leave you alone. If they spend a lot of time and money to come out with Revision1, and you already have your market-proven and mature Revision2 of the equivalent function, then they've spent a lot of time and money, and they're not going to have much chance to recoup those expenses. Although it sounds cruel, it actually has positive side effects: it forces companies to look at markets with an eye to understanding them, and enhancing them, instead of playing “me too” games.

Combine that model with one of consistent and fair pricing that recognizes that you've created a solution that, perhaps, several companies will help pay for, and you quickly see how you can underprice, but outperform, competitors. In other words: don't get greedy! If you spend $50,000 developing a package, don't charge each potential customer $50,000 — but don't give away the store either, and recognize that they would have to spend that same amount of money if your solution was not available. Should it be $10,000 spread across 10 customers, or $25,000 spread across two or three? Only your business opportunities can answer that question. It doesn't take much thinking to realize that if you offer that product to, say, two companies at $40,000 each, you have saved them weeks or months of work plus $10,000 (25%) each, but you've made more than it cost you to develop the package. Of course, if you try to charge that same $40,000 to 500 companies, then one of your competitors will surely see a great opportunity. With the former pricing, your chances for a successful business are improved, but your competitors find their own opportunities; and let's face it, if you go out of business you aren't doing yourself or your customers any kind of favor.

The growing role of embedded Linux

Embedded Linux does have an increasingly prominent role to play in a variety of emerging markets. The combination of a robust OS that is “free of encumbrances”, available at potentially low cost, and that honors international standards (without an “embrace, extend, extinguish” mentality), simply cannot be ignored, matched, or beaten.

Some will shed a tear for the passing of companies that don't find that “sweet spot”. If instead, you learn from past mistakes and fine tune your approach, your skills, and your model, you may indeed one day be able to boast that you work for a successful embedded Linux company.

And you'll smile.

---

About the author: John Drabik is the former VP and CTO for Digital Media at Lineo, and was the architect and Chief Engineer for their Residential Gateway plans. He joined Lineo before the wild explosion of “all things Linux”, and lived through the bursting bubble that came after. A strong believer in open standards and open source, balanced with the opportunity to grow a business, John is also one of those long-hair hippie freaks who likes Bach, Buxtehude, and Beethoven — and some Beatles thrown in for good “measure”.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.