CEO Interview: Victor Yodaiken, FSMLabs
Dec 16, 2003 — by LinuxDevices Staff — from the LinuxDevices Archive — 13 views
This interview with FSMLabs CEO Victor Yodaiken is the first in LinuxDevices.com's new “CEO/CTO Interview Series.” Yodaiken addresses general trends in the embedded market, as well as specific projects his company is involved with. Topics include robots that play catch, Tom Cruise's horse, jet engine construction, software patents, the international embedded Linux picture, BSD, development tools, the GPL, RTAI and ADEOS, Linux 2.6, and more. Enjoy . . . !
Victor Yodaiken, CEO, FSMLabs
Q: How has the year been for FSMLabs?
A: Fun. We've got some really great new technology out there now with a couple of surprises in the pipeline, our customers amazed us with some incredible applications, and the business is doing very well - another profitable year with revenues doubling.
Q: What's new technically?
A: The “wow” stuff is all in the applications. When people ask me what I do and I show them some code to minimize interrupt latency, their eyes glaze, but if we show them what Steve Rosenbluth does with RTLinux - making Tom Cruise's fake horse look ridiculously lifelike in “The Last Samurai,” or what Dean Annesser's group at Pratt&Whitney do with RTLinux for jet engine construction, there is a different reaction. The goals we have for the operating system kernel are to make it as boring as possible for users - it should just work out of the box and perform with no surprises. We think of ourselves as making something like structural beams. The glamor is in the apps. This year applications have included everything from low cost rockets to robots that can catch balls, a massive data storage system, a high end smart real-time router, UAV's (Unmanned Aerial Vehicles), wireless radio controllers for machinery, and automated warehouses.
One of my favorites is the engine manufacturing test system that the folks at Pratt&Whitney have based on RTLinuxPro. They are building on years of experience automating this really complex process and what they've done is create a single system that goes from pure simulation to production test. The engines they design are bleeding edge and contain many components that have to work together reliably under enormous stress. So the ability to start plugging components into a simulation as they arrive and keep adding components until you have a completed device is critical. And they are really pushing capabilities of RTLinux. They have a huge networked system where much of the application runs in protected memory, real-time 1394b Firewire connects components, and the Linux base is pushed hard too.
There are also very cool robotics work in Japan - I got to see one really advanced RTLinux based robot in Japan last month and Yuichiro Mori who runs FSMLabs Japan sent us some videos of RTLinux based robots that were shockingly quick and agile. I hope some of these will be released in Q1 of 2004.
Q: What's coming up in RTLinux Pro itself?
A: We've got two or maybe even three IDE's that will all be available next year. There is a very usable low-cost but solid system we will start including with subscriptions. There is a pretty advanced one under test developed by one partner company. And, we will start delivering CodeWarrior from Metrowerks as soon as packaging is complete in the next few months.
We've also got new networking, a couple of new processor ports, vastly improved trace and profile capability, and some continuing design simplification to improve basic reliability. We put in a lot of time on quality assurance and finding corner cases in timing. Next year, I hope to have some fault tolerance and security components out too. And I think people are going to take some time to realize how cool the new programming environment is. We're making the real-time programming look more and more like ordinary application programming and less and less like kernel programming.
Q:What about RTCoreBSD - what's that about?
A: We have customers who prefer BSD to Linux - some for historical reasons, some because it is easier to make it very lightweight, some prefer the technology. Some people are worried about the SCO lawsuit or about GPL issues and prefer the BSD license. Traditionally networking companies like BSD, and that's where our BSD sales have been so far. The RTCore kernel is the same for both Linux and BSD, and we have both FreeBSD and NetBSD supported.
Q: Speaking of GPL, is the GPL version of RTLinux still being developed?
A: Yes, very much so. The real center of work is at the University of Valencia in Spain and at other universities and companies in the European Union OCERA project. Over the last year they have produced a lot of innovative work - everything from POSIX timers to a standalone version of RTCore - without a Linux or BSD operating system. I'm trying to collect some material now for a 3.2 release that will contain some of the OCERA work and MatLab support and a few other new projects too.
Of course, most of the new work with RTLinux GPL is in the applications and we've seen a huge number of new applications recently. Linux has attracted a lot of developers who think that OS internals are fascinating, but RTLinux attracts developers who are more interested in robotics or machine tools or data acquisition. In some ways this is because RTLinux GPL is a pretty stable system - it does not need a lot of new development or tweaking for popular applications.
Q: What else is going on in FSMLabs business?
A: This year saw a lot of activity in India and, just in the last few months a rapid expansion of business in China. We should do a third of our revenue from Asia next year. We have a new partnership with Metrowerks that is picking up that could be a X factor. In the US we had a serious deal to supply a big networking company, and have a couple of customer products about to come to market, and we are selling software into a lot of new areas. 2004 will be interesting because more RTLinux based products are going into production.
Q: So what's the “business model”?
A: We use the same business model as IBM and Intel do for their Linux related business, although at a much smaller scale. We sell products, our RTCore software and related components, and we share in Linux and BSD open source development to make those products more valuable to customers. FSMLabs has no venture capital, we rely only on sales of software, IP, and services, and we are very much of an engineering-driven company. We don't know enough about marketing to get an edge that way, so we have to develop products that are obviously valuable to customers. This is the old HP theory.
Q: What about the “sell support give away software” model?
A: We didn't know how to make that work in our market. Maybe as we become better known and “build the brand,” as they say, we can rely on that more.
Q: RTAI is often claimed to be the “real” open source version of real-time Linux. Any comments?
A: That claim is disrespectful of the thousands of RTLinux GPL developers out there. They tend to be a pretty quiet bunch for some reason - maybe they are, like us, more engineers and scientists than marketers. If you go on Google and look for RTLinux projects, you can see that it's a very healthy open source community.
The problem with RTAI is that I don't see any way to consider it in compliance with the GPL. The COPYING file in RTAI even comes out and admits that parts are derived from GPL code but then goes on to give people permission to treat it as if it were LGPL. The GPL does not allow such permissions with other people's GPL code or with derived works. And Michael Barabanov and my copyrights are gone.
I looked in the RTAI March 2003 release and found a file called rtai_sched.c that is really similar to rtl_sched.c in an old version of RTLinux down to using an identical magic number, the same first few elements on the task structure, the same basic weird scheduling loop that Michael used to find the pre-empting thread. This is the core of the RTOS, not some isolated feature. Anyone can go find the history of RTAI from Mantegazza's earlier “myrtlinux” if they spend ten minutes on a search engine. Clearly some of the developers for RTAI are real open source developers, but until they fix the license and meet the GPL terms, I don't see how it can be called an open source project.
I should add - I am not a lawyer and nobody should consider any of this legal advice.
Q: What about ADEOS, which is claimed to not use the patented RTLinux method?
A: I downloaded ADEOS, saw that it was GPL licensed and didn't look at it any further. GPL projects based on RTLinux GPL or under the Open Patent License are something we support. I doubt Adeos avoids the patent - but that's a guess based only on the reliability of people claiming it does.
Q: Any reaction to Eben Moglen's letter giving RTAI clearance from patent claims?
A: I'm amazed at how carelessly people are reading that letter which, after all, begins with the statement:
If RTAI is released under GPL, as it can and should be, it is fully protected against any infringement claim …
Everything else Eben writes depends on this condition, but RTAI has not been released under the GPL, so it's a moot point. If RTAI were released properly under the GPL, I'm sure Eben would agree with three warnings: (1) you need to define terms like “application” exactly, (2) the intent of the license is to permit use of GPL software and there is no simple trick that can turn it into a general grant and (3) both the Open Patent License and the GPL on RTLinux Free are between FSMLabs and the user, and neither the Free Software Foundation nor Eben are direct parties to any agreement.
This stuff is not simple, even if you just stick to copyright. For example, it's well known that Linux user applications do not have to be GPL if they invoke the standard system calls. But that rule depends on good faith; it is not a recipe for evading GPL. If you took a kernel modification, placed it in a user program and then stored it in the kernel address space, using the standard “write” interface, you might still need to GPL. Just calling something an application or creating an artificial barrier (like a processor mode change) does not automatically escape the requirements of either GPL or the Open Patent License.
The Open Patent License is designed to make things easy for GPL developers. People who want to make non-GPL uses of the technology should buy licenses or get some specific analysis on whether they are under the terms of the Open Patent License. We're happy to answer questions or sell licenses - and our licenses are not expensive and are pretty flexible.
Can I take this chance to say a few words in favor of patents? The purpose of patents is to protect innovators against people or companies with more money who would otherwise simply appropriate the work. Everyone knows it doesn't work so well, but it does work sometimes.
Last year a guy from one of the huge telecommunications equipment companies that is heavily Linux-involved told me “We are very aware of RTLinux technology, but your patent makes things awkward for us.” He knows that the RTLinux Open Patent License allows royalty free use in GPL software, and his company has thousands of patents - not a single one released under GPL terms. So what he means is pretty clear, and it's clear that the patent is doing exactly what patents were intended to do. That particular company never asked us for our prices. Obviously, they would not have purchased support from us either. So next year, competing telecom products using our technology will start to come out, and the patent will have worked.
Q: How would you characterize the current state of embedded Linux, as a market and as a technology?
A: The success is more than anyone really could have imagined. Linux is now recognized as a standard tool for embedded systems and it is starting to become part of the basic infrastructure - we are seeing Linux as a bootloader in some boards, like those from SynergyMicro. The success is global, and in fact is going much faster outside the US. The technology is being recognized as a sort-of universal solvent for embedded devices both because it is so flexible and because it brings along a uniform application platform.
Q: What challenges do embedded Linux OS, tools, and services vendors face?
A: There is a question of how to pay for core engineering. In the server space, we see that vendors and customers have managed to put together what seems like a sustainable development effort. Linus and Andrew Morton, the RedHat and SuSE developers, and others are paid to work on basic engineering and keeping the core system going. The vendors and user groups like Debian are putting together solid, easy to use, packages. We don't see the same situation in embedded yet.
In embedded, we have a paradoxical situation where the hardware fragmentation makes the expense of maintaining up-to-date, solid BSPs high, but much of the market is unrealistic about what they will pay for a Linux BSP. RedHat Enterprise really just needs X86 to work, and the boards are all compatible. Compare that to the situation in embedded and you can see the costs for embedded OS vendors are going to be higher.
At the same time, as Michael Tiemann pointed out, the customers in server space don't mind paying someone else to take care of the OS platform, but in the embedded market the “perceived value” of Linux-as-board-support is zero. People need it, but they hate to pay for it.
Like everyone else in this market we get a stream of requests to port Linux to wacky, one-of-a-kind hardware for nothing or nearly nothing - and, of course, to maintain and support it for the same. My guess is that this is also a problem for Microsoft CE, but once customers are locked into .Net, Microsoft will be able to squeeze money out of them. No question about that.
For FSMLabs, the situation is not bad. We can afford to lose money on Embedded Linux in order to sell our non-GPL real-time products. But we'd be a lot happier if there were some form of cooperation between Linux tools and components vendors and customers. I still have hopes that the Embedded Linux Consortium can help. The example of the collapse of the original UNIX market into 20 competing midgets fighting each other while Microsoft became a giant, is something to remember.
Q: What market opportunities do you see for Linux in the embedded devices and systems market?
A: This market is expanding faster than anyone can measure it. Our focus is on factory automation, test equipment, aerospace, robotics, and network infrastructure - if you can call a list like that a “focus.” But we see a lot of interest in other areas, and our customers repeatedly show us that their imaginations are much better than ours.
Q: What challenges does embedded Linux face as a technology?
A: Bloat, bloat, bloat. The server guys run Linux, and although Linus continues to push for discipline, it's not enough. I keep getting surprised by how small the BSD components are compared to Linux. Over the next couple of years, I think that FSMLabs and other Embedded Linux vendors will need to invest some serious engineering efforts on the core of Linux to keep it flexible enough for our markets.
Q: What embedded Linux technology developments do you find exciting?
A: I'm happy that Linux has become a completely boring UNIX that runs on most platforms. Excitement is for applications.
Q: Can you share one or two of your company's most exciting successes?
A: Surviving the boom when everyone either gave us insane advice or told us they were going to run us over. At one point, I was thinking about installing a ticket counter so that all the people promising to crush us could take turns in an orderly fashion. We've been profitable for five years now, and it's been absolutely fun.
Q: How about a failure?
A: Our biggest failure, my biggest failure, has been public relations and marketing. We rely on technical edge, but you need some minimal marketing.
One thing I learned is that if you don't tell people what you are about, competitors will be happy to “help.” There were all these VC flush companies with charming salesmen, slick ads, and eager engineers who didn't need to bring in a profit, and we stayed in New Mexico writing code and not playing the game. Especially in the consumer electronics market in Japan and the industrial automation market in Germany, failure to market cost us, and we are only recovering now. Rick Lehrbaum and John Chueck (in Japan) and other smart people tried to explain this to me early but it took a long time to sink in.
Q: What's your vision of the future for embedded Linux? That is, how big will the embedded Linux market get, and when will it start to reach that peak?
A: UNIX in the form of Embedded Linux and BSD has won. There is no peak until something better is invented and gains enough market share to create its own market.
Q: How do you see the Eclipse tools platform fitting into the future of embedded Linux development?
A: No idea. We are moving along with several non-Eclipse based IDE solutions for RTLinuxPro and RTCoreBSD, but we're keeping an eye on Eclipse.
Q: Any comments on the SCO mess?
A: Until or unless SCO actually explains exactly what it thinks is in violation of its IP, their complaints are suspect. Maybe they have something there - I don't know, but they won't tell. On the good side: the case has created market awareness of the importance of getting the IP licenses clear, and this is good for all of us who develop code, GPL or not. Otherwise, not much to say except that the Canopy Group experiments with Linux have been mighty interesting.
Q: What do you think about Wind River's recent public endorsement and announcement of support for embedded Linux?
A: Great. The more tools and support the better. WindRiver's tools are widely liked. They have good technical depth. They can bring a lot to the Embedded Linux market.
Q: How do you see the 2.6 kernel affecting the embedded Linux market?
A: 2.6 is just more of the same - except that the integration of uClinux is very useful. I hope it stabilizes sometime in the next six months. We only have one engineer who is currently active in Linux core development - Zwane Mwaikambo - and we're waiting for him to give us the word when it is time to take a deeper look at 2.6.
About the CEO: Victor Yodaiken, CEO and Co-Founder of FSMLabs, came up with the basic RTLinux technology. Yodaiken began his career in 1983 as one of the chief developers of Auragen's distributed fault-tolerant UNIX and he had an active consulting business before starting FSMLabs. He has also worked in academia, as a professor and department chair at New Mexico Tech, and as a research professor and port-doctoral fellow at the University of Massachussetts in Amherst. Currently he is an adjunct faculty member at the University of New Mexico. Yodaiken is a technical advisor to EMBLIX Japan and is on the board of the Embedded Linux Consortium.
.
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.