This whitepaper from a senior developer with Squirrel Systems discusses how Squirrel used Linux in its highly successful POS (point-of-sales/service) system. It focuses on the advantages and challenges of choosing embedded Linux, and may be of interest to other companies considering embedded Linux. Enjoy . . . !

---

Summary

In 1998, restaurant point-of-sale (POS) vendor Squirrel Systems launched its next-generation restaurant management product, SquirrelONE. The architecture was based on a client/server model. After a significant technology assessment, Linux was selected as the client platform. However, the solution was not your vanilla, off-the-shelf Linux offering, as a number of constraints required a unique Linux solution. Along with obvious advantages, Linux presented unique challenges. Analyzing his experience of employing open-source software in commercial products, the author hopes his findings might help those wondering whether the use of Linux would make sense for them as well.

After starting operation in 1984, Squirrel Systems has emerged as a leading provider of POS restaurant management systems. Squirrel's customers consist of large multi-unit chains, fast casual, fine dining, cafeteria, clubs, hotel restaurants, airports, and the like. Squirrel's POS software runs on more than 50,000 terminals, installed in approximately 10,000 restaurants, hotels, and clubs worldwide. About one third of those terminals currently operate on a Squirrel-developed Linux implementation called Squirrel Embedded Linux (SEL), which was introduced to the industry in 1998.

Squirrel POS

It should be noted that, being focused on the table service market, Squirrel POS is not standalone software, but rather part of a turnkey solution that includes hardware, software, installation, training, service, consulting services, and 24x7x365 worldwide support.

Figure 1. Squirrel POS Architecture
(Click to enlarge)
Hardware

The Squirrel system consists primarily of a backoffice server PC, one or more terminals, peripheral devices, and Ethernet networking gear. The default terminal is manufactured by Squirrel. It is a retail-hardened, diskless, flatpanel, touch-screen unit with integrated credit card and server ID readers. However, Squirrel's open architecture allows customers to order other workstation platforms certified by Squirrel. An example of a third party terminal operating SquirrelONE software is the IBM SurePos 5xx series retail terminal. In addition, SquirrelONE supports wireless remote ordering on PocketPC devices, such as the Symbol and FIC AquaPad, and Windows Tablet PCs such as those from Acer. The Squirrel terminal supports a number of peripherals — printer, scale, barcode reader, customer display, bar pour gun, etc. — which are required for POS operation.

Figure 2. Squirrel diskless terminal
(Click to enlarge)
Software

After much technical due diligence, Squirrel decided to deploy a rather unusual software scheme for its system. First of all, it uses Microsoft Windows as a Server OS, and Linux as a client OS, in a heterogeneous model. Curiously, back in 1997/1998, Linux was mostly recognized for its use in servers, and much less as a “front end” OS, so this “reverse” architecture was quite a risky endeavor. These days, the fact that Linux scales down similarly well to embedded devices wouldn't surprise anyone.

Secondly, contrary to the “thin client” use of diskless terminals one would normally expect, Squirrel's terminals are “thick.” That is, the POS application is actually running on the terminal, and not on the server machine. (Note that Squirrel has since developed and piloted a thin-client or dumb-terminal version intended to allow older 486-style terminals to run the new SquirrelONE application.)

The restaurant application is extremely GUI-rich and performance-driven. A bartender in a high volume bar measures things in keystrokes, and if they can perform one less keystroke per transaction, it makes a big difference. There can be no hesitation.

Figure 3. Squirrel POS Client GUI
(Click to enlarge)
So, overall, Squirrel POS has a typical three-tier system architecture (see Figure 1), in which the POS server stores/retrieves its data in real time at/from a Microsoft SQL Server DB, while communicating with a POS Client via TCP/IP messages. Local POS servers can be controlled from a remote Corporate POS Server in multi-unit chains from an enterprise level. Both the POS Server and the SQL Server are running on an MS Windows machine. The POS Server is written in C++, while the POS Client is written in Java.

The reader may want to check my previous article regarding why Java was chosen as the development language for the POS Client. The POS client normally runs on a Linux (SEL) terminal, but because it is written in Java, it could also be run on an MS Windows machine. Some of our customers choose the Windows OS platform at the terminal, in the event that they have other Windows-only based applications that must coexist. In fact, one of Squirrel's products aimed at small establishments, Presto!, has both the POS Client and the POS Server running on the same machine.

We thought then and still believe now that the above architecture is the optimal solution to satisfy the numerous and often mutually exclusive requirements we faced. Of course your mileage may vary.

Linux at Squirrel

Linux as the operating system of choice for the Squirrel terminal is one of the four key components of the Squirrel software (Client, Server, database, terminal OS). It was chosen primarily for technical reasons, but as time passed, it became clear that there are some other less-known benefits associated with this choice. I will talk about them a bit later. However, choosing Linux came at a price, as there were a number of technical hurdles that had to be resolved in order to integrate Linux into the new Squirrel product.

Problems Squirrel had to overcome

I have to say that this new product, SquirrelOne, in which Linux was about to make its debut, was built ground-up and new to the bone. It had new hardware and a database engine, the client was written from scratch, and the terminal OS, as well as the development language for the application, were also new. On the other hand, it is a very complicated product, with many developers involved. Couple that with the expectation that the new program had to also include all the features accumulated in the legacy system over the last 12 years, and you have some major expectations to deliver on. So, Squirrel faced quite a few challenges that were not all Linux-related. I will only touch on those I've been involved with.

Lack of Linux expertise

This was likely the single biggest problem for the Squirrel development team. Out of the diverse programming team at the time SquirrelOne was conceived, no one on our team had any prior Linux experience, not to mention embedded Linux experience. And, although there was great support from the community, a wealth of documentation, and sources, the risk was perceived as high, and fear remained. While I admit that the learning curve was steep, this problem turned out to be less difficult than it first appeared. A good example of this would be the custom touch-screen device driver written by our novice Linux developer. It was his first Linux driver, and yet it remains practically unchanged from its original release. This driver just happens to be so robust that it can handle a very high volume of touches with ease (One customer site with 31 terminals registered 1,000,000+ touches per day).

No Embedded Linux Distribution

While the Squirrel terminal is not deeply embedded, its design is tailored to the specifics of the restaurant industry, and as such it requires a highly customized OS. Unfortunately, at the time of development there were no off-the-shelf, shrink-wrapped embedded Linux distributions available on the market, or as an open source. Development tools were also a big issue. Squirrel had no choice but to build its own distribution from scratch, if it was going to achieve the technology required to solve the real-time and customizable demands emerging in the industry. These days, of course, there are more choices, and you may find it more efficient to adjust open source embedded distributions for your needs, or use one of the commercial distributions offered by MontaVista, TimeSys, LynxWorks, SuSE, and others.

Limited support

This perceived problem actually turned out to be a misconception. It is true that unless you buy a support contract, either as part of a commercial distribution or from an independent support provider, you are on your own. In practice, though, “free” (you still have to invest time to find the help you need) support from the community of Linux developers was so great that I'd even say it was better than many paid support programs I've come across. Once you figure out how to use the available Linux resources effectively, you'll be fine.

Software footprint/storage size

To simplify development, as well as to address the scalability concerns for its first Linux POS, Squirrel chose a “thick” client architecture instead of a “thin” one. This meant that all required software (OS, X, JVM, client application) had to run on the terminal, rather than on the server box. Because the terminal is diskless, it also meant that we either had to build-in all this software into a loadable RAM image (don't forget the memory prices in those days, not to mention the limited choice of boards able to support a RAM module of the required size) or make it accessible from remote storage.

No embedded JVM

In fact, there was not even a choice of regular JVMs for Linux, not to mention embedded ones. The only JVM available at the time was Randy Chapman's “clean room implementation” of the 1.02 specification (our sincere thanks to Randy). Standard JVMs are big, slow, hard to control, and require resource-thirsty X, lots of RAM, and fast CPUs. We heard a lot of promises, but unfortunately a JVM that was able to satisfy all of our requirements completely never materialized.

No hardware support in Java

The Squirrel terminal needs to control various peripherals attached to its serial and parallel ports, as well as the system clock and speaker. It also needs to turn on/off LCD backlight from the Java application.

No cursor control in Java

Touch-screen driven applications like POS or kiosks require the cursor to be hidden. Since JVM up until v1.2 did not support cursor control, this issue had to be resolved at the OS level.

Retail-hardened custom touch-screen controller

The Squirrel terminal historically uses an in-house designed touch-screen controller. Obviously, no Linux distribution has any support for this custom design, so we had to write the device driver ourselves.

Customer's requirement to have the server running on an MS Windows PC

For a large number of our customers the Squirrel Server is the only server box they will ever have on site. Although our marketplace understood the value of the terminal flexibility, the overwhelming demand for the backoffice PC Server was for a Windows-based platform. As much as we'd like to be a technology leader and suggest alternatives, we are a customer-oriented company first and foremost. Customers leverage the platform we deliver to run other applications such as MS Office, accounting and HR software, etc., and the marketplace feels they have more capability on the backend with a Windows environment . . . at least so far.

Squirrel Embedded Linux (SEL)

In order to overcome the aforementioned problems, Squirrel developed its own embedded distribution. Interested readers can find a detailed description of how the first SEL version was developed here, but I'll just touch on a few basics. Keep in mind, though, that the current version is quite different from its ancestor. The most interesting features that I believe set this particular distribution apart from some others are: the support for diskless devices, and a two-stage boot from Windows PC. Some might also find it interesting that SEL supports hardware control from Java, via included native libraries, and “no cursor” appearance through modified fonts.

Diskless

Even today, there are not many POS systems that use diskless workstations. And, although SEL has versions to support devices booting from hard drive or Compact Flash cards, it was designed primarily for diskless terminals. As you know, the hard drive is one of the major contributors to the system's MTBF (mean time between failures), and it also consumes a significant amount of energy. No wonder Squirrel has been doing diskless terminals for 20 years! Fortunately, the main Linux building block components that are required for a diskless operation have been known for years, so it wasn't a big deal to build the rest of the distribution around those components.

Two-stage net boot

To address the “small RAM, but lots of software” issue, the Squirrel terminal uses a two-stage boot. First, its Etherboot client loads the Linux image (the kernel along with the minimal root file system) from the tftp server, then it mounts the remote file system via SMBFS, and boots the rest of the software from there. This way, space-demanding JVM, the POS Client, and X do not need to be included in the bootable image, so no RAM is being wasted for storage. Interestingly, the terminal boots off the Windows machine and not the Linux server, as some might expect for a Linux client. The remote file system is also located on the Windows Server.

Cursor

Getting rid of the cursor on the Linux platform is a good illustration of some advantages of working with open sources. It wasn't a big deal, even without prior knowledge of how the fonts work on Unix, to figure out which font source file was being used by the JVM to draw the cursor on Linux/X (incidentally it happens to be cursor.pcf :-), modify it, and rebuild it to hide the cursor.

Java native libraries

At the time SEL was being designed, JVM didn't have low-level support for serial or parallel ports. And, some other Squirrel POS functions also required direct hardware access that JVM couldn't provide. To overcome this issue, Java native libraries were developed and included as part of the SEL distribution. These libraries use the RNI API to interface with Java.

Touch-screen device driver

Although there's nothing special about this driver as part of the distribution, what's interesting is how it was developed. As mentioned above, the developer of this driver had no prior experience with either Linux applications or driver development. Yet, using sources for the parallel port, mouse driver, and xf86Xinput X Server extension as samples, he was able to quickly write a driver for our retail-hardened custom controller.

Now that we know what challenges Squirrel was facing in developing its upcoming all-new product, what tools it decided to use to address them, and how they were resolved, let's talk about what we've learned using this solution, to see if this knowledge could be useful for you, too.

Facts learned

Looking back at our experience, we believe that the deployment of Linux as a terminal OS for our product allowed us to gain certain advantages in the following areas:

• Technology leadership
• Hardware design
• TCO
• Maintenance
• Security
• Scalability

Let's talk about that in detail.

Technology leadership

Being the recognized technology leader for its first-in-the-industry, totally integrated Intel-based Touch POS terminal, first flat panel Touch PC, and other pioneering work, Squirrel designed six years ago what these days many industry experts have validated and believe is the future of POS: the diskless intelligent client. Although, as we'll discuss in a moment, there are financial advantages associated with a Linux-based solution, cost was not the key factor in choosing the OS. Surprising as it may sound, the decision to use Linux was actually based on the fact that out of a number of OSs evaluated, mostly COTS (commercial-off-the-shelf), Linux was the only one able to support the full set of innovative design ideas planned for our new product. What's even more surprising is that we were able to get a working prototype running within days, as compared to weeks of unsuccessful attempts with COTS counterparts.

Hardware

Linux's flexibility and openness, along with excellent support for various hardware components, gave us full control of the OS, and thus freedom to design our terminal with specific industry requirements in mind, rather than the limitations associated with other OSs.

Total cost of Ownership (TCO)

Let's have a look at this subject from two different points of view, that of the vendor and that of the customer. Lower vendor TCO often allows the pass-through of savings to the customer, which makes the product more competitive. However, TCO is quite a complicated issue, and it has many components. I'll touch on only the most important components here. And it's not my area of expertise, so don't hesitate to correct me if you find the analysis below erroneous.

Vendor TCO

• Acquisition (and royalties) costs

  As a general rule, even the most expensive open source flavors, such as commercial embedded Linux distributions, are much cheaper to acquire than a proprietary OS. Depending on the vendor's pricing policy, they may or may not include runtime royalties. You can also modify one of the distributions available for free download, or collect the relevant sources from the 'Net and make your own. However, keep in mind that lowering acquisition costs in this way in your particular situation may lead to prohibitively high development costs, and a lengthy time to market. So, it may be more efficient to purchase a commercial distribution, along with a support contract. You really have to do the math up front to weigh the time to market versus savings you may be able to employ by customizing your own distribution.

• Support costs

  We chose not to buy a support contract, but used the 'Net as a help source instead. It worked well enough to resolve arising issues in a timely manner. I recall only a few questions that we posted that remained unanswered, and usually we received help within 24 hours, often from the lead developer of the project in question. Thus, support from the Linux community was outstanding.

• Development costs

  As already mentioned, Squirrel decided to build its own distribution, rather than use a COTS product. That apriori meant higher development costs up front, but tremendous savings on a per-unit basis when you are shipping thousands of units. However, contrary to our concerns about our lack of previous Linux experience, development actually went relatively easy, to the point that it was all done by a single developer. That included drivers, native Java libraries, tools, build environment, and the overall distribution. What's more, the same developer also handled all internal Linux maintenance and support. Since development was completed in a relatively short period in our case, it was a cost-effective alternative to the COTS purchase option. On the other hand, our expenses for customer support were minimal. Much to our surprise, Linux-related customer issues are so rare that we don't even have a Linux dedicated support employee. It is handled by the same person who does development. The terminal is pretty much perceived as an appliance that you plug in and it runs and runs for years.

Customer TCO

The Squirrel POS system has one of the lowest TCO in the Industry, and the Squirrel terminal running Linux is the major contributor to that achievement.

• Acquisition costs

  Squirrel's customers benefit from not having the expense of a hard drive at the terminal, nor a heavy duty, expensive power supply to drive it, as in other vendors' workstations. And, additional savings result from a greater control over the OS, and the luxury it provides of a broad choice among cost-effective hardware components. Software-wise, the customer benefits from being free of any obligation to buy a terminal OS or anti-virus licenses.

• Support costs

  Because of greater terminal OS stability, the customer has less of a need to buy a Linux support contract.

• Hardware and software maintenance costs

  The aforementioned greater control over the OS gives more freedom in the selection of terminal components, with reliability and energy efficiency in mind, as well as functionality and price. This, along with the diskless design, resulted in a seven-year Squirrel terminal performance and support guarantee, unprecedented in the industry. This translates into significant maintenance savings. Another result of this freedom was the achievement of an EPA Energy Star compliance status. This may not sound as important, but knowing that a medium-size restaurant chain has about 250 terminals, consuming at least 20 watts less energy than a typical Windows-based PC terminal results in a seven-year saings of 152,040,000 Watts of savings (250 workstations x 20 watts x 12 hrs/day x 365 days x 7yrs). Another benefit of having a diskless terminal is reduced software maintenance costs. In case of a software upgrade or a virus compromise, the recovery involves the backoffice PC only. Terminals will be done automatically. For a chain of say 200 stores, that makes a big difference.

  Without arguing about which OS is more prone to virus or security attacks, it is important to note that embedded Linux, at least currently, is much less likely to be the subject of attacks, compared to other (widespread) OSs. Moreover, even if an attack takes place, the cure is likely just a terminal reboot away, while disk-based workstations would need recovery work.

Scalability

Squirrel's heterogeneous three-tier POS System with a Linux “thick client” terminal proved to be quite scalable. We have customers running setups ranging from one-terminal up to thirty-three terminals per server, where the activity is similar to being at the Christmas food fair all the time. Some corporate setups employ as many as 2000 terminals across their enterprise. It also happens to be a very flexible system in terms of hardware, as it runs on not only standard Squirrel diskless terminals, but also on IBM SurePOS terminals, wireless tablets, handhelds, and remote terminals that are miles away from the server and the like.

Conclusion

If you're in the process of making a decision about whether or not to employ Linux, please think carefully as you've read that there are not only advantages but also challenges associated with this choice. However, if you've made a positive decision, go for it without fear, because Linux is mature enough to make your product successful. And 16,000+ Squirrel terminals running Linux are proof of that.

---

Resources:

• SquirrelPOS article at Embedded Linux Journal Online
• Linux-from-NT HOWTO
• EtherBoot
• Embedded Linux Distributions Quick Reference Guide
• Open source embedded distributions
• Commercial embedded distributions

---

About the author: After earning his PhD, Pavel Tkatchouk been busy teaching Computer Science, and also taking part in a number of projects for the Russian Space Program. He is currently employed as a senior analyst/developer at Squirrel Systems. Pavel shares his spare time between his family, skiing, and his passion, Linux multimedia gateway development. You can send him email at [email protected].

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.