When Linux NetworX announced a new appliance-like device that is used for managing rack-mounted Linux clusters, called the ICE Box, LinuxDevices.com naturally wanted to crack open the ICE Box's door to find out what's inside. In this article, Linux NetworX Cluster Tools product manager Jason Lowry opens up the ICE Box and takes you on a tour of its internal hardware and software, explaining what it does… and how Embedded Linux helps get its job done.

---

What's an 'ICE Box' and what's inside one?
by Jason Lowry

Linux NetworX is focused on providing Linux clustering solutions. Initially formed in 1989 to provide custom board-level products, the company began developing Linux clusters in 1997 and in 1997 delivered the world's first commercial Linux cluster. Today, Linux NetworX cluster systems are being used in a wide range of industries including oil and gas exploration, biotechnology research and modeling, graphics rendering, gaming, and aeronautics.

The importance of cluster management

Linux clusters are the high performance compute engine of choice for many industries seeking raw number crunching power. They provide greater flexibility, reliability, scalability, and price/performance over traditional supercomputers. Despite all the positives, clustering also presents some challenges such as cluster system management.

As cluster systems scale from dozens, to hundreds, and even thousands of nodes, management becomes exponentially complex and requires significant effort. To simplify this effort, Linux NetworX has developed Integrated Cluster Environment (ICE), a complete cluster management solution that consists of the ClusterWorX management software and the ICE Box hardware appliance.

What's an ICE Box?

ICE Box is a hardware appliance designed specifically for management of Linux clustering. It provides cluster monitoring and control functionality such as power control, node reset capabilities, and temperature monitoring. ICE Box probes and logs current node information to a remote cluster host.

ICE Box supplies administrators with various ways to control and monitor node functions and allows administrators direct node access through a serial console, providing them an alternate way to access the node in case of network failure, failed filesystem checks, bad hard drives, or other failures. The scalable nature of ICE Box allows it to grow with the cluster. Powered by Embedded Linux for a stable runtime environment, ICE Box can be updated without cluster downtime. ICE Box integrates with ClusterWorX software, creating a complete cluster management solution for high availability or high performance clusters.

The unit mounts to the back rails of a 19-inch rack and provides a serial terminal switch and a remote controlled power switch with support for temperature monitoring and system reset to cluster nodes and devices. It has support for ten nodes and two auxiliary devices. Each box has a network connection and unique IP address, which allows multiple boxes to create a highly scalable IP-based communication network that can seamlessly support several hundred nodes. ICE Box also has an out-of-band (redundant) serial network which can be daisy-chained to support multiple boxes.

ICE Box can be used by logging into it over the network, accessing it via a serial port, or plugging in a null modem cable. It also has a full command set with both network (NIMP: Network ICE Management Protocol) and serial (SIMP) compatibility, allowing it to be used with Linux NetworX ClusterWorX software or other software. New to ICE Box v2.2, there is compatibility with SNMP, allowing ICE Box to be configured with an SNMP management tool. The net-snmp libraries were used for SNMP support. An LCD and keypad mounted directly on the ICE Box allows local access.

The software to allow system administrators to communicate with the ICE Box is included within the Linux NetworX ClusterWorX cluster management software package, which offers an easy-to-use graphical user interface.

What's in the ICE Box?

Here's a summary of the hardware functions within the ICE Box . . .

• Embedded processor: 33MHz 386 embedded processor
  
• System RAM memory: 4MB
  
• Flash disk nonvolatile memory: 8MB SanDisk
  
• I/O Ports:
  • 15 RS232 serial ports, allocated as follows: 12 for nodes and auxiliary; 2 for SIMP (Serial ICE Management Protocol); and 1 for serial console
    
  • 12 custom I/O ports, used for remote environment and reset (environment channels are I2C and currently provide temperature)
    
  • 12 AC power outlets, 10 of which are switchable
    
  • 1 10BaseT (10 Mbit/s) Ethernet with network access or NIMP (Network ICE Management Protocol) support
    
  • 2 RS485 serial ports, to support multiple boxes on SIMP

The soft side of the ICE Box

ICE Box contains its own embedded version of Linux, based on a 2.2 kernel. There is a lot of custom hardware and, as a result, many custom device drivers were written. The Linux implementation was derived from kernel.org sources, plus uClibc. BusyBox was used during development of the device, but is not included in the final released product.

The ICE Box local user interface consists of a serially interfaced LCD display and keypad. The LCD panel is controlled by user-space software via a serial protocol that is unique to the LCD. The 4-button keypad is controlled via a custom kernel device driver.

Development challenges

The biggest problems occurred at the boundaries between different components in the system. The first challenge was getting device drivers written and debugged for the custom hardware. This was helped tremendously by the wealth of source code for drivers in the kernel distribution, and also by the Linux Device Drivers book from O'Reilly.

Later, we ran into some rough edges when linking together net-snmp libraries, uClibc, and our own code. Without the support of the kernel mailing lists and the ability to read and modify the source code to these libraries, integrating them would have been very difficult, if not impossible.

The benefits of Embedded Linux

Why run the device on an Embedded Linux operating system?

Linux is the platform with which we have the most development experience, it is freely available, and it is flexible enough to grow and change as our requirements do. We've been very pleased with our decision, and as our development progresses the benefits are even more evident.

For certain applications, such as networked appliances and other complex home electronics, Linux seems like a perfect fit. Smaller systems and ones with hard real-time requirements may not see much benefit from it. So long as there's a demand for the former kind of devices, I think Embedded Linux will do well.

The embedded market is a good match for Linux because people aren't necessarily trying to make money from their software. The software is used to make the hardware usable. Using Embedded Linux allows a lot of software to be added at very little cost and speeds up development time for custom software.

Incidentally, Linux NetworX makes significant contributions to the LinuxBIOS project — which aims to replace the commercial PC BIOS with an open Linux-based BIOS which runs in Linux user space.

---

About the author: As Linux NetworX Cluster Tools Product Manager, Jason Lowry is responsible for product positioning and future development planning of cluster management tools ClusterWorXT and ICE BoxT. In his free time Jason enjoys attending Utah Jazz basketball games, eating good Sushi, playing in ping-pong tournaments, and writing Perl scripts.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.