Storage Area Networking (SAN) equipment has long been inaccessible to mere Linux-using mortals, due to the prohibitive costs of fiber channel networking equipment. A small startup has set out to change all that, by developing a SAN architecture based on the ATA-over-Ethernet protocol, which allows… inexpensive commodity IDE disks to be used in SAN devices that work over standard Ethernet networks.

(Click for slightly larger view of Coraid EtherDrive)

The startup, called Coraid, recently shipped its first two products, which support only Linux so far (other open OSes to follow). The “EtherDrive” SANs and are priced as low as $1.75 per GB, and can scale to 16,384TB when used with the latest Ethernet switches. The SANs target cost-sensitive early adopters needing fast, scalable storage with low power and footprint requirements.

Coraid CEO Jim Kemp said, “We're initially positioning EtherDrive for Linux servers. Frankly, Linux didn't have a suitable network storage option. We offer that.”

Why Ethernet?

Kemp looks back at the history of storage area networking in explaining why he thinks Ethernet is a better choice than fiber channel. “Fiber channel networking was chosen over Ethernet in the early days of SAN technology, because at 10Mbps, Ethernet was deemed too slow. Since then, Ethernet has actually become faster than fiber channel. There was a lapse of memory for a decade here, as people built an industry around fiber channel.”

Kemp believes Ethernet provides a better, less expensive alternative to fiber channel, and his company developed the ATA over Ethernet (AoE) protocol as an open standard in order to make Ethernet-based SANs possible. Basically, AoE converts parallel ATA signals to serialized Ethernet format, which enables ATA block storage to be remotely accessed over an Ethernet LAN in an ATA (IDE) compatible manner. A big advantage of AoE is that it makes use of standard, inexpensive, ATA (IDE) hard drives commonly used in desktop PCs.

GPLed open source AoE drivers are available for Linux 2.4 and 2.6. The drivers have been submitted for inclusion in the mainstream kernel, although acceptance could take a few months, according to Kemp. Drivers for other open source operating systems are the next priority for Coraid, according to Kemp, who adds, “OpenBSD is next on our list, but we have to be market driven there.”

Kemp admits there will be marketing challenges to overcome, given the widespread association people make between Ethernet and “best-effort” TCP/IP networking. “That's exactly the problem,” Kemp concedes. “People think Ethernet is TCP/IP. But it's a Layer 2 transport, IP is Layer 3, and TCP is right above that. AoE rides right on Ethernet, and passes just fine through the switches in use today.”

Kemp adds that the technology works best on decent quality switches with backpressure and flow control.

Some storage technologies, notably iSCSI, have used TCP/IP, enabling remote storage devices. AoE is not a routable protocol, although it could be implemented under an iSCSI head in applications requiring remote storage, according to Kemp, who adds, “iSCSI is a great companion technology.”

As a result of not including the complexities of TCP/IP routing, the AoE protocol stack is much simpler than that of iSCSI, as seen in the following diagram.

AoE vs. iSCSI

In comparing AoE to iSCSI, Kemp points out the following advantages of AoE:

• AoE is a simpler software stack
• Server processing load for iSCSI is much higher than AoE for equivalent throughput
• AoE is not a routable protocol (this provides inherent security)
• AoE Ethernet frames are passed by standard switches

EtherDrive is the first product to implement the AoE protocol, Kemp says.

Inside the EtherDrive

The EtherDrive storage products are based on “shelves” with slots for storage blades. Each blade has a 100Mbps Ethernet port. Blades are connected to normal, off-the-shelf Ethernet switches with a 1Gbps Ethernet upstream port. These GB upstream ports can themselves be aggregated using a switch with 1Gbps downstream ports and a 10Gbps upstream port. Up to 256 shelves, or 368.64TB of storage, can be aggregated into a single network.

Because each blade has an embedded processor, performance actually scales incrementally as blades are added, according to Coraid. Coraid lists 5MB/sec sustained throughput, and 5,000 IOPS (I/O operations per second) for a single blade, with figures exactly ten times higher for an EtherDrive with 10 blades. EtherDrives support striping, to increase performance, mirroring, for reliability, or both, according to Kemp.

Architecture of a single EtherDrive blade

Once attached to the Ethernet, the EtherDrive might be configured for use by a single Linux server that would serve as an NFS head exporting files to the rest of the network. Or, individual machines on the network might claim use of individual partitions on the EtherDrive(s). Alternatively, the EtherDrives could be configured using a filesystem such as GFS (global file system) that allows a cluster of servers to read and write to the same disks.

GFS was developed by a company called Sistina, which was recently bought by Red Hat. There is also a free GFS implementation called OpenGFS. Kemp grows animated, talking about GFS: “You never had the capability to really worry about these things before. These kinds of more advanced architectures will be facilitated by our more cost-effective products.”

Shipping Coraid products

Coraid has been shipping two EtherDrive products for about two months, now. One is a 3U chassis, or “shelf,” holding up to 10 storage blades with 3-1/2-inch desktop drives. A single 3U shelf can hold up to 4TB of storage. Power requirements are listed at 12W per blade. Pricing is set at about $1.75 per raw GB, according to Kemp.

The other shipping Coraid product is a 2U shelf holding up to 18 blades with 2-1/2-inch laptop drives. A 2U shelf can hold 1.44TB with the drives available today, although the shelves are shallow enough to double up, at the front and back of a rack, for a capacity of 2.8TB per 2U.

Kemp calls the 2U EtherDrive a “boutique item” for low power or space constrained applications. Power requirements are 65 watts per 18-drive shelf, or 5.5W per blade worst case, 3.6W per blade typical, and 2.1W per blade idle. Pricing for the 2U unit is higher, at $7.06 per GB, due to the higher cost of smaller drives and their smaller capacities.

Power management software can actually spin down inactive blades, for use in “near-line storage” applications such as MAID arrays (massive arrays of idle disks), Kemp said.

Another interesting capability Coraid is working on is something called hard drive WORM (write once read many). WORM is required for certain high security applications, according to Kemp, and is a “near-term project” for Coraid that will be released as a firmware option.

Early response

Kemp reports that early customers have deployed the shelves successfully, and are planning to expand their installations. Coraid plans to step up marketing efforts over the summer, with a focus on Linux users.

Kemp concludes, “We truly are a SAN solution, but SAN is not in the vocabulary of Linux people, because SAN is equated with fiber channel, and fiber channel is too expensive. But now, there's 'poor man SAN.'”

Sounds good to us. Got milk?

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.