Foreword: In this technical whitepaper, Viosoft CEO Hieu Tran uses his company's debugging tools to explore QuickLogic's FPGA-based PCI bridge for Intel's PXA-27x, and an Atheros Wi-Fi card. He turns up an interesting attempt by Atheros to balance GPL compliance while protecting confidential third-party information.

The article outlines Tran's experience in setting up and using the QuickLogic PXA Wi-Fi daughter board to add a Atheros 802.11 A/G wireless card to Intel's PXA27x Mainstone evaluation board. By using a kernel debugger to set a few breakpoints in selected source files, he gathers some important clues about the operation of the bridge.

---

Interfacing PCI-based peripherals to the Intel PXA27x

by Hieu T. Tran

In March 2005, QuickLogic announced the availability of the PXA Wi-Fi daughter board, allowing designers to interface PCI-based devices to the Intel PXA27x application processor. The idea was that for high-performance I/O devices such as 802.11 A/G wireless, PCI connectivity offered superior performance to other expandability options, namely PCMCIA and SDIO.

On a slow afternoon, I set out to figure how this bridge actually worked. This article is a summary of that experience.

System Configuration

Courtesy of the local QuickLogic marketing folks, I was loaned the PXA Wi-Fi daughter board, and an Atheros mini-PCI 802.11 A/G wireless card. The daughter board was designed to connect to the expansion connector on the Intel PXA27x Mainstone evaluation board. The Atheros wireless card was installed in the mini-PCI slot on the upper side of the daughter board. The whole setup fit neatly within the existing footprint of the Mainstone board, and was secured in place by one screw in matter of seconds (Figure 1). I powered up the Mainstone board after installing the bridge and booted up Linux 2.6.9. Off to a good start.

Figure 1 — PXA27x Mainstone with QuickLogic Bridge and Atheros Wireless Card
(Click to enlarge)

Next, I built from source a version of the Atheros madwifi driver that QuickLogic engineers have modified to work with the bridge. This process was straight forward, and generated about a half dozen loadable modules in several directories. Initially, I expected that an unmodified Atheros PCI driver should just work once it was recompiled for the Mainstone platform. So I was curious to learn why the modifications were necessary.

Debugging Setup

My main method of discovery was to put the driver through a debugger (Viosoft Arriba Embedded Linux Edition for Intel PXA27x Processor, version 2.1) and step through its execution. The debug agent consisted of a pair of loadable modules that ran on the Mainstone board. The first of these modules, vmon-tcpip.ko provided an API used by the second module, vmon.ko, to communicate with the host computer over the Mainstone Ethernet port. Because vmon.ko executed concurrently with the Linux kernel, it did not pre-empt the servicing of incoming interrupts, allowing the Mainstone board to be fully operable under debug. This setup worked in my favor since my madwifi driver modules were accessed from the Mainstone board over an NFS-mounted partition. Other intrusive debugging methods, such as with JTAG, suffered from the annoyance of having the NFS volume locking up on the resumption of execution.

Since the madwifi modules loaded on demand, their code and data memory addresses were dynamically assigned by the kernel at load time. Because of this, they could be tricky to debug. At a minimum, I would need my debugger to correctly relocate the symbol information for these modules to match those assigned to them by the Linux kernel. The following commands, which were added to a start-up TCL1 script invoked by the debugger at connect time, did that:

proc OnConnect {} {
  sendCommand "mwatch insmod wlan /install/mainstone/wlan.ko"
  sendCommand "mwatch insmod ath_hal /install/mainstone/ath_hal.ko"
  sendCommand "mwatch insmod ath_pci /install/mainstone/ath_pci.ko"
  ...
}

Each of the above commands set a trigger point that put the target into debug each time a module from the above list was loaded. Since I didn't know in advance which module needed to be debugged, I added all of them to the watch list so that their symbol information would be available when needed.

A Linux loadable module can optionally define the canonical init_module function to be invoked at the time that module is loaded. Not having prior knowledge of the modules or their relationship, I wanted to step through the init_module function of each module. The mwatch trigger put me right at the entry point of init_module, which was nice.

Target Connection

To connect my debugger to the Mainstone target, I loaded the vmon-tcpip.ko module, followed by the vmon.ko module. Once loaded, vmon.ko waited for a connection request from the host debugger over port 1234 of the Mainstone Ethernet. From the host computer, I established a connection using the debugger GUI, and resumed target execution. The debug agent now sat in the background, waiting for a trigger to enter into debug.

Tracing the MADWIFI Driver

My objectives were to 1) understand the programming interface between the QuickLogic Wi-Fi daughter board and the PCI device, 2) understand the operation of the Atheros driver, and 3) understand why it was necessary to modify the Atheros driver to work with the daughter board.

To avoid typing the same thing over again, I created a small shell script to load the necessary modules and run it:

#!/bin/sh
cd /mnt/mainstone
insmod ./wlan.ko
insmod ./ath_hal.ko
insmod ./ath_rate_onoe.ko
insmod ./ath_pci.ko

The modules were loaded one by one. Each time, the debug agent put the Mainstone board under debug at the entry point to the module init_module function. Apparently, none of these modules define an init_module function, because the debugger stopped at the default init_module function supplied by the kernel.

Next, I wanted to trace through the initialization flow. “ifconfig -a” showed that ath0 was now one of the available network interfaces, so I decided to probe if_ath.c for starters. A quick glance at the source showed that the open function was mapped to ath_init, so I set a breakpoint there. I triggered the breakpoint by opening the ath0 device:

% ifconfig ath0 192.168.1.109 up

This put me right at the start of ath_init. I saw a call to ath_hal_reset at line 870, and wishing to see how it worked, I stepped into it. Strangely enough, I found myself inside a function with a rather obscure name: zz0002dbd2. A call trace was shown in Figure 2. A search of the driver source files for function zz0002dbd2 revealed matches only in binary files:

bash-2.05b$ grep -r zz0002dbd2 .
Binary file ./ath_hal/ath_hal.ko matches
Binary file ./ath_hal/ath_hal.o matches
Binary file ./ath_hal/hal.o matches

Figure 2 — Tracing through ath_init and hal_ath_reset
(Click to enlarge)

I looked in the local Makefile to see how it built ath_hal.o and hal.o, and found:

$(obj)/hal.o:
        ${UUDECODE} ${HAL}/${OS}/${TARGET}.hal.o.uu

It appeared that hal.o was not generated by compiling hal.c, but rather from a uuencoded image of a pre-built binary. To be certain, I examined the corresponding hal.o.uu file for the Mainstone board. Uudecoding it yielded hal.o, which was an ELF binary. And check this: the file header contained an official copyright comment that, to the author's credit, purported to make hal.o.uu available under a GPL-like license. (At this point, Conan, my Pomeranian who has been following all of this over my shoulder laughed so hard, he felt off the futon.) Open source anyone?

The Pomeranian laughed because the author of the code felt it was necessary to put a copyright statement in the uuencoded file; as if releasing a binary image with encrypted function name wasn't enough of a deterrent against copyright infringement.

I believe what I discovered was an attempt on behalf of Atheros to strike a balance between complying the the GPL and protecting the confidential information of third parties — the binary code implements the FCC's rules and regulations regarding how the signals of the wireless antenna are governed. This brings up an interesting issue regarding open-source code that implements proprietary standards and/or specifications. How should such source be distributed to comply with GPL?

We saw something similar a long time ago when Cygnus was releasing a GCC
implementation for the Sony Emotion engine, sans support for the portion of the CPU that was kept proprietary.

So much for figuring out how ath_init (and everything else in hal.o) worked. I turned my attention to the interface between the Atheros device and the QuickLogic Wi-Fi daughter board. I reckoned that at a minimum, the Atheros PCI device must probe for PCI at start-up. Inspecting the sources revealed that this probing was likely to occur in if_ath_pci.c at ath_pci_probe. I set a breakpoint there. Since the current instantiation of the driver modules would have executed beyond the PCI probing stage, I unloaded and reloaded them. The breakpoint hit.

Stepping through ath_pci_probe, I picked up several important clues. First, the file that contains ath_pci_probe, and several others have been modified to work with the daughter board, and the modification are kept under the “#ifdef QLXPB” toggle. This could be convenient if I wanted to search the entire driver source tree for all the changes that were made by the QuickLogic engineers, assuming that such changes were all kept under the same toggle. Second, the QuickLogic PCI registers seemed to be memory-mapped and accessed via a variable called ql_pci_bridge_base. A “grep” of this variable in the driver source tree turned up only extern declarations, so I did an “nm1” dump of the driver modules, which showed:

bash-2.05b$ nm *.ko | grep ql_pci_bridge_base
         U ql_pci_bridge_base
         U ql_pci_bridge_base

The 'U' preceeding each entry meant that references to that ql_pci_bridge_base variable from within the drivers are unresolved. Since the same driver modules loaded fine, the conclusion was that the variable was defined and exported by the kernel. A grep of the kernel source tree took a while, but showed an expected match in arch/arm/mach-pxa/mainstone_pci.c:

static int __init mainstone_pci_init(void)
{
   int ret = 0;
   void *fpga_base;

   if (!request_mem_region(MST_EXP_BASE, MST_EXP_SIZE, "PCI Bridge Registers"))
      printk(KERN_ERR "unable to reserve regionn");
   else {
      ql_pci_bridge_base = ioremap(MST_EXP_BASE, MST_EXP_SIZE);
      if (!ql_pci_bridge_base) {
         printk(KERN_ERR "unable to map regionn");
         ret = -ENOMEM;
         goto out;
      }
   }
   ...

Whereas, MST_EXP_BASE was defined to be 0x14000000 in asm/arch/mainstone.h, and ql_pci_bridge_base was 0xc2800000 as shown by the debugger. The picture got clearer:

1. The base address of the expansion card was fixed by the PXA27x processor at 0x14000000 (but you already know that if you have the yellow books)
2. This address was re-mapped to a virtual address at kernel boot time. Because the address assigned by ioremap was arbitrary, its value was stored in the ql_pci_bridge_base variable, and exported by the Linux kernel to be accessed by the driver modules at a later time

It now made sense why it was necessary for QuickLogic engineers to modify the madwifi driver: on a typical x86 PC, a PCI device can do a DMA transfer of incoming data directly into SDRAM of the main CPU. For network devices, data is passed up and down the protocol stack in a structure called skbuf, which consistes of a memory buffer and length word, and which can be accessed by an assortment of clever C macros. The PCI device DMA-transfers the incoming data packet into skbuf storage buffer and interrupts the device driver, which simply moves the data marker within skbuf to account for the amount transferred. Most PCI network drivers, including madwifi, work this way.

Because the Intel PXA27x exchanged data with the PCI device via shared memory at MST_EXP_BASE, packet data must be explicitly copied by the driver from this location to its intended destination in skbuf. Put another way, the modifications to the driver were needed because of the lack of DMA-transfer capability from the PCI device to the PXA27x main memory. This was evident in the following code fragment:

static void
ath_rx_capture(struct net_device *dev, struct ath_desc *ds, struct sk_buff *skb)
{
        ...
#ifdef QLXPB
        u_int32_t dummy;
#endif

        ...
#ifdef QLXPB
        /* Copy buffer data from SRAM to skb */
        dummy = (ds->ds_data & 0x1FFFF) >> 11;
        REG16(ql_pci_bridge_base + MST_EXP_SRAM_RD_UAB) = dummy;
        memcpy(skb_put(skb, roundup(len, 4)), ql_pci_bridge_base + MST_EXP_SRAM_
BUF_OFFSET, roundup(len, 4));
        if (dummy != REG16(ql_pci_bridge_base + MST_EXP_SRAM_RD_UAB))
                printk("nath_rx_capture: SRAM_RD_UAB mismatch!n");
#else
        skb_put(skb, len);
#endif
        ...

Note that the skb_put macro advanced and returned a pointer to the start of the skbuf data buffer. On the PXA27x platform (code shown under the QLXPB toggle), incoming packet data was copied from the bridge shared memory to the skbuf data buffer. In contrast, when DMA transfer took place, the driver simply advanced the pointer by the transferred amount.

Conclusions

This article outlined my experience in setting up and using the QuickLogic PXA Wi-Fi daughter board to add the Atheros 802.11 A/G wireless card to the PXA27x Mainstone evaluation board. By using a kernel debugger to set a few breakpoints in selected source files, I gathered some important clues about the operation of the bridge, and gained some surprising insights on the Atheros driver sources.

---

About the author — Hieu T. Tran is the founder and chief grunt at Viosoft Corporation. He can be contacted at htran [at] viosoft.com.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.