Wind River has announced a test automation framework aimed at streamlining the software quality assurance process for embedded Linux developers. Wind River Test Management (WRTM) enables development teams to plan and execute tests, perform fact-based analysis, and rapidly resolve testing issues, says the… company.

(Click for larger view of a Wind River Test Management test plan screen)

Expanding upon Wind River's existing diagnostics products, WRTM is described as a scalable, distributed testing framework that links device software development and quality assurance (QA) teams in a collaborative workflow. When QA teams find a defect, for example, developers can immediately access detailed logs and test-lab fault data. The framework helps to shorten the testing process, reduce development costs, and improve code quality, claims Wind River.

WRTM features and benefits are said to include:

• Test planning facilities for creating manual and automated test cases
• Test execution framework for automating execution of multiple test suites on selected targets
• Automatic collection of test and fault data for reproducing defects
• Diagnostics interface for shortening defect resolution process
• Auto-generated “dynamic” code coverage and performance measurement, collected without special test builds or additional test harnesses
• “Virtual Lab Management” (VLM) for managing boards, provisioning software builds, and sharing access to boards
• Wind River Workbench integration
• Support for Linux and VxWorks run-time platforms

Wind River Test Management test execution screen
(Click to enlarge)
Like Wind River's existing diagnostics products, WRTM is written as a J2EE (Java 2 Enterprise Edition) application. WRTM will integrate other Wind River diagnostic products, or with those from other vendors, says the company.

Wind River announced its first post-development device management products back in 2006 with its Field Diagnostics and Workbench Diagnostics software, which allow device vendors to remotely monitor, diagnose, and repair deployed devices. In February 2007, it announced a version of Field Diagnostics called Lab Diagnostics that was specifically designed for product testing labs and QA teams. The application lets developers dynamically instrument systems under test, without recompiling. It is also said to streamline the introduction of sample operating conditions that might otherwise take considerable effort to simulate using real-world testing techniques. In July of last year, Wind River released a new version called Lab Diagnostics 2.1 that added features aimed at accelerating automated testing, as well as improving support for large, complex test builds.

Wind River SensorPoint technology (Click to enlarge)

Wind River's diagnostic products communicate over TCP/IP with “SensorPoint” agents that reside on deployed devices. The lightweight SensorPoint runtimes are said to enable “dynamic instrumentation” in running applications, enabling programmers to insert printf statements or even whole new functions into running applications. In this way, testers can artificially induce specific operating conditions without recompiling and restarting applications.

Testing, testing with Amit Ronen

Wind River cites a Venture Development Corporation (VDC) study that claims that the worldwide annual growth rate of the test automation market is expected to grow at more than 12 percent between 2006 through 2009. The market is being driven by increasingly complex design requirements, especially due to the incorporation of multicore processors, says the study. Increasing time-to-market pressures and higher consumer expectations on device quality are also said to be fueling the testing industry.

Amit Ronen

“We have learned that in devices, testing represents thirty to fifty percent of the device development lifecycle, and a growing part of that is testing the software,” said Amit Ronen, Wind River's VP and GM of the new Device Management unit, which boasts some 60 employees. “Testing is growing in importance as the complexity of devices grows almost exponentially,” he continued. “Quality is becoming a big business issue. For some vendors, a single customer call to tech support can mean losing margins on that sale.”

Aside from the largest device engineering companies, which spend a lot of money building their own internal testing suites, most embedded Linux developers tend to use a “hodgepodge” of open source tools and manual testing techniques, said Ronen. Many use well-known open-source harnesses such as TestLink, a web-based QA tracking tool that was listed as the most popular open source software testing tools, according to a survey conducted in April by OpenSourceTesting.org (see chart below). Developers then tend to “fill in by writing scripts for developing manual tests,” he added. According to Ronen, WRTM does not currently offer a tracking service, but can integrate with TestLink and other tracking products.

Most popular open source software testing tools survey results
(Source: OpenSourceTesting.org)

Although WRTM can be deployed as a standalone product, one touted benefit is that it can be integrated with Wind River's Workbench integrated development environment (IDE). “Testing frameworks have been available from open source for some time but they are usually rudimentary and not integrated,” said Ronen. “Our product is an Eclipse plugin, so we can plug in on top of Workbench. It lets developers become part of solution, so you can have better collaboration between development and testing. We want to help get those two groups together.”

Aside from integration, WRTM's most significant advantage over typical test management packages is its dynamic code coverage capability, says Ronen. “It is very resource efficient, and everything is very dynamic,” he said. “Instead of reporting how good the code is, it provides feedback on the quality of the testing.”

Ronen related an anecdote of a Japanese CGL customer that mapped all the defects reported from the field back to the source code. The study discovered that 80 percent of the defects were matched to code the company had not tested.

“For some of the defects, it was because they had run out of time and decided to take a risk, but many other defects were related to code that they didn't know they hadn't tested, because it was hidden in these obscure places in the code,” said Ronen. “If you can run code coverage as an integral part of planning, you can dive deeper into the code and find the hidden defects. With Test Management, developers can start to look under the hood and see what's happening in an application, so when things go wrong they can find where to look in the code. It's not quite white-box testing, but we're moving in that direction. Call it gray-box testing.”

Another key feature is the Virtual Lab Management (VLM) module, said Ronen. This features aims to let testers incorporate remote hardware and emulated hardware targets into their test beds. Via telnet, systems in the virtual lab can be powercycled, configured, scheduled, and otherwise controlled.

Much of WRTM is devoted to managing the complex collaborative and testing process of distributed development, said Ronen, who acknowledged that much testing is outsourced off-shore. “You have the mothership application, and then you can drop in smaller satellite servers in remote locations,” he said. “The architecture offers control over execution engines that can be deployed remotely, so you can run a test locally, and then record it back to the mothership via a secure connection.”

Availability

Wind River Test Management is due out for Linux and VxWorks platforms in the third quarter, says the company. It will be sold in an entry-level “Team” edition, as well as a more scalable “Corporate” edition, but both will offer the same basic functionality, says Wind River. More information on Wind River's development solutions portfolio may be found here.

---

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.