News Archive (1999-2012) | 2013-current at LinuxGizmos | Current Tech News Portal |    About   

Open source-based high-resolution cameras for Web developers

Apr 11, 2007 — by LinuxDevices Staff — from the LinuxDevices Archive — 43 views

Foreword: This article describes the products and product design philosophies of a small Utah-based company offering high-quality, intelligent, network-enabled cameras based on open source hardware and software. Elphel hopes its newest modular camera design will attract Linux software and FPGA engineers interested in exploring high-definition videography, among other innovative applications.

The article was written by Andrey Filippov, a Russian physicist who emigrated to the US in 1995, and founded Elphel six years later. This is Filippov's sixth whitepaper for LinuxDevices — the others are listed at the end. Enjoy . . . !

High Resolution Cameras for Web Developers

by Andrey Filippov

I believe that with our new 353/363 series of Linux-based network cameras we are one step closer to my dream of building hardware that is friendly to software developers. This time it is not only about gurus who can easily hack kernels and create drivers, but about usability for a much wider audience.

This article covers both the new hardware developed for the 353/363 series of the cameras, as well as the software these cameras are running or could run in the future.

Evolution of Elphel cameras

From our very first camera, I tried to keep the designs as universal as possible so that the camera (or parts of it) could be used for as-yet unimagined purposes. Hardware development (from the circuit design through PCB design, testing, and manufacturing) is rather expensive, and the cost of even minor changes on the boards is high. Multiple hardware revisions also result in incompatible hardware that increases the cost of customer support.

Our original Model 303 was designed around the Axis ETRAX100LX processor, which has the nice feature of a hardwired network boot loader so that without any additional connectors and cables, the camera firmware can be easily field-upgradeable even if the contents of the flash memory are corrupted by accident. The camera consists of two boards — the “computer” part and the sensor board — allowing us to upgrade the camera when new sensors became available without changing the main board.

The next level of universality was achieved when we included a reconfigurable FPGA in the subsequent Model 313 camera. This new part both interfaces to the sensor board and provides the resources for image processing and compression. All of the signal pins on the sensor interface connector are reconfigurable (flexible functional designation, direction, and optional pull-ups and pull-downs are available) to accommodate different types of imagers — directly for the CMOS sensors and through the additional FPGA for the CCDs.

This approach paid off when we were developing our third-generation Model 323 camera, which supports large Kodak KAI-11000 and KAI-11002 CCD image sensors. In this camera, we had only to develop a circuit board around the sensor — the rest was done by using a pair (to increase processing speed) of unmodified processor boards from the Model 313 camera. Even most of the FPGA and software code was reused, which allowed us to significantly decrease development time.

As we began work on our fourth-generation camera, we targeted several different applications, and developed multiple boards that can be interconnected in different combinations to make different devices. The core module (10353) preserves the same overall design ideas of the earlier cameras — a computer running GNU/Linux, network connection with PoE (IEEE802.3af), and a reconfigurable FPGA that interfaces the sensor board, has individual SDRAM memory and interfaces with the embedded computer both in PIO and DMA modes.


Processor board of Elphel models 353/363 cameras
(Click to enlarge)

The physical dimensions of the 10353 board are the same as those of its predecessors — 3.5 x 1.5 inches — but most of components are upgraded:

  • 200MHz Axis ETRAX FS processor
  • 100Mbps Ethernet port
  • 1200K gates Xilinx Spartan 3e FPGA
  • 64MB of system SDRAM
  • 128MB of NAND flash
  • 64MB of image buffer DDR SDRAM attached directly to the FPGA
  • 30-pin flex cable connector for the sensor boards
  • 30-pin high density daughter board connector, that includes serial port, USB and 12 GPI/O pins connected to the FPGA that can be used as 6 differential pairs
  • 40-pin connector that carries IDE signals for an optional direct hard drive connection
  • 4-pin header to provide power to the daughter board (48VDC input power and 3.3VDC system power), alternatively it can be used for powering the 10353 board itself

Elphel 353/363 camera modules

The processor board is accompanied by a number of add-on boards (see detailed description on a wiki page) that can be combined to make different products. Initially we offer two types of the cameras:

  • The Model 353 has the same dimensions as our earlier model 333, and is intended for video applications. It has a 1/2-inch format CMOS image sensor and CS lens mount.
  • The Model 363 (available in Q3 2007, and expected to start at about $4,000) is intended for acquisition of several frames of high resolution still images per second. It includes a 35mm format interline Kodak CCD (11 or 16MPix), and is compatible with standard 35mm lenses, including the electrical interface.


Elphel 353/363 camera modules
(Click to enlarge)

Currently we have two boards that use the inter-board connectors of the 10353:

  • The 10357 board has eight CompactFlash slots, and a reconfigurable FPGA that connects them to the 10353 board's IDE port. With currently available 16GB CF cards, it can provide up to 128GB of solid-state storage for the camera. Initially we plan to use the FPGA as a simple multiplexer of four master/slave IDE devices (CF cards in IDE mode), but it is possible to implement parallel access to the cards (i.e. RAID of some type), and present them to the CPU as a single IDE device. Alternatively this board can be used as a general purpose FPGA I/O device, as most of the FPGA pins go directly to pin headers and can be programmed for custom functions.
  • The 10349 interface board includes RS-232 connector useful as a debug console for software development. By default, it gets all the boot messages and printk output. It also has an IDE connector (2mm pitch, compatible with 2.5″ hard drives), and a USB hub with one external USB connector. In addition to built-in functions, this board can be used for prototyping, since it has multiple solder points and additional pin headers for custom I/Os. Custom prototype boards can be mounted on top of the 10349 with wire connections between the two.

The currently available sensor board for the 10353 is the 10338, which includes a 5MPix Micron CMOS image sensor. The two boards together comprise the electronic part of the Model 353 camera.

The Model 363 camera uses a sandwich of two boards. One is the 10347, which includes an FPGA that performs preliminary image processing, and controls the CCD timing, video ADC, and analog voltages required for the imager operation, as well as providing the electronic interface for the lens and external synchronization. The other is a sensor board — a 10342 board with KAI-11002 for the 11MPix model, and a 10344 with KAI-16000 for the 16MPix model. Both of these boards include high-current CCD phase drivers and DACs for bias voltages.

Another board, the 10359, may be used to connect several sensor boards to a single processor board. It is not just a video multiplexer, but rather has the same 1200K gate FPGA and 64MB SDRAM memory as the processor board, and can reuse some of the Verilog code. This board can be programmed to perform additional image processing before video data reaches the processor board. For instance, it can be used for stereo image processing to extract 3-d information, or track camera orientation using an additional high frame rate/low resolution sensor, in order to compensate for Electronic Rolling Shutter distortion. Such distortion takes place because each line in most CMOS sensors is acquired at a different time, so if the camera is panned horizontally, then vertical objects will look tilted.

The FPGAs on the 10359 and 10347 boards are configured using the JTAG port, with these signals sharing lines in the flex cable with the data signals. One line is dedicated to configuration, while the rest switch to normal operation mode as soon as the FPGA starts.

Camera for web developers

Last year in AJAX, LAMP, and liveDVD for a Linux-based camera I described a web interface for the camera that used JavaScript on the client computer. But while developing this AJAX application I had to “cheat,” because CGI applications that already existed in the camera did not have all the server functionality I needed. So, I had to interrupt JavaScript/HTML coding, and modify/add camera CGI executables and write additional shell scripts to fill the gaps. Of course, this approach would be much more difficult for somebody less familiar with the camera internals. Writing C applications and shell scripts to run in the camera is not really a “rocket science,” but still limits the number of developers who can create web applications for the camera.

It would be much nicer if AJAX developers had familiar tools at their disposal for both client and server programming. Having just upgraded both system memory and the CPU of the camera, I was shopping around for a better solution — even if it would be more expensive (in terms of memory/CPU usage). On the Axis developer WIKI, I found that it is easy to run PHP on ETRAX processors. I tried it, and it really is. The stripped binary turned to be just over 2MB — not a problem for the 64MB system. After some quick patching of the Boa web server that comes as a part of the software distribution for Axis Development Boards, I was happy to see phpinfo() served by the camera.

I did not yet make any benchmarks, but loading 2MB executable for even the simplest request to the camera seemed a little pricey to me. But, this is how CGI works: the program is launched by the web server for each request to it, and generates a response and terminates. The solution to this problem is a well known extension to the CGI specs called FastCGI. FastCGI allows persistent programs to serve multiple requests without termination and reloading. For a big program like PHP, the gain is obvious. Additionally, the same instance of PHP can execute many different scripts.

Unfortunately Boa does not support fastCGI, and instead of modifying it, I looked for alternative servers that already have fastCGI support. I decided to try lighttpd. I started with the latest version (lighttpd-1.5.0-pre), installed it in parallel with the existent boa, and tried to download a big memory-based file with wget. Something was wrong, because lighttpd was nearly five times slower than boa. I tried this and that without any success, and then installed the previous lighttpd release, version 1.4.13. Bingo! Performance was consistently 15 percent faster than the pre-installed boa (8.3MB/s vs. 7.2MB/s). I changed a configure parameter for PHP, recompiled it, tweaked lighttpd config, and phpinfo() replied with “Server API CGI/FastCGI.” So now we have lighttpd+fastCGI+PHP working together in the camera. We still preserved boa and PHP in CGI mode in the camera, though, pending more benchmarks for both servers and CGI vs. FastCGI.

More work is needed to give PHP scripts full control of the camera. Current software (most parts ported from earlier camera models) interfaces with driver parameters using ioctl functions that are not available in PHP. So, while preserving the legacy interface, we are adding alternative access to drivers using PHP-friendly fread() and fwrite() functions.

Community-driven applications

Working on development of network cameras we did not seriously consider Elphel products for filming and cinema applications. But that is changing now because of community initiative. It first started when Phil Stone began to use these cameras (first the model 313, that had a lower frame rate, then later the 333) for outdoor filming of special high-resolution video for bicycle trainers.

Next, we started to feel interest (and some pressure too) from the community of digital video enthusiasts, at least some of whom are eager to hack their cameras and push the limits of the available technology. This audience is very interesting for us, as they are open to experimentation and are able to test and use our cameras before these products are ready for the general customers. Most inspiring is that they are able to use the cameras in ways different from what we designed them for.


Custom DV camera based on Elphel Model 333 (by Daniel Lipats)
(Click to enlarge)

Naturally enough this community only partially intersects with the community of Linux users and not so many of them are experienced programmers. And our attempt to lower the entrance barrier for the camera software development is targeted to suit these particular users too.

I hope that some of the LinuxDevices readers may get interested enough to be able to join the community project of developing a high definition cinema camera (there is a page for it started on Elphel Wiki). Combined expertise has a great potential.

About the Author — Andrey N. Filippov has over 25 years of experience in embedded systems design. Since graduation from the Moscow Institute for Physics and Technology in 1978, he worked for the General Physics Institute (Moscow, Russia) in the area of high-speed high-resolution, mixed signal design, application of PLDs and FPGAs, and microprocessor-based embedded system hardware and software design. Andrey holds a PhD in Physics from the Moscow Institute for Physics and Technology. In 1995 Andrey moved to the United States and after working for Cordin Company (Salt Lake City, Utah) for six years in 2001, he started Elphel, Inc., dedicated to doing business in emerging field of open systems based on free (GNU/GPL) software and open hardware. This photo of the author was made using a Model 303 High Speed Gated Intensified Camera.

More by this author

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.

Related Posts:



Comments are closed.