AJAX, LAMP, and liveDVD for a Linux-based camera
Mar 27, 2006 — by LinuxDevices Staff — from the LinuxDevices Archive — 11 viewsForeword: This article explains how a small, Utah-based company used asynchronous Javascript and XML (AJAX) to add advanced features to the user interfaces in its Linux-based cameras. The article was written by Andrey Filippov, a Russian physicist who emigrated to the US in 1995, and founded Elphel six years later. Enjoy . . . !
Background
Previously, I have written about our cameras with a focus on the technology of the images and video acquisition and compression. These articles covered:
- The Model 303, a high speed gated Intensified camera that used embedded Linux running on an Axis ETRAX100LX processor
- The Model 313, which incorporates a high performance reconfigurable FPGA
- The Model 333, which was the first to combine high resolution and frame rate with the low bit rate of the Ogg Theora video format
I have not previously written about the software that runs our cameras, though. In specific, this article covers the most important part of the software — that which is visible to human users of the cameras.
Web interface for devices
Several years before I started working on network cameras, I was already sure that web interfaces are ideal for embedded devices. No software to install — nice for the users — and it's easier to write some HTML, instead of programming all these user interface elements in executable programs, and then making sure they support major OSes and all their versions. Originally, I was impressed by this idea when I got a Toshiba ISDN router — it still works in my basement, although only for phone lines now, since the Internet was switched to ADSL, which was not available back then.
When I started the network cameras project, I did not have any doubt as to the nature of the interface software: the GNU/Linux software for the Axis ETRAX100LX processor even included a web server that was able to serve html pages and images, and interface with CGI programs and scripts that ran in the camera.
That was nice — as soon as the hardware was ready, it did not take me long to modify the software, add the JPEG library, and have basic camera functionality that supported passing acquisition parameters from a simple GET method HTML form and receiving images back. Problems started later, however, when I needed to add some otherwise basic functionality that was not easily available with this technology.
Asynchronous JavaScript
I wanted to receive not just images, but usable data, such as status information. I did not expect a problem with such a simple task, but it proved difficult. For example, the camera can be used for capturing single events that are triggered externally, by sensors, for example. The first approach I tried was to request an image from the camera. The camera was armed by this request, but would not send an image until after the trigger event had happened. After an interval, the browser would abandon waiting, and there was no easy way to reset the waiting — not a really nice solution.
So it could be much better it the client side JavaScript code could use some polling of (if not interrupts from) the camera. I'm sure many other developers in the pre-AJAX era had similar problems, and were using similar solutions like I did. Our model 303 gated camera used JavaScript and dynamically generated HTML and CSS to show/hide menus and other interface elements. It also used a 1-pixel high HTML frame that had no visible elements — just some JavaScript code. This page was refreshed once a second from the camera where the CGI application was able to communicate data to the browser by JavaScript assignments like document.parameter1=value1;. It seemed to work nicely with Netscape in 2001, but had a fatal problem with Internet Explorer (I tried 5.x, 6.0) that I failed to find a fix (or detour) for — after several thousands refreshes, the computer was loosing the network completely, and any running applications, such as email programs, were getting a “no sockets available” error.
Next were “XML islands.” These were used with the Cortona VRML viewer plugin to manipulate 3D models of laser beam intensity (z-axis) generated as compressed VRML code right in the camera. But it was an IE-specific feature, so I abandoned it later, as this browser did not run on the computers I normally use.
I switched back to the method that worked with Netscape (and then Mozilla, and later Firefox). But about two years ago I received an invitation to Gmail, tried it, and realized that it was doing exactly what I was looking for. And it worked not just with the IE, but with Mozilla, such as I was using. And that was called…
AJAX
What exactly was new for us there? It was XMLHttpRequest, which works nicely with most browsers, only requiring a small “if”-branch in JavaScript to combine code for IE and other browsers (a familiar trick to most web designers). In the “camvc” program that provides the interface for current Elphel cameras, this method is used to control image acquisition in the camera, video compression, and steaming. XMLHttpRequest is also used for simpler tasks, such as editing the camera's internal configuration files. The interface for this is provided by JavaScript running on the client, rather than by some inflexible forms provided by the camera scripts.
JavaScript programs running in modern browsers on modern computers can do really nice things, and provide familiar GUIs with all different kinds of buttons, sliders, tabs, draggable and resizable frames, and translucent layers.
Camvc screenshot
(Click to enlarge)
Of course this program/web page has response time higher than those of binary applications, and dragging image frames is not as smooth as I would like to see it. However, computers are constantly getting faster, so this kind of improvement will come without any special effort on the designer's side. Development time is rather short, and as a result, easy-to-reuse JavaScript is now a mainstream technology used by many. It is easy to cut-and-paste code and reuse it in other applications — see JavaScript tips and code links below.
There is a Boa web server in the camera that provides an interface to the camera CGI applications that control image sensor operation, video processing and compression, and streaming. As noted earlier, CGI applications also perform supplemental tasks, such as editing data in the camera internal configuration files.
Acquired video data is compressed in the FPGA, and then either served as individual images or uploaded to the remote server. Or, it can be sent out as a continuous RTP video stream (multi- or unicast). The camera can also send individual captured frames without interrupting in-progress streams. Static images are rendered by the browser itself, which supports overlaying the images with semi-transparent controls that can can be minimized, leaving a single row of buttons in the top left corner.
Camera and a client
Embedding video
Naturally, the camera control page incorporates a video viewer. We used a powerful and flexible program called Mplayer, together with the GenReS plugin for Mozilla browsers. This plugin allows embedding of the video into the browser window, and provides a two-directional interface through JavaScript functions, variables, and events, making it suitable for use as an integral part of an AJAX application.
High-bandwidth video data reaches the player, bypassing the web server in the camera and browser on the client computer. We use an “ElphelOgm” application in a pipe with Mplayer to perform pre-filtering of the video, including reducing the frame rate by dropping a calculated number of frames on slow computers, while still allowing full frame rate video to be recorded, in parallel with viewing.
Using the video viewer does require additional software to be installed on the client computer. That limits the portability of the solution somewhat. However, the videoplayer and the plugin for embedding it within the browser window are general-purpose pieces of software that are not directly tied with our cameras. That means that in the future, we may be able to incorporate enhancements created by others, rather than having to develop everything ourselves.
So, right now, we have a working solution for systems running GNU/Linux. For those who do not use GNU/Linux, we have another solution: preinstalled software on a…
Live GNU/Linux disk
This liveCD/DVD is based on a popular distribution, known as Knoppix, that does an excellent job of running on most types of computers, and is easy to use for those who have no experience with GNU/Linux. Custom additions to our Knoppix liveCD include network camera discovery, preinstalled Mplayer with RTP support, a Mozilla plugin for embedding Mplayer, ElphelOgm video recording software, and several utilities. One utility, a fail-safe program to update/restore camera firmware via http, utilizes a low-level network boot protocol of Axis processors.
Video recording
A camera that only allows you to watch video is not very useful — we need to be able to record video, and play it back. Originally we used Mplayer's companion program — Mencoder — but later developed ElphelOgm in order to use OGM rather than AVI file formats, and to reduce the CPU load (Mencoder is designed to do complex video transcoding, and is overkill for just recording video streams).
There still are some problems in using a browser to manipulate video recording, even when the recorder programs itself via AJAX. This is because browsers are designed to keep web page code as far from local storage as possible — for the sake of security. Of course it is possible to solve this problem using browser plugins and helper applications, but such solutions lead to having more non-portable code, and the user interface (if it is a helper application, not the plugin) is out of control of the AJAX GUI developer, because it is defined by the helper application itself.
Luckily there is a rather easy way to integrate the video recording utility in such a way that it provides flexibility and allows the web page developer full control of the GUI. Our approach also provides the additional benefit of supporting remote recording. This solution uses…
DVR on LAMP
Camera, client, and the DVR
It is much more natural to interact with mass storage through the web server than through web browsers. And, the Knoppix distribution comes with an Apache web server. It also provides a MySQL database, useful for handling video record metadata, and PHP, a scripting language that can be used to easily glue together applications, databases, and the web server. We used these programs together to create a DVR server application.
In a minimal system, the DVR application physically coexists with the browser on the same computer, but it is also possible to separate them. It is possible, for example, to have camera(s) and the DVR server reside on the same fast LAN, but the client can be connected through a lower bandwidth channel. This application suggests additional services to be implemented on the DVR side — such as a PHP-wrapper for Mencoder that makes it possible to download preview versions of stored video (like thumbnail images make with ImageMagick). Another PHP-wrapper for Mencoder allows transcoding to different video formats using available codecs.
That seemed nice and easy, but when I tried to make an AJAX application communicate with both the camera (when the camvc control page is loaded in the browser) and the DVR server running on localhost — the second connection was refused by Firefox, for security reasons. It turned out to be a cross-domain XMLHttpRequest (the rejected connection is shown as a dashed line on the diagram). This error can be easily avoided, however, by reducing the security settings in the browser, or using the camera as a proxy between the client and the DVR server. The XML traffic between the client and the server is only needed for the metadata (video files during playback go directly to the client), so the extra bandwidth and the camera CPU load overhead is not high.
Credits
I would like to express my gratitude to Klaus Knopper for Knoppix, a solid foundation for our live GNU/Linux distributions (and many others', too), Diego Biurrun and Alex Beregszaszi of the MPlayer team, and Elphel developers Alexander Melichenko, Mike Malishko, Sergey Khlutchin, and Andrey Latin, who made this project possible.
Availability
Elphel Model 333 cameras are available for $800 (C/CS-mount lens not included), with $100 per camera discount for those who develop software for the camera under GNU/GPL or a compatible license.
Used in the camvc program. I believe that some of the solutions found are rather universal and can be applied to the different designs.
- Cover controls images (buttons, sliders, draggable frames) with transparent (empty) layers (
div> tags). This prevents the images from being treated by the browsers like documents that can be dragged (i.e. you try to drag a slider pointer and instead the document icon appears by the mouse pointer that holds the url of the slider image). This method is universal and works with all browsers. Other solution could be the use of background-image instead of the img/> tag, but they can not be stretched if needed.
- Get rid of a huge number of the individual images for the buttons (each button needs at least two images — one for pressed and one for released states) by having a single large image that holds all the images placed side-by-side and shown through a small window. There is a library to handle different types of buttons in elphelButtons.js – graphic buttons use a single 2-dimensional array of images.
- Translucency (does not work with current versions of IE – probably will be available in the next version) is implemented by layers having backgrounds, consisting of 1×1 pixel semi-transparent PNG files. Camvc uses 0%/25%/50%/75%/100% transparency, for 25, 50 and 75 there are individual images for each of the colors used and the background file name is calculated from the color and transparency value – here are the ones used in the program.
- When developing sliders for camvc I realised that mouse coordinates are available with the screen pixel resolution and sometimes it is not enough, so our sliders now combine coarse and fine adjustments. When you drag it by the bottom half of the pointer it behaves as a regular slider, when by the top – the slider pointer lags behind the mouse pointer so the controlled parameter is adjusted in smaller steps. Similar idea is used in the 2-dimensional frames that need to be positioned with high precision – dragging them by the corners works as normal, by the sides — allows fine resizing (it is needed when the frame works as a magnifying glass for the background image).
- Color semitransparent tabs are dynamically generated by the program. It is not practical to have static HTML code — the curved portion of the overlapping tabs consist of one pixel wide layers, needed to provide the combination of intuitive look and translucency.
Andrey N. Filippov has over 25 years of experience in the embedded systems design. Since graduation from the Moscow Institute for Physics and Technology in 1978 he worked for the General Physics Institute (Moscow, Russia) in the area of high-speed high-resolution,mixed signal design, application of PLDs and FPGAs, and microprocessor-based embedded system hardware and software design. Andrey holds a PhD in Physics from the Moscow Institute for Physics and Technology. In 1995 Andrey moved to the United States and after working for Cordin Company (Salt Lake City, Utah) for 6 years in 2001 started Elphel, Inc. dedicated to doing business in emerging field of open systems based on free (GNU/GPL) software and open hardware. This photo of the author was made using a Model 303 High Speed Gated Intensified Camera.
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.