Bluecherry is hiring an Ubuntu developer!

My old employer, Bluecherry, is looking to hire an Ubuntu developer with the following experience:

• Extensive knowledge of the Video4Linux2 and ALSA sound API
  
• Prior experience in Linux based software design / implementation
  
• Prior knowledge of Ubuntu, including building / maintaining Debian packages
  
• Prior experience with gstreamer and RTSP
  
• Prior experience with MYSQL.
  
• Prior experience with Matroska file containers and video encoding
  
• Excellent verbal and written communication skills
  
• Strong knowledge of C
  
• Previous work with and understanding of working with video / audio formatting / codecs including MPEG4 and H.264
  
• Internet and operating system security fundamentals
  
• Sharp analytical abilities and proven design skills
  
• Strong sense of ownership, urgency, and drive
  
• Demonstrated ability to achieve goals in a highly innovative and fast paced environment
  

Yes it's a long list, but you are replacing me, so get used to high expectations, but a highly rewarding job. You don't need to know all of this, but you should have a good enough foundation that you can embrace the current codebase and not shy away from a steep learning curve. Visit Bluecherry's job posting for more information.

Bluecherry Releases beta1 of their new DVR product

After much delay, and much anticipation, we at Bluecherry have finally released our first public beta of version 2 of our DVR product.

For extensive details, please see the announcement.

I'm honored to have worked on this project. While I've been dealing with the driver (solo6x10) and backend server for over a year, the final parts of the product have fallen into place in just the last 6 months. It was an overwhelming effort by just 4 developers on a project that spanned from hardware to UI.

Solo6x10: Recording from video

I've finally gotten around to writing an example program for recording from Solo6x10 devices to a file. This program is very basic. It leaves the video device in it's default state (resolution, frame rate, etc). So you can modify those settings separately, and then use this program to record at those settings.

I also did not put motion detection examples in this, mainly because I have not satisfied my desire to create a decent API in v4l2 yet for that that.

Next step, I will add sound recording into this.

You can find the example source here.

To compile it, run:

gcc -lavformat bc-record.c -o bc-record

Execute it with a single command line option, the device for a solo6x10 encoder (e.g. /dev/video1). It will record until you hit ^C.

Happy recording!

Using your new Bluecherry MPEG-4 codec card and driver...

Now that the dust has settled and people are taking notice of the new driver for Bluecherry's MPEG-4 codec cards, here's a quick How-To for using it.

You will notice that there are two types of v4l2 devices created for each card. One device for the display port that produces uncompressed YUV and one for each input that produces compressed video in either MPEG-4 or MJPEG.

We'll start with the display port device. When loading the driver, a display port is created as the first device for that card. You can see in dmesg output something like this:

solo6010 0000:03:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
solo6010 0000:03:01.0: Enabled 2 i2c adapters
solo6010 0000:03:01.0: Initialized 4 tw28xx chips: tw2864[4]
solo6010 0000:03:01.0: Display as /dev/video0 with 16 inputs (5 extended)
solo6010 0000:03:01.0: Encoders as /dev/video1-16
solo6010 0000:03:01.0: Alsa sound card as Softlogic0

This is for a 16-port card. The output for a 4-port card would show "Encoders as /dev/video1-4" and similarly for 8-port show /dev/video1-8.

The display port allows you to view and configure what is shown on the video out port of the card. The device has several inputs and depends on which card you have installed:

• 4-port: 1 input per port and 1 virtual input for all 4 inputs in 4-up mode.
• 8-port: 1 input per port and 2 virtual inputs for 4-up on inputs 1-4 and 5-8 respectively.
• 16-port: 1 input per port and 5 virtual inputs for 4-up on inputs 1-5, 5-8, 9-12 and 13-16 and 1 virtual input for 16-up on all inputs.

You do not have to open this device for the video output of the card to work. If you open the device and set the input to input 2, and close it (without viewing any of the video) it will continue to show that input on the video out of the card. So you can simply change inputs using v4l2's ioctl's.

This is useful if you want to have live display to a CRT and use a simple program that rotates through the inputs (or multi-up virtual inputs) a few second intervals.

You can still use vlc, mplayer or whatever to view this device (you can open it multiple times).

Now for the encoder devices. There's obviously one device for each physical input on the card. The driver will allow you to record MPEG-4 and MJPEG from the same device (but you must open it twice, one for each feed). The video format cannot be configured once recording starts. So if you open the device for MPEG-4 and full D1 resultion at 30fps, that's what you're going to get if you also open a simultaneous record for MJPEG.

However, it's good to note here that MJPEG will automatically skip frames when recording. This allows you to pipe the output to a network connection (e.g. MJPEG over HTTP) with no worry of the remote connection being overloaded on bandwidth.

However, this isn't so for MPEG-4. It is possible if you are too slow at recording (not likely) to fall behind the card's internal buffer. I was not able to do this writing the full frames to disk on 44 records (4 cards of 16, 16, 8 and 4 ports).

Unlike any card before this supported by v4l2, the Bluecherry cards produce containerless MPEG-4 frames. Most v4l2 applications expect some sort of MPEG-2 stream such as program or transport. Since these programs do not expect MPEG-4 raw frames, I don't know of any that are capable of playing the encoders directly (much less being able to record from them). You can do something simple like 'cat /dev/video1' and somehow pipe it to vlc (I haven't tested this), or write a program that just writes the frames to disk (I have tested this, most programs can play the raw m4v files produced from the driver).

However, since most people will record to disk, the easiest way is to write the video frames straight out to disk.

Now on to the audio. The cards produce what is known as G.723, which is a voice codec typically found on phone systems (especially VoIP).

Since Alsa currently doesn't have a format for G.723, the driver shows it as unsigned 8-bit PCM audio. However, I can assure you that it isn't. I have sent a patch that was included in alsa-kernel (hopefully getting synced to mainline soon). But this only defines the correct format, it doesn't change the way you handle it at all.

You must convert G.723-24 (3-bit samples at 8khz) yourself. The example program I provide in my next post will show you how to do this, as well as how to convert it to MP2 audio, and record all of this to a container format on disk for later playback.

Softlogic 6010 4/8/16 Channel MPEG-4 Codec Card Driver Released

As I've talked about before, the company I work for has been dedicated to producing stable video surveillance products based on Linux.

Bluecherry's primary device for their video surveillance applications is the Softlogic based MPEG-4 codec card, which is available in 4, 8 and 16 channel models. The original driver for this card, although available as Open Source, was pretty pathetic to say the least. Most of it was just a kludge of the Windows driver, exposing all of the functionality, but with little effort to make it Linux savvy.

That's where I came in. I've since rewritten the driver so that it makes use of Linux's Video4Linux2 and Alsa driver API's. It's currently 90% functional, and many times more efficient than the original OEM driver.

Here is a quick run-down of some of the features and plus-ones against the original driver:

• Video4Linux2 interface allows easy use of existing capture software
• Alsa interface allows for easy audio capture (however, see G.723 caveats from my previous posts)
• Zero-copy in the driver. The original driver DMA'd and then copied the MPEG frames to userspace. The new driver makes use of v4l2 buffers and can DMA directly to an MMAP buffer for userspace.
• Simultaneous MPEG/MJPEG feed per channel, selectable via v4l2 format
• Standard v4l2 uncompressed video YUV display with multi-channel display format (4-up)

Now that the driver is nearing completion, it's about time to release it. I've done so via Launchpad.

• PPA Archive
• Tarball
• git clone git://git.kernel.org/pub/scm/linux/kernel/git/bcollins/solo6x10

If you are on an Ubuntu system, you can install the DKMS package from the PPA archive using these commands:

sudo add-apt-repository ppa:ben-collins/solo6x10
sudo apt-get update
sudo apt-get install solo6010-dkms

Note, I've only supplied this for Lucid right now, but if you download the .deb or the .tar.gz you should be able to install it on any recent kernel.

The joy of writing a php5 module

As a follow up to my last post, I wanted to give a quick update.

As it turns out, I ended up writing a php5 Zend module to wrap up some functions I use to access v4l2 devices. I have to say that writing a php5 module was pretty straight forward. Big thanks to the Extension Writing tutorial I found, which was well written, and did not leave me with any questions.

I was able to get the module ready in a few hours, and spent the rest of this morning cleaning it up and tweaking it a bit.

Now I'm completely able to read my v4l2 devices and mjpeg stream them from my PHP script :)

Request-for-help: PHP and Video4Linux

As it turns out, I do not like writing web applications. Give me registers and DMA, keep the CSS and JS...thanks.

Anyway, I have to find a way to feed an MJPEG output from a PHP script. WAIT! I know this sounds easy, and if not for all the caveats of what I have to adhere to, it would be very simple...maybe.

It seems that PHP doesn't have a way to use ioctl()'s. I can open() a v4l2 device just fine, and read() from it with ease, but your SOL if you want to do something cool with that file handle in PHP.

I need to be able to do ioctl()'s because my v4l2 device requires at least one so I can put it into MJPEG format (as opposed to the default MPEG format). I can serve up these JPEG/MJPEG's through apache perfectly using a C program I've written.

However, I have to use PHP because the rest of the web application is written in PHP and there is already authentication mechanisms storing credentials in PHP sessions. I don't want to have to parse PHP sessions in a C program.

The normal method I found for doing JPEG from a PHP script worked like this:

<?
header("Content-Type: application/jpeg");
passthru("/usr/bin/grabjpeg");
?>

This works perfectly. Except I want to be able to do a MJPEG feed which requires sending one image after the next with headers in between. PHP doesn't like this much, nor does it like the fact that I want all of these headers to come directly from my C program and not from the PHP script. I also do not want to call grabjpeg for each frame, since that's too much overhead in between frames.

What ends up happening is that my headers from the C program are sent as part of the content that the client thinks is the JPEG file.

Right now, I can only see one way to handle this, and that's to write a PHP module to expose libv4l, but I'm open to suggestions on being able to call an ioctl() from within a PHP script.

Review: Softlogic 6010 based MPEG-4/G.723 compression cards

So the company I work for (Bluecherry, LLC) is busy developing some products around the Softlogic 6010 based compression card. My job there has been to rewrite the driver from scratch in order to make it more Linux friendly. So to make things clear, I am writing this review from a programmer's perspective. I want to point out that I am not an MPEG expert, so I may skimp on some of the encoder details.

Let's start of with some specs. The base card supports full D1-quad compression of video into MPEG-4 video format. What this means is that it can encode 704x480 sized video at a rate of 120fps for NTSC, or 704x576 at a rate of 100fps for PAL. This breaks down to 4 full streams at 30fps and 25fps respectively. Alternately it can do CIF encoding (1/4 size of D1) at 4 times that frame rate, or for the math-lazy, 16 channels at 30fps at 320x240 frame size.

The card can be purchased in 4, 8 or 16 channel input models. So to take advantage of all 16 channels on the top model, you would either have to record in CIF mode (320x240) or reduce the frame interval to get 7.5fps per channel for full D1 mode (704x480). I will be speaking mostly in NTSC, but the card does support PAL, so do the conversions as we go.

The card allows for the usual MPEG encoding settings including GOP (Group of Pictures), Quantization and Intervals. Intervals are sort of the opposite of frames-per-second, but correlate the same way. An interval of 1 means that the encoder captures every frame, while an interval of 3 means it skips 2 frames between every frame it encodes. The video muxer on the card performs at 30fps, so the interval setting will decide how many of these frames get encoded.

The encoder itself performs quite well. It performs all encoding to an on-board SDRAM chip, and can DMA the frames directly to the host memory, which is great for performance. The original driver did not take advantage of this since it copied the frames to user space. The new driver I've written makes use of v4l2 and it's videobuf-dma-contiguous framework and thus allows for memory mapped buffers with userspace. This gives us zero-copy to userspace.

The encoder also supports side-by-side MJPEG compression of video frames. So while you can be recording the compressed MPEG-4 to disk, you can also frame grab JPEG images. This is useful for tools that want to do such frame grabbing for video analytics, or for live viewing over a web server (it's very easy to send frame grabs via an MJPEG cgi script).

All of this is built properly now on top of Linux's v4l2 API. Unfortunately the API does not expect compression cards to pipe MPEG-4 video, so most clients using v4l2 expect compressed video to be either MJPEG or MPEG-1/2 streams of some sort.

Currently the only drawback from the MPEG encoder is that the frames are self-standing MPEG-4 video frames. I have to add a header to the key frames for them to be usable by most decoders.

Overall the video capture is great. I've run 44 simultaneous records (16, 16, 8, 4 channel cards) on a Core2Duo with a system load average of 1.65, and only about 10% CPU usage. Most of the load is disk I/O.

Each encoder input also supports a graphical overlay that can be programmed at pixel level with varying colors. This is great for textual overlays. Currently we use it to place a descriptive name on the recording along with a timestamp.

In addition to the encoders, the card supports one uncompressed display port. It's currently exposed via v4l2 as a standard analog YUV device. It can be configured to show any of the inputs ports in tons of configurations. So you can do things like a 4-up display. This live display also supports a graphical overlay.

The display is sent to the video-out port on the card (hard-wired), so it can be hooked to a monitor as well (good for surveillance applications such as what Bluecherry offers).

Finally we'll discuss my least favorite part of this card. While it's not a killer, it is just odd that the card supports sound only in G.723 format. For surveillance applications this is just fine. Delivering 3-bit samples at 8Khz sample rate, which is a 24kbs. While this is good for bandwidth, it's bad for anything that needs better audio quality. Not to mention that storing the audio and video together in any sane format requires converting G.723 to linear PCM.

However, the G.723 to linear PCM conversion isn't much overhead on performance, and neither is the encoding to 16Khz MP2 audio, which is how we store it for our surveillance products. Overall, our format is MPEG-4/Video and MP2/Audio in a Matroska/mkv container. This is exactly how it was stored in my 44 stream example above.

So Pros:

• Fast and efficient
  
• Can handle multiple inputs easily
  
• The new driver works well with v4l2 and alsa
  
• Perfect for security applications
  
• Nice OSD capabilities
  
• Motion detection supported per input
  
• Side-by-side MPEG-4 and JPEG capture modes per input
  

Cons:

• MPEG-4 video. The SOLO-6110 will support H.264
  
• Low quality audio is not great for anything other than special applications (no TV DVR)
  
• G.723 audio format has been obsoleted twice since it was introduced. Nothing uses it so you must always re-encode it.
  

Video4Linux2 Hardware Motion Detection Support

In about a week or so I'll be making a proposal for V4L2 to have API support for hardware that offers motion detection. Since my experience with this is limited to only one type of hardware, I'm hoping to gain feedback on making sure that the approach I'm offering is as generic as possible.

I'll describe the hardware that I'm working with, which is a Softlogic 6010 MPEG4/G.723 encoder board supporting 4, 8 and 16 input channels (and all of them being able to be encoded at once). Note that all of this applies to the SOLO-6110 card as well (h.264 variant).

The motion detection exposed by the SOLO-6010 is on a per-input basis. It can be configured, when motion detection is enabled, to either signal start of motion events only, or signal start and stop events with a configurable delay after actual motion has stopped (i.e. it will not send the stop signal until there is no motion for n amount of seconds).

Next, SOLO-6010 allows you to set a threshold for when the hardware will detect an event. In my case, the higher the threshold, the less sensitive. It has a range of 0 (anal) to 65535 (off) with a default of 768.

Exposing this via v4l2 controls is quite simple. In my current version of the solo6010 driver, I expose this via private CIDs (Control IDs) which can be easily converted to native CIDs in v4l2.

#define V4L2_CID_MOTION_ENABLE    (V4L2_CID_PRIVATE_BASE+0)
#define V4L2_CID_MOTION_THRESHOLD (V4L2_CID_PRIVATE_BASE+1)
#define V4L2_CID_MOTION_MODE      (V4L2_CID_PRIVATE_BASE+2)
#define V4L2_CID_MOTION_EASE_OFF  (V4L2_CID_PRIVATE_BASE+3)

In this case, V4L2_CID_MOTION_ENABLE is a boolean to turn motion detection on or off, V4L2_CID_MOTION_THRESHOLD is the threshold value I spoke of (slider with said range), V4L2_CID_MOTION_MODE is a menu control for "Start events only" and "Start and stop events" and V4L2_CID_MOTION_EASE_OFF is the seconds of non-motion required before the stop event is triggered.

Now, I could combine V4L2_CID_MOTION_ENABLE and V4L2_CID_MOTION_MODE as just a menu control with "Disabled" as one option, but I'm not sure what the consensus would be. It would be confusing as a standard control for hardware that only supported on/off tuning of this feature.

Note that in "Start event only" mode, my hardware will continually produce motion events when the card sees it, and thus I can emulate V4L2_CID_MOTION_EASE_OFF and a stop event in software.

Whether it is a good idea to always have this support from the control, and have the v4l2 middle layer take care of using the hardware or it's own software to handle it, is up for discussion. I'm all for implementing it as transparent to the user, with the middle-layer handling the guts and the drivers deciding if they allow the middle-layer to do it, or expose the hardware support for it.

Now we just need to make userspace aware of these events. I've found the easiest way for me was to define some extra flags for struct v4l2_buffer that get set during dqbuf.

#define V4L2_BUF_FLAG_MOTION_ON         0x00000400
#define V4L2_BUF_FLAG_MOTION_START      0x00000800
#define V4L2_BUF_FLAG_MOTION_STOP       0x00001000

The reason for V4L2_BUF_FLAG_MOTION_ON is because we need userspace to be able to tell that motion detection is on without querying the controls every second or two. Remember that controls can be changed even while a recorder is on (and in the case of motion detection, I suspect that's a wanted feature).

So if userspace is reading packets, it knows that motion detection is on or off depending on that flag, and can act accordingly. The start and stop flags are self-explanatory.

Now, this is a good reason to promote software side (perhaps libv4l2?) ease-off on motion detection. Without creating another flag, there's no way to know if motion detection is in a start-only or start-stop mode. If we always implement the ease-off, then we know we'll get a stop event eventually, whether or not the hardware supports it.

Moving back to threshold values...SOLO-6010 actually supports a motion detection grid with block sizes of 32x32 pixels. SOLO-6010s NTSC viewable field is 704x480, 704x576 for PAL. So that's either a 22x15 or 22x18 grid of blocks that can have individual threshold settings each. I'm still up in the air about how to do this in a standard v4l2 API. For SOLO-6010 I am using the low 16-bits of the control value to pass back a threshold level, and the high 16-bits to determine the block being affected (0xff000000 being the x value and 0x00ff0000 being the y value on the grid). This works well in practice but is obviously not generic enough.

Well that's all I have for today.