Video4Linux2 Hardware Motion Detection Support

In about a week or so I'll be making a proposal for V4L2 to have API support for hardware that offers motion detection. Since my experience with this is limited to only one type of hardware, I'm hoping to gain feedback on making sure that the approach I'm offering is as generic as possible.

I'll describe the hardware that I'm working with, which is a Softlogic 6010 MPEG4/G.723 encoder board supporting 4, 8 and 16 input channels (and all of them being able to be encoded at once). Note that all of this applies to the SOLO-6110 card as well (h.264 variant).

The motion detection exposed by the SOLO-6010 is on a per-input basis. It can be configured, when motion detection is enabled, to either signal start of motion events only, or signal start and stop events with a configurable delay after actual motion has stopped (i.e. it will not send the stop signal until there is no motion for n amount of seconds).

Next, SOLO-6010 allows you to set a threshold for when the hardware will detect an event. In my case, the higher the threshold, the less sensitive. It has a range of 0 (anal) to 65535 (off) with a default of 768.

Exposing this via v4l2 controls is quite simple. In my current version of the solo6010 driver, I expose this via private CIDs (Control IDs) which can be easily converted to native CIDs in v4l2.

#define V4L2_CID_MOTION_ENABLE    (V4L2_CID_PRIVATE_BASE+0)
#define V4L2_CID_MOTION_THRESHOLD (V4L2_CID_PRIVATE_BASE+1)
#define V4L2_CID_MOTION_MODE      (V4L2_CID_PRIVATE_BASE+2)
#define V4L2_CID_MOTION_EASE_OFF  (V4L2_CID_PRIVATE_BASE+3)

In this case, V4L2_CID_MOTION_ENABLE is a boolean to turn motion detection on or off, V4L2_CID_MOTION_THRESHOLD is the threshold value I spoke of (slider with said range), V4L2_CID_MOTION_MODE is a menu control for "Start events only" and "Start and stop events" and V4L2_CID_MOTION_EASE_OFF is the seconds of non-motion required before the stop event is triggered.

Now, I could combine V4L2_CID_MOTION_ENABLE and V4L2_CID_MOTION_MODE as just a menu control with "Disabled" as one option, but I'm not sure what the consensus would be. It would be confusing as a standard control for hardware that only supported on/off tuning of this feature.

Note that in "Start event only" mode, my hardware will continually produce motion events when the card sees it, and thus I can emulate V4L2_CID_MOTION_EASE_OFF and a stop event in software.

Whether it is a good idea to always have this support from the control, and have the v4l2 middle layer take care of using the hardware or it's own software to handle it, is up for discussion. I'm all for implementing it as transparent to the user, with the middle-layer handling the guts and the drivers deciding if they allow the middle-layer to do it, or expose the hardware support for it.

Now we just need to make userspace aware of these events. I've found the easiest way for me was to define some extra flags for struct v4l2_buffer that get set during dqbuf.

#define V4L2_BUF_FLAG_MOTION_ON         0x00000400
#define V4L2_BUF_FLAG_MOTION_START      0x00000800
#define V4L2_BUF_FLAG_MOTION_STOP       0x00001000

The reason for V4L2_BUF_FLAG_MOTION_ON is because we need userspace to be able to tell that motion detection is on without querying the controls every second or two. Remember that controls can be changed even while a recorder is on (and in the case of motion detection, I suspect that's a wanted feature).

So if userspace is reading packets, it knows that motion detection is on or off depending on that flag, and can act accordingly. The start and stop flags are self-explanatory.

Now, this is a good reason to promote software side (perhaps libv4l2?) ease-off on motion detection. Without creating another flag, there's no way to know if motion detection is in a start-only or start-stop mode. If we always implement the ease-off, then we know we'll get a stop event eventually, whether or not the hardware supports it.

Moving back to threshold values...SOLO-6010 actually supports a motion detection grid with block sizes of 32x32 pixels. SOLO-6010s NTSC viewable field is 704x480, 704x576 for PAL. So that's either a 22x15 or 22x18 grid of blocks that can have individual threshold settings each. I'm still up in the air about how to do this in a standard v4l2 API. For SOLO-6010 I am using the low 16-bits of the control value to pass back a threshold level, and the high 16-bits to determine the block being affected (0xff000000 being the x value and 0x00ff0000 being the y value on the grid). This works well in practice but is obviously not generic enough.

Well that's all I have for today.