Video for Linux Two is a set of APIs and standards for handling video devices on Linux. Video for Linux Two is a replacement for the Video for Linux API that comes with the kernel.
This document is the specification for Video for Linux Two video effects
devices. This is a companion document to the V4L2
Devices document.
Devices
A V4L2 video effects device is a device that can do image effects, filtering, or combining of two or more images or image streams, for example video transitions or wipes. Applications send data to be processed and receive the result data either with read() and write() functions, or through memory-mapped buffers. Using memory-mapped buffers is essential if there are two input images.
Refer to the Device document for the device file names used by effects devices.
Multiple Opens per Device
Drivers can support non-I/O opens in addition to an effect context,
or multiple opens where each open is an independent effect device context,
as makes sense for the device.
Query Capabilities - VIDIOC_QUERYCAP
This ioctl call is used to obtain the capability information for the device. See the Device document.
Codec drivers have a type of V4L2_TYPE_FX. Effects devices
will probably support the read() and write() calls indicated by the V4L2_FLAG_READ
and V4L2_FLAG_WRITE flags. An effects driver can also set V4L2_FLAG_STREAMING
if it supports I/O through memory-mapped buffers, and V4L2_FLAG_SELECT
if it supports the select() call.
Enumerating Supported Effects - VIDIOC_ENUMFX
An application can query the list of supported image effects using the
VIDIOC_ENUMFX
ioctl. The application fills in the index field of a struct v4l2_fxdesc,
and then passes it to the VIDIOC_ENUMFX ioctl, which fills in the
rest of the fields. The application should use
index values from
0 on up; the ioctl will return an error when the index goes out of range.
| int index | Number of effects in the list of supported effects | |
| char name[32] | A name for the effects, suitable for a menu on a user interface | |
| int inputs | The number of input images or streams | |
| int controls | The number of controls the effect has | |
| __u32 flags | Effect flags | |
| __u32 reserved[2] | reserved |
Selecting an Effect - VIDIOC_G_EFFECT, VIDIOC_S_EFFECT
Select the desired effect by calling the VIDIOC_S_EFFECT ioctl
with the index of the effect as the parameter. The VIDIOC_G_EFFECT
ioctl takes a pointer to an integer, and returns the currently selected
effect.
Enumerating Supported Image Formats - VIDIOC_ENUM_PIXFMT
An application can query the list of supported image formats with the VIDIOC_ENUM_PIXFMT ioctl. The application fills in the index field of a struct v4l2_fmtdesc, and then passes it to the VIDIOC_ENUM_PIXFMT ioctl, which fills in the rest of the fields. The application should use index values from 0 on up; the ioctl will return an error when the index goes out of range.
The struct v4l2_fmtdesc structure is defined in the Capture
spec.
Selecting The Image Format - VIDIOC_G_FMT, VIDIOC_S_FMT
Refer to the Device document for information regarding the video image format structure, struct v4l2_format. Use V4L2_BUF_TYPE_EFFECTSIN for the type field of v4l2_format. Typically, all streams use the same format, so setting the format on one stream sets it on all streams.
These get and set the format of the images which are going to be processed. All input and output images are the same format. These ioctls work as described in the Capture spec.
Remember that the input data is the data input to the effect, and the output data is the processed data coming out of the effect, which means, paradoxically, you write the input data to the device and read the output data from the device.
If you are planning on using memory mapped device buffers, know that
VIDIOC_S_FMT
is illegal if any device image buffers are currently memory mapped. VIDIOC_S_FMT
undoes the effect of VIDIOC_REQBUFS.
Memory-Mapping Device Buffers - VIDIOC_REQBUFS, VIDIOC_QUERYBUF
See the Device document for instructions on
allocating memory-mapped device buffers. For video data input and output
buffers, use V4L2_BUF_TYPE_EFFECTSIN (, V4L2_BUF_TYPE_EFFECTSIN2,
...) and V4L2_BUF_TYPE_EFFECTSOUT respectively.
Writing and Reading Images - write(), read()
Convert images by sending them to the driver with the write() call, and get the converted images by reading them with read(). Each call to write() or read() should be used to transfer a complete frame. The application may have to write several frames before the first frame is available for reading because a filter or effect may involve combining or processing several images in sequence. The read() function will return the error EPERM if the read cannot complete because more input frames are needed.
If an application is both writing and reading, it should use non-blocking
I/O calls to prevent it from deadlocking. It is possible for one program
to write data while another reads the output. Non-blocking write() will
return the error EAGAIN if a frame or frames need to be read before more
data can be accepted. Non-blocking read() will return the error EAGAIN
if a conversion is in progress, but not yet complete. Note that this is
a different condition from the more-input-frames-needed error condition
mentioned in the previous paragraph, which returns the EPERM code. Read()
and write() do not work if memory-mapped I/O is in use.
I/O Through Pre-Allocated Buffers - VIDIOC_QBUF, VIDIOC_DQBUF, VIDIOC_STREAMON, VIDIOC_STREAMOFF
This mode is supported if the V4L2_FLAG_STREAMING flag is set in the struct v4l2_capability.
This mode uses memory-mapped buffers for transferring video data to and from the driver. First set up the image format, and allocate the buffers as described in the section titled Memory-Mapping Device Buffers in the Device document. You will need at least two sets of buffers, one set for the original data, another for the converted data. Use buffer type V4L2_BUF_TYPE_EFFECTSIN (, V4L2_BUF_TYPE_EFFECTSIN2, ...) for the original data buffers and V4L2_BUF_TYPE_EFFECTSOUT for the converted data buffers.
Streaming data to and from the device with these ioctls works as described
in the Codec spec. Except that an effect may
have more than one input image stream.
Waiting for Frames Using select()
The driver supports the select() call on its file descriptors if the
V4L2_FLAG_SELECT
flag is set in the struct v4l2_capability. If streaming mode is
off, select() returns when there is data ready to be read with the read()
call and/or if the driver is ready to accept new input data through the
write() call. If streaming mode is on, select() returns when there is a
buffer ready to be dequeued. The caller should be sure there is a buffer
queued first.
Controls
See the Controls section of the Device document. Video effects typically have several user-tweakable controls. These controls will be exposed through the V4L2 controls ioctls. In general, the set of controls will be different for each effect, so the application must select the effect before enumerating the controls. The controls field of the struct v4l2_fxdesc object for the effect indicates how many controls the effect has. To enumerate the controls, call VIDIOC_QUERYCTRL with id values starting from V4L2_CID_EFFECT_BASE, and counting up until the indicated number of valid control id's have been identified.
It is normal when streaming data to an effects device to change the
effect controls' values on a frame-by-frame basis on the fly. To do that
you use an effect control stream. The effect control stream works like
the data streams, but the buffers contain control information instead of
pictures. The buffer type is V4L2_BUF_TYPE_FXCONTROL. The effect
control stream runs in parallel with the effects-in stream(s). The format
of the data in the fxcontrol buffer consists of a list of control id-value
pairs, or stated another way the buffer contains an array of struct
v4l2_fxcontrol structures. There may be any number of control values
specified in a buffer. The bytesused field of the v4l2_buffer
will indicate how many controls are in the buffer; the bytesused will be
sizeof(struct v4l2_fxcontrol) * number of controls.
| __u32 id | Control ID | |
| __u32 value | Value for the control for this frame |
Non-I/O Opens
Non-I/O opens can query device properties and operate controls, but
cannot perform I/O operations or change related driver settings such as
the image format.