16. API - mmalobj¶
This module provides an object-oriented interface to libmmal
which is the
library underlying picamera, raspistill
, and raspivid
. It is provided
to ease the usage of libmmal
to Python coders unfamiliar with C and also
works around some of the idiosyncrasies in libmmal
.
Warning
This part of the API is still experimental and subject to change in future versions. Backwards compatibility is not (yet) guaranteed.
16.1. The MMAL Tour¶
MMAL operates on the principle of pipelines:
- A pipeline consists of one or more MMAL components
(
MMALBaseComponent
and derivatives) connected together in series. - A
MMALBaseComponent
has one or more ports. - A port (
MMALControlPort
and derivatives) is either a control port, an input port or an output port (there are also clock ports but you generally don’t need to deal with these as MMAL sets them up automatically):- Control ports are used to accept and receive commands, configuration parameters, and error messages. All MMAL components have a control port, but in picamera they’re only used for component configuration.
- Input ports receive data from upstream components.
- Output ports send data onto downstream components (if they’re connected), or to callback routines in the user’s program (if they’re not connected).
- Input and output ports can be audio, video or sub-picture (subtitle) ports, but picamera only deals with video ports.
- Ports have a
format
which (in the case of video ports) dictates the format of image/frame accepted or generated by the port (YUV, RGB, JPEG, H.264, etc.) - Video ports have a
framerate
which specifies the number of images expected to be received or sent per second. - Video ports also have a
framesize
which specifies the resolution of images/frames accepted or generated by the port. - Finally, all ports (control, input and output) have
params
which affect their operation.
- An output port can have a
MMALConnection
to an input port. Connections ensure the two ports use compatible formats, and handle transferring data from output ports to input ports in an orderly fashion. A port cannot have more than one connection from/to it. - Data is written to / read from ports via instances of
MMALBuffer
.- Buffers belong to a port and can’t be passed arbitrarily between ports.
- The size of a buffer is dictated by the format and frame-size of the port
that owns the buffer. The memory allocation of a buffer (readable from
size
) cannot be altered once the port is enabled, but the buffer can contain any amount of data up its allocation size. The actual length of data in a buffer is stored inlength
. - Likewise, the number of buffers belonging to a port is fixed and cannot be altered without disabling the port, reconfiguring it and re-enabling it. The more buffers a port has, the less likely it is that the pipeline will have to drop frames because a component has overrun, but the more GPU memory is required.
- Buffers also have
flags
which specify information about the data they contain (e.g. start of frame, end of frame, key frame, etc.) - When a connection exists between two ports, the connection continually requests a buffer from the output port, requests another buffer from the input port, copies the output buffer’s data to the input buffer’s data, then returns the buffers to their respective ports (this is a simplification; various tricks are pulled under the covers to minimize the amount of data copying that actually occurs, but as a mental model of what’s going on it’s reasonable).
- Components take buffers from their input port(s), process them, and write the result into a buffer from the output port(s).
16.1.1. Components¶
Now we’ve got a mental model of what an MMAL pipeline consists of, let’s build one. For the rest of the tour I strongly recommend using a Pi with a screen (so you can see preview output) but controlling it via an SSH session (so the preview doesn’t cover your command line). Follow along, typing the examples into your remote Python session. And feel free to deviate from the examples if you’re curious about things!
We’ll start by importing the mmalobj
module with a convenient
alias, then construct a MMALCamera
component, and a
MMALRenderer
component.
>>> from picamera import mmal, mmalobj as mo
>>> camera = mo.MMALCamera()
>>> preview = mo.MMALRenderer()
16.1.2. Ports¶
Before going any further, let’s have a look at the various ports on these components.
>>> len(camera.inputs)
0
>>> len(camera.outputs)
3
>>> len(preview.inputs)
1
>>> len(preview.outputs)
0
The fact the camera has three outputs should come as little surprise to those who have read the Camera Hardware chapter (if you haven’t already, you might want to skim it now). Let’s examine the first output port of the camera and the input port of the renderer:
>>> camera.outputs[0]
<MMALVideoPort "vc.ril.camera:out:0": format=MMAL_FOURCC('I420')
buffers=1x7680 frames=320x240@0fps>
>>> preview.inputs[0]
<MMALVideoPort "vc.ril.video_render:in:0" format=MMAL_FOURCC('I420')
buffers=2x15360 frames=160x64@0fps>
Several things to note here:
- We can tell from the port name what sort of component it belongs to, what its index is, and whether it’s an input or an output port
- Both ports are currently configured for the I420 format; this is MMAL’s name for YUV420 (full resolution Y, quarter resolution UV).
- The ports have different frame-sizes (320x240 and 160x64 respectively), buffer counts (1 and 2 respectively) and buffer sizes (7680 and 15360 respectively).
- The buffer sizes look unrealistic. For example, 7680 bytes is nowhere near enough to hold 320 * 240 * 1.5 bytes (YUV420 requires 1.5 bytes per pixel).
Now we’ll configure the camera’s output port with a slightly higher resolution, and give it a frame-rate:
>>> camera.outputs[0].framesize = (640, 480)
>>> camera.outputs[0].framerate = 30
>>> camera.outputs[0].commit()
>>> camera.outputs[0]
<MMALVideoPort "vc.ril.camera:out:0(I420)": format=MMAL_FOURCC('I420')
buffers=1x460800 frames=640x480@30fps>
Note that the changes to the configuration won’t actually take effect until
the commit()
call. After the port is committed, note that the
buffer size now looks reasonable: 640 * 480 * 1.5 = 460800.
16.1.3. Connections¶
Now we’ll try connecting the renderer’s input to the camera’s output. Don’t
worry about the fact that the port configurations are different. One of the
nice things about MMAL (and the mmalobj
layer) is that connections try very
hard to auto-configure things so that they “just work”. Usually,
auto-configuration is based upon the output port being connected so it’s
important to get that configuration right, but you don’t generally need to
worry about the input port.
The renderer is what mmalobj
terms a “downstream component”. This is a
component with a single input that typically sits downstream from some feeder
component (like a camera). All such components have the
connect()
method which can be used to connect the
sole input to a specified output:
>>> preview.connect(camera)
<MMALConnection "vc.ril.camera:out:0/vc.ril.video_render:in:0">
>>> preview.connection.enable()
Note that we’ve been quite lazy in the snippet above by simply calling
connect()
with the camera
component. In this case, a
connection will be attempted between the first input port of the owner
(preview
) and the first unconnected output of the parameter (camera
).
However, this is not always what’s wanted so you can specify the exact ports
you wish to connect. In this case the example was equivalent to calling:
>>> preview.inputs[0].connect(camera.outputs[0])
<MMALConnection "vc.ril.camera:out:0/vc.ril.video_render:in:0">
>>> preview.inputs[0].connection.enable()
Note that the connect()
method returns the connection
that was constructed but you can also retrieve this by querying the port’s
connection
attribute later.
As soon as the connection is enabled you should see the camera preview appear on the Pi’s screen. Let’s query the port configurations now:
>>> camera.outputs[0]
<MMALVideoPort "vc.ril.camera:out:0(OPQV)": format=MMAL_FOURCC('OPQV')
buffers=10x128 frames=640x480@30fps>
>>> preview.inputs[0]
<MMALVideoPort "vc.ril.video_render:in:0(OPQV)": format=MMAL_FOURCC('OPQV')
buffers=10x128 frames=640x480@30fps>
Note that the connection has implicitly reconfigured the camera’s output port to use the OPAQUE (“OPQV”) format. This is a special format used internally by the camera firmware which avoids passing complete frame data around, instead passing pointers to frame data around (this explains the tiny buffer size of 128 bytes as very little data is actually being shuttled between the components). Further, note that the connection has automatically copied the port format, frame size and frame-rate to the preview’s input port.
16.1.4. Opaque Format¶
At this point it is worth exploring the differences between the camera’s three output ports:
- Output 0 is the “preview” output. On this port, the OPAQUE format contains a pointer to a complete frame of data.
- Output 1 is the “video recording” output. On this port, the OPAQUE format contains a pointer to two complete frames of data. The dual-frame format enables the H.264 video encoder to calculate motion estimation without the encoder having to keep copies of prior frames itself (it can do this when something other than OPAQUE format is used, but dual-image OPAQUE is much more efficient).
- Output 2 is the “still image” output. On this port, the OPAQUE format contains a pointer to a strip of an image. The “strips” format is used by the JPEG encoder (not to be confused with the MJPEG encoder) to deal with high resolution images efficiently.
Generally, you don’t need to worry about these differences. The mmalobj
layer knows about them and negotiates the most efficient format it can for
connections. However, they’re worth bearing in mind if you’re aiming to get the
most out of the firmware or if you’re confused about why a particular format
has been selected for a connection.
16.1.5. Component Configuration¶
So far we’ve seen how to construct components, configure their ports, and connect them together in rudimentary pipelines. Now, let’s see how to configure components via control port parameters:
>>> camera.control.params[mmal.MMAL_PARAMETER_SYSTEM_TIME]
177572014208
>>> camera.control.params[mmal.MMAL_PARAMETER_SYSTEM_TIME]
177574350658
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS]
Fraction(1, 2)
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS] = 0.75
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS]
Fraction(3, 4)
>>> fx = camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT]
>>> fx
<picamera.mmal.MMAL_PARAMETER_IMAGEFX_T object at 0x765b8440>
>>> dir(fx)
['__class__', '__ctypes_from_outparam__', '__delattr__', '__dict__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
'__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__',
'__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__',
'__weakref__', '_b_base_', '_b_needsfree_', '_fields_', '_objects', 'hdr',
'value']
>>> fx.value
0
>>> mmal.MMAL_PARAM_IMAGEFX_NONE
0
>>> fx.value = mmal.MMAL_PARAM_IMAGEFX_EMBOSS
>>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = fx
>>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS] = 1/2
>>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = mmal.MMAL_PARAM_IMAGEFX_NONE
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pi/picamera/picamera/mmalobj.py", line 1109, in __setitem__
assert mp.hdr.id == key
AttributeError: 'int' object has no attribute 'hdr'
>>> fx.value = mmal.MMAL_PARAM_IMAGEFX_NONE
>>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = fx
>>> preview.disconnect()
Things to note:
- The parameter dictates the type of the value returned (and accepted, if the parameter is read-write).
- Many parameters accept a multitude of simple types like
int
,float
,Fraction
,str
, etc. However, some parameters usectypes
structures and such parameters only accept the relevant structure. - The easiest way to use such “structured” parameters is to query them first, modify the resulting structure, then write it back to the parameter.
To find out what parameters are available for use with the camera component,
have a look at the source for the PiCamera
class, especially
property getters and setters.
16.1.6. File Output (RGB capture)¶
Let’s see how we can produce some file output from the camera. First we’ll perform a straight unencoded RGB capture from the still port (2). As this is unencoded output we don’t need to construct anything else. All we need to do is configure the port for RGB encoding, select an appropriate resolution, then activate the output port:
>>> camera.outputs[2].format = mmal.MMAL_ENCODING_RGB24
>>> camera.outputs[2].framesize = (640, 480)
>>> camera.outputs[2].commit()
>>> camera.outputs[2]
<MMALVideoPort "vc.ril.camera:out:2(RGB3)": format=MMAL_FOURCC('RGB3')
buffers=1x921600 frames=640x480@0fps>
>>> camera.outputs[2].enable()
Unfortunately, that didn’t seem to do much! An output port that is participating in a connection needs nothing more: it knows where its data is going. However, an output port without a connection requires a callback function to be assigned so that something can be done with the buffers of data it produces.
The callback will be given two parameters: the MMALPort
responsible
for producing the data, and the MMALBuffer
containing the data. It is
expected to return a bool
which will be False
if further data is
expected and True
if no further data is expected. If True
is returned,
the callback will not be executed again. In our case we’re going to write data
out to a file we’ll open before-hand, and we should return True
when we see
a buffer with the “frame end” flag set:
>>> camera.outputs[2].disable()
>>> import io
>>> output = io.open('image.data', 'wb')
>>> def image_callback(port, buf):
... output.write(buf.data)
... return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
...
>>> camera.outputs[2].enable(image_callback)
>>> output.tell()
0
At this stage you may note that while the file exists, nothing’s been written to it. This is because output ports 1 and 2 (the video and still ports) won’t produce any buffers until their “capture” parameter is enabled:
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
>>> output.tell()
921600
>>> camera.outputs[2].disable()
>>> output.close()
Congratulations! You’ve just captured your first image with the MMAL layer. Given we disconnected the preview above, the current state of the system looks something like this:
16.1.7. File Output (JPEG capture)¶
Whilst RGB is a useful format for processing we’d generally prefer something like JPEG for output. So, next we’ll construct an MMAL JPEG encoder and use it to compress our RGB capture. Note that we’re not going to connect the JPEG encoder to the camera yet; we’re just going to construct it standalone and feed it data from our capture file, writing the output to another file:
>>> encoder = mo.MMALImageEncoder()
>>> encoder.inputs
(<MMALVideoPort "vc.ril.image_encode:in:0": format=MMAL_FOURCC('RGB2')
buffers=1x15360 frames=96x80@0fps>,)
>>> encoder.outputs
(<MMALVideoPort "vc.ril.image_encode:out:0": format=MMAL_FOURCC('GIF ')
buffers=1x81920 frames=0x0@0fps>,)
>>> encoder.inputs[0].format = mmal.MMAL_ENCODING_RGB24
>>> encoder.inputs[0].framesize = (640, 480)
>>> encoder.inputs[0].commit()
>>> encoder.outputs[0].copy_from(encoder.inputs[0])
>>> encoder.outputs[0]
<MMALVideoPort "vc.ril.image_encode:out:0": format=MMAL_FOURCC('RGB3')
buffers=1x81920 frames=640x480@0fps>
>>> encoder.outputs[0].format = mmal.MMAL_ENCODING_JPEG
>>> encoder.outputs[0].commit()
>>> encoder.outputs[0]
<MMALVideoPort "vc.ril.image_encode:out:0(JPEG)": format=MMAL_FOURCC('JPEG')
buffers=1x307200 frames=0x0@0fps>
>>> encoder.outputs[0].params[mmal.MMAL_PARAMETER_JPEG_Q_FACTOR] = 90
Just pausing for a moment, let’s re-cap what we’ve got: an image encoder constructed, configured for 640x480 RGB input, and JPEG output with a quality factor of “90” (i.e. “very good” - don’t try to read much more than this into JPEG quality settings!). Note that MMAL has set the buffer size at a size it thinks will be typical for the output. As JPEG is a lossy format this won’t be precise and it’s entirely possible that we may receive multiple callbacks for a single frame (if the compression overruns the expected buffer size).
Let’s continue:
>>> rgb_data = io.open('image.data', 'rb')
>>> jpg_data = io.open('image.jpg', 'wb')
>>> def image_callback(port, buf):
... jpg_data.write(buf.data)
... return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
...
>>> encoder.outputs[0].enable(image_callback)
16.1.8. File Input (JPEG encoding)¶
How do we feed data to a component without a connection? We enable its input port with a dummy callback (we don’t need to “do” anything on data input). Then we request buffers from its input port, fill them with data and send them back to the input port:
>>> encoder.inputs[0].enable(lamdba port, buf: True)
>>> buf = encoder.inputs[0].get_buffer()
>>> buf.data = rgb_data.read()
>>> encoder.inputs[0].send_buffer(buf)
>>> jpg_data.tell()
87830
>>> encoder.outputs[0].disable()
>>> encoder.inputs[0].disable()
>>> jpg_data.close()
>>> rgb_data.close()
Congratulations again! You’ve just produced a hardware-accelerated JPEG encoding. The following illustrates the state of the system at the moment (note the camera and renderer still exist; they’re just not connected to anything at the moment):
Now let’s repeat the process but with the encoder attached to the
still port on the camera directly. We can re-use our image_callback
routine
from earlier and just assign a different output file to jpg_data
:
>>> encoder.connect(camera.outputs[2])
<MMALConnection "vc.ril.camera:out:2/vc.ril.image_encode:in:0">
>>> encoder.connection.enable()
>>> encoder.inputs[0]
<MMALVideoPort "vc.ril.image_encode:in:0(OPQV)": format=MMAL_FOURCC('OPQV')
buffers=10x128 frames=640x480@0fps>
>>> jpg_data = io.open('direct.jpg', 'wb')
>>> encoder.outputs[0].enable(image_callback)
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
>>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
>>> jpg_data.tell()
99328
>>> encoder.connection.disable()
>>> jpg_data.close()
Now the state of our system looks like this:
16.1.9. Threads & Synchronization¶
The one issue you may have noted is that image_callback
is running in a
background thread. If we were running our capture extremely fast our main
thread might disable the capture before our callback had run. Ideally we want
to activate capture, wait on some signal indicating that the callback has
completed a single frame successfully, then disable capture. We can do this
with the communications primitives from the standard threading
module:
>>> from threading import Event
>>> finished = Event()
>>> def image_callback(port, buf):
... jpg_data.write(buf.data)
... if buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END:
... finished.set()
... return True
... return False
...
>>> def do_capture(filename='direct.jpg'):
... global jpg_data
... jpg_data = io.open(filename, 'wb')
... finished.clear()
... encoder.outputs[0].enable(image_callback)
... camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
... if not finished.wait(10):
... raise Exception('capture timed out')
... camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
... encoder.outputs[0].disable()
... jpg_data.close()
...
>>> do_capture()
The above example has several rough edges: globals, no proper clean-up in the case of an exception, etc. but by now you should be getting a pretty good idea of how picamera operates under the hood.
The major difference between picamera and a “typical” MMAL setup is that upon
construction, the PiCamera
class constructs both a
MMALCamera
(accessible as PiCamera._camera
) and a
MMALSplitter
(accessible as PiCamera._splitter
). The splitter
remains permanently attached to the camera’s video port (output port 1).
Furthermore, there’s always something connected to the camera’s preview port;
by default it’s a MMALNullSink
component which is switched with a
MMALRenderer
when the preview is started.
Encoders are constructed and destroyed as required by calls to
capture()
, start_recording()
,
etc. The following illustrates a typical picamera pipeline whilst video
recording without a preview:
16.1.10. Debugging Facilities¶
Before we move onto the pure Python components it’s worth mentioning the
debugging capabilities built into mmalobj
. Firstly, most objects have
useful repr()
outputs (in particular, it can be useful to simply evaluate
a MMALBuffer
to see what flags it’s got and how much data is stored in
it). Also, there’s the print_pipeline()
function. Give this a port and
it’ll dump a human-readable version of your pipeline leading up to that port:
>>> preview.inputs[0].enable(lambda port, buf: True)
>>> buf = preview.inputs[0].get_buffer()
>>> buf
<MMALBuffer object: flags=_____ length=0>
>>> buf.flags = mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END
>>> buf
<MMALBuffer object: flags=E____ length=0>
>>> buf.release()
>>> preview.inputs[0].disable()
>>> mo.print_pipeline(encoder.outputs[0])
vc.ril.camera [2] [0] vc.ril.image_encode [0]
encoding OPQV-strips --> OPQV-strips encoding JPEG
buf 10x128 10x128 buf 1x307200
bitrate 0bps 0bps bitrate 0bps
frame 640x480@0fps 640x480@0fps frame 0x0@0fps
16.1.11. Python Components¶
So far all the components we’ve looked at have been “real” MMAL components which is to say that they’re implemented in C, and all talk to bits of the firmware running on the GPU. However, a frequent request has been to be able to modify frames from the camera before they reach the image or video encoder. The Python components are an attempt to make this request relatively simple to achieve from within Python.
The means by which this is achieved are inefficient (to say the least) so don’t
expect this to work with high resolutions or framerates. The mmalobj
layer
in picamera includes the concept of a “Python MMAL” component. To the user
these components look a lot like the MMAL components you’ve been playing with
above (MMALCamera
, MMALImageEncoder
, etc). They are
instantiated in a similar manner, they have the same sort of ports, and they’re
connected using the same means as ordinary MMAL components.
Let’s try this out by placing a transformation between the camera and a preview
which will draw a cross over the frames going to the preview. For this we’ll
subclass picamera.array.PiArrayTransform
. This derives from
MMALPythonComponent
and provides the useful capability of
providing the source and target buffers as numpy arrays containing RGB data:
>>> from picamera import array
>>> class Crosshair(array.PiArrayTransform):
... def transform(self, source, target):
... with source as sdata, target as tdata:
... tdata[...] = sdata
... tdata[240, :, :] = 0xff
... tdata[:, 320, :] = 0xff
... return False
...
>>> transform = Crosshair()
That’s all there is to constructing a transform! This one is a bit crude in as much as the coordinates are hard-coded, and it’s very simplistic, but it should illustrate the principle nicely. Let’s connect it up between the camera and the renderer:
>>> transform.connect(camera)
<MMALPythonConnection "vc.ril.camera.out:0(RGB3)/py.component:in:0">
>>> preview.connect(transform)
<MMALPythonConnection "py.component:out:0/vc.ril.video_render:in:0(RGB3)">
>>> transform.connection.enable()
>>> preview.connection.enable()
>>> transform.enable()
At this point we should take a look at the pipeline to see what’s been configured automatically:
>>> mo.print_pipeline(preview.inputs[0])
vc.ril.camera [0] [0] py.transform [0] [0] vc.ril.video_render
encoding RGB3 --> RGB3 encoding RGB3 --> RGB3 encoding
buf 1x921600 2x921600 buf 2x921600 2x921600 buf
frame 640x480@30fps 640x480@30fps frame 640x480@30fps 640x480@30fps frame
Apparently the MMAL camera component is outputting RGB data (which is extremely large) to a “py.transform” component, which draws our cross-hair on the buffer and passes it onto the renderer again as RGB. This is part of the inefficiency alluded to above: RGB is a very large format (compared to I420 which is half its size, or OPQV which is tiny) so we’re shuttling a lot of data around here. Expect this to drop frames at higher resolutions or framerates.
The other source of inefficiency isn’t obvious from the debug output above
which gives the impression that the “py.transform” component is actually part
of the MMAL pipeline. In fact, this is a lie. Under the covers mmalobj
installs an output callback on the camera’s output port to feed data to the
“py.transform” input port, uses a background thread to run the transform, then
copies the results into buffers obtained from the preview’s input port. In
other words there’s really two (very short) MMAL pipelines with a hunk of
Python running in between them. If mmalobj
does its job properly you
shouldn’t need to worry about this implementation detail but it’s worth bearing
in mind from the perspective of performance.
16.1.12. Performance Hints¶
Generally you want to your frame handlers to be fast. To avoid dropping frames they’ve got to run in less than a frame’s time (e.g. 33ms at 30fps). Bear in mind that a significant amount of time is going to be spent shuttling the huge RGB frames around so you’ve actually got much less than 33ms available to you (how much will depend on the speed of your Pi, what resolution you’re using, the framerate, etc).
Sometimes, performance can mean making unintuitive choices. For example, the
Pillow library (the main imaging library in Python these days) can construct
images which share buffer memory (see Image.frombuffer
), but only for the
indexed (grayscale) and RGBA formats, not RGB. Hence, it can make sense to use
RGBA (a format even larger than RGB) if only because it allows you to avoid
copying any data when performing a composite.
Another trick is to realize that although YUV420 has different sized planes, it’s often enough to manipulate the Y plane only. In that case you can treat the front of the buffer as an indexed image (remember that Pillow can share buffer memory with such images) and manipulate that directly. With tricks like these it’s possible to perform multiple composites in realtime at 720p30 on a Pi3.
Here’s a (heavily commented) variant of the cross-hair example above that uses
the lower level MMALPythonComponent
class instead, and the Pillow
library to perform compositing on YUV420 in the manner just described:
from picamera import mmal, mmalobj as mo, PiCameraPortDisabled
from PIL import Image, ImageDraw
from signal import pause
class Crosshair(mo.MMALPythonComponent):
def __init__(self):
super(Crosshair, self).__init__(name='py.crosshair')
self._crosshair = None
self.inputs[0].supported_formats = mmal.MMAL_ENCODING_I420
def _handle_frame(self, port, buf):
# If we haven't drawn the crosshair yet, do it now and cache the
# result so we don't bother doing it again
if self._crosshair is None:
self._crosshair = Image.new('L', port.framesize)
draw = ImageDraw.Draw(self._crosshair)
draw.line([
(port.framesize.width // 2, 0),
(port.framesize.width // 2, port.framesize.height)],
fill=(255,), width=1)
draw.line([
(0, port.framesize.height // 2),
(port.framesize.width , port.framesize.height // 2)],
fill=(255,), width=1)
# buf is the buffer containing the frame from our input port. First
# we try and grab a buffer from our output port
try:
out = self.outputs[0].get_buffer(False)
except PiCameraPortDisabled:
# The port was disabled; that probably means we're shutting down so
# return True to indicate we're all done and the component should
# be disabled
return True
else:
if out:
# We've got a buffer (if we don't get a buffer here it most
# likely means things are going too slow downstream so we'll
# just have to skip this frame); copy the input buffer to the
# output buffer
out.copy_from(buf)
# now grab a locked reference to the buffer's data by using
# "with"
with out as data:
# Construct a PIL Image over the Y plane at the front of
# the data and tell PIL the buffer is writeable
img = Image.frombuffer('L', port.framesize, data, 'raw', 'L', 0, 1)
img.readonly = False
img.paste(self._crosshair, (0, 0), mask=self._crosshair)
# Send the output buffer back to the output port so it can
# continue onward to whatever's downstream
try:
self.outputs[0].send_buffer(out)
except PiCameraPortDisabled:
# The port was disabled; same as before this probably means
# we're shutting down so return True to indicate we're done
return True
# Return False to indicate that we want to continue processing
# frames. If we returned True here, the component would be
# disabled and no further buffers would be processed
return False
camera = mo.MMALCamera()
preview = mo.MMALRenderer()
transform = Crosshair()
camera.outputs[0].framesize = '720p'
camera.outputs[0].framerate = 30
camera.outputs[0].commit()
transform.connect(camera)
preview.connect(transform)
transform.connection.enable()
preview.connection.enable()
preview.enable()
transform.enable()
camera.enable()
pause()
It’s a sensible idea to perform any overlay rendering you want to do in a
separate thread and then just handle compositing your overlay onto the frame in
the MMALPythonComponent._handle_frame()
method. Anything you can do to
avoid buffer copying is a bonus here.
Here’s a final (rather large) demonstration that puts all these things together
to construct a MMALPythonComponent
derivative with two purposes:
- Render a partially transparent analogue clock in the top left of the frame.
- Produces two equivalent I420 outputs; one for feeding to a preview renderer, and another to an encoder (we could use a proper MMAL splitter for this but this is a demonstration of how Python components can have multiple outputs too).
import io
import datetime as dt
from threading import Thread, Lock
from collections import namedtuple
from math import sin, cos, pi
from time import sleep
from picamera import mmal, mmalobj as mo, PiCameraPortDisabled
from PIL import Image, ImageDraw
class Coord(namedtuple('Coord', ('x', 'y'))):
@classmethod
def clock_arm(cls, radians):
return Coord(sin(radians), -cos(radians))
def __add__(self, other):
try:
return Coord(self.x + other[0], self.y + other[1])
except TypeError:
return Coord(self.x + other, self.y + other)
def __sub__(self, other):
try:
return Coord(self.x - other[0], self.y - other[1])
except TypeError:
return Coord(self.x - other, self.y - other)
def __mul__(self, other):
try:
return Coord(self.x * other[0], self.y * other[1])
except TypeError:
return Coord(self.x * other, self.y * other)
def __floordiv__(self, other):
try:
return Coord(self.x // other[0], self.y // other[1])
except TypeError:
return Coord(self.x // other, self.y // other)
# yeah, I could do the rest (truediv, radd, rsub, etc.) but there's no
# need here...
class ClockSplitter(mo.MMALPythonComponent):
def __init__(self):
super(ClockSplitter, self).__init__(name='py.clock', outputs=2)
self.inputs[0].supported_formats = {mmal.MMAL_ENCODING_I420}
self._lock = Lock()
self._clock_image = None
self._clock_thread = None
def enable(self):
super(ClockSplitter, self).enable()
self._clock_thread = Thread(target=self._clock_run)
self._clock_thread.daemon = True
self._clock_thread.start()
def disable(self):
super(ClockSplitter, self).disable()
if self._clock_thread:
self._clock_thread.join()
self._clock_thread = None
with self._lock:
self._clock_image = None
def _clock_run(self):
# draw the clock face up front (no sense drawing that every time)
origin = Coord(0, 0)
size = Coord(100, 100)
center = size // 2
face = Image.new('L', size)
draw = ImageDraw.Draw(face)
draw.ellipse([origin, size - 1], outline=(255,))
while self.enabled:
# loop round rendering the clock hands on a copy of the face
img = face.copy()
draw = ImageDraw.Draw(img)
now = dt.datetime.now()
midnight = now.replace(
hour=0, minute=0, second=0, microsecond=0)
timestamp = (now - midnight).total_seconds()
hour_pos = center + Coord.clock_arm(2 * pi * (timestamp % 43200 / 43200)) * 30
minute_pos = center + Coord.clock_arm(2 * pi * (timestamp % 3600 / 3600)) * 45
second_pos = center + Coord.clock_arm(2 * pi * (timestamp % 60 / 60)) * 45
draw.line([center, hour_pos], fill=(200,), width=2)
draw.line([center, minute_pos], fill=(200,), width=2)
draw.line([center, second_pos], fill=(200,), width=1)
# assign the rendered image to the internal variable
with self._lock:
self._clock_image = img
sleep(0.2)
def _handle_frame(self, port, buf):
try:
out1 = self.outputs[0].get_buffer(False)
out2 = self.outputs[1].get_buffer(False)
except PiCameraPortDisabled:
return True
if out1:
# copy the input frame to the first output buffer
out1.copy_from(buf)
with out1 as data:
# construct an Image using the Y plane of the output
# buffer's data and tell PIL we can write to the buffer
img = Image.frombuffer('L', port.framesize, data, 'raw', 'L', 0, 1)
img.readonly = False
with self._lock:
if self._clock_image:
img.paste(self._clock_image, (10, 10), self._clock_image)
# if we've got a second output buffer replicate the first
# buffer into it (note the difference between replicate and
# copy_from)
if out2:
out2.replicate(out1)
try:
self.outputs[0].send_buffer(out1)
except PiCameraPortDisabled:
return True
if out2:
try:
self.outputs[1].send_buffer(out2)
except PiCameraPortDisabled:
return True
return False
def main(output_filename):
camera = mo.MMALCamera()
preview = mo.MMALRenderer()
encoder = mo.MMALVideoEncoder()
clock = ClockSplitter()
target = mo.MMALPythonTarget(output_filename)
# Configure camera output 0
camera.outputs[0].framesize = (640, 480)
camera.outputs[0].framerate = 24
camera.outputs[0].commit()
# Configure H.264 encoder
encoder.outputs[0].format = mmal.MMAL_ENCODING_H264
encoder.outputs[0].bitrate = 2000000
encoder.outputs[0].commit()
p = encoder.outputs[0].params[mmal.MMAL_PARAMETER_PROFILE]
p.profile[0].profile = mmal.MMAL_VIDEO_PROFILE_H264_HIGH
p.profile[0].level = mmal.MMAL_VIDEO_LEVEL_H264_41
encoder.outputs[0].params[mmal.MMAL_PARAMETER_PROFILE] = p
encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_INLINE_HEADER] = True
encoder.outputs[0].params[mmal.MMAL_PARAMETER_INTRAPERIOD] = 30
encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_INITIAL_QUANT] = 22
encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_MAX_QUANT] = 22
encoder.outputs[0].params[mmal.MMAL_PARAMETER_VIDEO_ENCODE_MIN_QUANT] = 22
# Connect everything up and enable everything (no need to enable capture on
# camera port 0)
clock.inputs[0].connect(camera.outputs[0])
preview.inputs[0].connect(clock.outputs[0])
encoder.inputs[0].connect(clock.outputs[1])
target.inputs[0].connect(encoder.outputs[0])
target.connection.enable()
encoder.connection.enable()
preview.connection.enable()
clock.connection.enable()
target.enable()
encoder.enable()
preview.enable()
clock.enable()
try:
sleep(10)
finally:
# Disable everything and tear down the pipeline
target.disable()
encoder.disable()
preview.disable()
clock.disable()
target.inputs[0].disconnect()
encoder.inputs[0].disconnect()
preview.inputs[0].disconnect()
clock.inputs[0].disconnect()
if __name__ == '__main__':
main('output.h264')
16.1.13. IO Classes¶
The Python MMAL components include a couple of useful IO classes:
MMALSource
and MMALTarget
. We could have used these instead
of messing around with output callbacks in the sections above but it was worth
exploring how those callbacks operated first (in order to comprehend how
Python transforms operated).
16.2. Components¶
-
class
picamera.mmalobj.
MMALBaseComponent
[source]¶ Represents a generic MMAL component. Class attributes are read to determine the component type, and the OPAQUE sub-formats of each connectable port.
-
close
()[source]¶ Close the component and release all its resources. After this is called, most methods will raise exceptions if called.
-
enable
()[source]¶ Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.
-
control
¶ The
MMALControlPort
control port of the component which can be used to configure most aspects of the component’s behaviour.
-
-
class
picamera.mmalobj.
MMALCamera
[source]¶ Bases:
picamera.mmalobj.MMALBaseComponent
Represents the MMAL camera component. This component has 0 input ports and 3 output ports. The intended use of the output ports (which in turn determines the behaviour of those ports) is as follows:
- Port 0 is intended for preview renderers
- Port 1 is intended for video recording
- Port 2 is intended for still image capture
Use the
MMAL_PARAMETER_CAMERA_CONFIG
parameter on the control port to obtain and manipulate the camera’s configuration.-
annotate_rev
¶ The annotation capabilities of the firmware have evolved over time and several structures are available for querying and setting video annotations. By default the
MMALCamera
class will pick the latest annotation structure supported by the current firmware but you can select older revisions withannotate_rev
for other purposes (e.g. testing).
-
class
picamera.mmalobj.
MMALCameraInfo
[source]¶ Bases:
picamera.mmalobj.MMALBaseComponent
Represents the MMAL camera-info component. Query the
MMAL_PARAMETER_CAMERA_INFO
parameter on the control port to obtain information about the connected camera module.-
info_rev
¶ The camera information capabilities of the firmware have evolved over time and several structures are available for querying camera information. When initialized,
MMALCameraInfo
will attempt to discover which structure is in use by the extant firmware. This property can be used to discover the structure version and to modify the version in use for other purposes (e.g. testing).
-
-
class
picamera.mmalobj.
MMALComponent
[source]¶ Bases:
picamera.mmalobj.MMALBaseComponent
Represents an MMAL component that acts as a filter of some sort, with a single input that connects to an upstream source port. This is an asbtract base class.
-
close
()[source]¶ Close the component and release all its resources. After this is called, most methods will raise exceptions if called.
-
connect
(source, **options)[source]¶ Connects the input port of this component to the specified source
MMALPort
orMMALPythonPort
. Alternatively, as a convenience (primarily intended for command line experimentation; don’t use this in scripts), source can be another component in which case the first unconnected output port will be selected as source.Keyword arguments will be passed along to the connection constructor. See
MMALConnection
andMMALPythonConnection
for further information.
-
disconnect
()[source]¶ Destroy the connection between this component’s input port and the upstream component.
-
enable
()[source]¶ Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.
-
connection
¶ The
MMALConnection
orMMALPythonConnection
object linking this component to the upstream component.
-
-
class
picamera.mmalobj.
MMALSplitter
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents the MMAL splitter component. This component has 1 input port and 4 output ports which all generate duplicates of buffers passed to the input port.
-
class
picamera.mmalobj.
MMALResizer
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents the MMAL VPU resizer component. This component has 1 input port and 1 output port. This supports resizing via the VPU. This is not as efficient as
MMALISPResizer
but is available on all firmware verions. The output port can (and usually should) have a different frame size to the input port.
-
class
picamera.mmalobj.
MMALISPResizer
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents the MMAL ISP resizer component. This component has 1 input port and 1 output port, and supports resizing via the VideoCore ISP, along with conversion of numerous formats into numerous other formats (e.g. OPAQUE to RGB, etc). This is more efficient than
MMALResizer
but is only available on later firmware versions.
-
class
picamera.mmalobj.
MMALEncoder
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents a generic MMAL encoder. This is an abstract base class.
-
class
picamera.mmalobj.
MMALVideoEncoder
[source]¶ Bases:
picamera.mmalobj.MMALEncoder
Represents the MMAL video encoder component. This component has 1 input port and 1 output port. The output port is usually configured with
MMAL_ENCODING_H264
orMMAL_ENCODING_MJPEG
.
-
class
picamera.mmalobj.
MMALImageEncoder
[source]¶ Bases:
picamera.mmalobj.MMALEncoder
Represents the MMAL image encoder component. This component has 1 input port and 1 output port. The output port is typically configured with
MMAL_ENCODING_JPEG
but can also useMMAL_ENCODING_PNG
,MMAL_ENCODING_GIF
, etc.
-
class
picamera.mmalobj.
MMALDecoder
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents a generic MMAL decoder. This is an abstract base class.
-
class
picamera.mmalobj.
MMALVideoDecoder
[source]¶ Bases:
picamera.mmalobj.MMALDecoder
Represents the MMAL video decoder component. This component has 1 input port and 1 output port. The input port is usually configured with
MMAL_ENCODING_H264
orMMAL_ENCODING_MJPEG
.
-
class
picamera.mmalobj.
MMALImageDecoder
[source]¶ Bases:
picamera.mmalobj.MMALDecoder
Represents the MMAL iamge decoder component. This component has 1 input port and 1 output port. The input port is usually configured with
MMAL_ENCODING_JPEG
.
-
class
picamera.mmalobj.
MMALRenderer
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents the MMAL renderer component. This component has 1 input port and 0 output ports. It is used to implement the camera preview and overlays.
-
class
picamera.mmalobj.
MMALNullSink
[source]¶ Bases:
picamera.mmalobj.MMALComponent
Represents the MMAL null-sink component. This component has 1 input port and 0 output ports. It is used to keep the preview port “alive” (and thus calculating white-balance and exposure) when the camera preview is not required.
16.3. Ports¶
-
class
picamera.mmalobj.
MMALControlPort
(port)[source]¶ Represents an MMAL port with properties to configure the port’s parameters.
-
enable
(callback=None)[source]¶ Enable the port with the specified callback function (this must be
None
for connected ports, and a callable for disconnected ports).The callback function must accept two parameters which will be this
MMALControlPort
(or descendent) and anMMALBuffer
instance. Any return value will be ignored.
-
capabilities
¶ The capabilities of the port. A bitfield of the following:
- MMAL_PORT_CAPABILITY_PASSTHROUGH
- MMAL_PORT_CAPABILITY_ALLOCATION
- MMAL_PORT_CAPABILITY_SUPPORTS_EVENT_FORMAT_CHANGE
-
enabled
¶ Returns a
bool
indicating whether the port is currently enabled. Unlike other classes, this is a read-only property. Useenable()
anddisable()
to modify the value.
-
index
¶ Returns an integer indicating the port’s position within its owning list (inputs, outputs, etc.)
-
params
¶ The configurable parameters for the port. This is presented as a mutable mapping of parameter numbers to values, implemented by the
MMALPortParams
class.
-
type
¶ The type of the port. One of:
- MMAL_PORT_TYPE_OUTPUT
- MMAL_PORT_TYPE_INPUT
- MMAL_PORT_TYPE_CONTROL
- MMAL_PORT_TYPE_CLOCK
-
-
class
picamera.mmalobj.
MMALPort
(port, opaque_subformat='OPQV')[source]¶ Bases:
picamera.mmalobj.MMALControlPort
Represents an MMAL port with properties to configure and update the port’s format. This is the base class of
MMALVideoPort
,MMALAudioPort
, andMMALSubPicturePort
.-
commit
()[source]¶ Commits the port’s configuration and automatically updates the number and size of associated buffers according to the recommendations of the MMAL library. This is typically called after adjusting the port’s format and/or associated settings (like width and height for video ports).
-
connect
(other, **options)[source]¶ Connect this port to the other
MMALPort
(orMMALPythonPort
). The type and configuration of the connection will be automatically selected.Various connection options can be specified as keyword arguments. These will be passed onto the
MMALConnection
orMMALPythonConnection
constructor that is called (see those classes for an explanation of the available options).
-
copy_from
(source)[source]¶ Copies the port’s
format
from the sourceMMALControlPort
.
-
enable
(callback=None)[source]¶ Enable the port with the specified callback function (this must be
None
for connected ports, and a callable for disconnected ports).The callback function must accept two parameters which will be this
MMALControlPort
(or descendent) and anMMALBuffer
instance. The callback should returnTrue
when processing is complete and no further calls are expected (e.g. at frame-end for an image encoder), andFalse
otherwise.
-
get_buffer
(block=True, timeout=None)[source]¶ Returns a
MMALBuffer
from the associatedpool
. block and timeout act as they do in the correspondingMMALPool.get_buffer()
.
-
send_buffer
(buf)[source]¶ Send
MMALBuffer
buf to the port.
-
bitrate
¶ Retrieves or sets the bitrate limit for the port’s format.
-
buffer_count
¶ The number of buffers allocated (or to be allocated) to the port. The
mmalobj
layer automatically configures this based on recommendations from the MMAL library.
-
buffer_size
¶ The size of buffers allocated (or to be allocated) to the port. The size of buffers is typically dictated by the port’s format. The
mmalobj
layer automatically configures this based on recommendations from the MMAL library.
-
connection
¶ If this port is connected to another, this property holds the
MMALConnection
orMMALPythonConnection
object which represents that connection. If this port is not connected, this property isNone
.
-
format
¶ Retrieves or sets the encoding format of the port. Setting this attribute implicitly sets the encoding variant to a sensible value (I420 in the case of OPAQUE).
After setting this attribute, call
commit()
to make the changes effective.
-
opaque_subformat
¶ Retrieves or sets the opaque sub-format that the port speaks. While most formats (I420, RGBA, etc.) mean one thing, the opaque format is special; different ports produce different sorts of data when configured for OPQV format. This property stores a string which uniquely identifies what the associated port means for OPQV format.
If the port does not support opaque format at all, set this property to
None
.MMALConnection
uses this information when negotiating formats for a connection between two ports.
-
supported_formats
¶ Retrieves a sequence of supported encodings on this port.
-
-
class
picamera.mmalobj.
MMALVideoPort
(port, opaque_subformat='OPQV')[source]¶ Bases:
picamera.mmalobj.MMALPort
Represents an MMAL port used to pass video data.
-
class
picamera.mmalobj.
MMALSubPicturePort
(port, opaque_subformat='OPQV')[source]¶ Bases:
picamera.mmalobj.MMALPort
Represents an MMAL port used to pass sub-picture (caption) data.
-
class
picamera.mmalobj.
MMALAudioPort
(port, opaque_subformat='OPQV')[source]¶ Bases:
picamera.mmalobj.MMALPort
Represents an MMAL port used to pass audio data.
-
class
picamera.mmalobj.
MMALPortParams
(port)[source]¶ Represents the parameters of an MMAL port. This class implements the
MMALControlPort.params
attribute.Internally, the class understands how to convert certain structures to more common Python data-types. For example, parameters that expect an MMAL_RATIONAL_T type will return and accept Python’s
Fraction
class (or any other numeric types), while parameters that expect an MMAL_BOOL_T type will treat anything as a truthy value. Parameters that expect the MMAL_PARAMETER_STRING_T structure will be treated as plain strings, and likewise MMAL_PARAMETER_INT32_T and similar structures will be treated as plain ints.Parameters that expect more complex structures will return and expect those structures verbatim.
16.4. Connections¶
-
class
picamera.mmalobj.
MMALBaseConnection
(source, target, formats=())[source]¶ Abstract base class for
MMALConnection
andMMALPythonConnection
. Handles weakrefs to the source and target ports, and format negotiation. All other connection details are handled by the descendent classes.-
source
¶ The source
MMALPort
orMMALPythonPort
of the connection.
-
target
¶ The target
MMALPort
orMMALPythonPort
of the connection.
-
-
class
picamera.mmalobj.
MMALConnection
(source, target, formats=default_formats, callback=None)[source]¶ Bases:
picamera.mmalobj.MMALBaseConnection
Represents an MMAL internal connection between two components. The constructor accepts arguments providing the source
MMALPort
and targetMMALPort
.The formats parameter specifies an iterable of formats (in preference order) that the connection may attempt when negotiating formats between the two ports. If this is
None
, or an empty iterable, no negotiation will take place and the source port’s format will simply be copied to the target port. Otherwise, the iterable will be worked through in order until a format acceptable to both ports is discovered.Note
The default formats list starts with OPAQUE; the class understands the different OPAQUE sub-formats (see MMAL for more information) and will only select OPAQUE if compatible sub-formats can be used on both ports.
The callback parameter can optionally specify a callable which will be executed for each buffer that traverses the connection (providing an opportunity to manipulate or drop that buffer). If specified, it must be a callable which accepts two parameters: the
MMALConnection
object sending the data, and theMMALBuffer
object containing data. The callable may optionally manipulate theMMALBuffer
and return it to permit it to continue traversing the connection, or returnNone
in which case the buffer will be released.Note
There is a significant performance penalty for specifying a callback between MMAL components as it requires buffers to be copied from the GPU’s memory to the CPU’s memory and back again.
-
default_formats
= (MMAL_ENCODING_OPAQUE, MMAL_ENCODING_I420, MMAL_ENCODING_RGB24, MMAL_ENCODING_BGR24, MMAL_ENCODING_RGBA, MMAL_ENCODING_BGRA)¶ Class attribute defining the default formats used to negotiate connections between MMAL components.
-
16.5. Buffers¶
-
class
picamera.mmalobj.
MMALBuffer
(buf)[source]¶ Represents an MMAL buffer header. This is usually constructed from the buffer header pointer and is largely supplied to make working with the buffer’s data a bit simpler. Using the buffer as a context manager implicitly locks the buffer’s memory and returns the
ctypes
buffer object itself:def callback(port, buf): with buf as data: # data is a ctypes uint8 array with size entries print(len(data))
Alternatively you can use the
data
property directly, which returns and modifies the buffer’s data as abytes
object (note this is generally slower than using the buffer object unless you are simply replacing the entire buffer):def callback(port, buf): # the buffer contents as a byte-string print(buf.data)
-
acquire
()[source]¶ Acquire a reference to the buffer. This will prevent the buffer from being recycled until
release()
is called. This method can be called multiple times in which case an equivalent number of calls torelease()
must be made before the buffer will actually be released.
-
copy_from
(source)[source]¶ Copies all fields (including data) from the source
MMALBuffer
. This buffer must have sufficientsize
to storelength
bytes from the source buffer. This method implicitly setsoffset
to zero, andlength
to the number of bytes copied.Note
This is fundamentally different to the operation of the
replicate()
method. It is much slower, but afterward the copied buffer is entirely independent of the source.
-
copy_meta
(source)[source]¶ Copy meta-data from the source
MMALBuffer
; specifically this copies all buffer fields with the exception ofdata
,length
andoffset
.
-
release
()[source]¶ Release a reference to the buffer. This is the opposing call to
acquire()
. Once all references have been released, the buffer will be recycled.
-
replicate
(source)[source]¶ Replicates the source
MMALBuffer
. This copies all fields from the source buffer, including the internaldata
pointer. In other words, after replication this buffer and the source buffer will share the same block of memory for data.The source buffer will also be referenced internally by this buffer and will only be recycled once this buffer is released.
Note
This is fundamentally different to the operation of the
copy_from()
method. It is much faster, but imposes the burden that two buffers now share data (the source cannot be released until the replicant has been released).
-
command
¶ The command set in the buffer’s meta-data. This is usually 0 for buffers returned by an encoder; typically this is only used by buffers sent to the callback of a control port.
-
data
¶ The data held in the buffer as a
bytes
string. You can set this attribute to modify the data in the buffer. Acceptable values are anything that supports the buffer protocol, and which containssize
bytes or less. Setting this attribute implicitly modifies thelength
attribute to the length of the specified value and setsoffset
to zero.Note
Accessing a buffer’s data via this attribute is relatively slow (as it copies the buffer’s data to/from Python objects). See the
MMALBuffer
documentation for details of a faster (but more complex) method.
-
dts
¶ The decoding timestamp (DTS) of the buffer, as an integer number of microseconds or
MMAL_TIME_UNKNOWN
.
-
flags
¶ The flags set in the buffer’s meta-data, returned as a bitmapped integer. Typical flags include:
MMAL_BUFFER_HEADER_FLAG_EOS
– end of streamMMAL_BUFFER_HEADER_FLAG_FRAME_START
– start of frame dataMMAL_BUFFER_HEADER_FLAG_FRAME_END
– end of frame dataMMAL_BUFFER_HEADER_FLAG_KEYFRAME
– frame is a key-frameMMAL_BUFFER_HEADER_FLAG_FRAME
– frame dataMMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO
– motion estimatation data
-
length
¶ The length of data held in the buffer. Must be less than or equal to the allocated size of data held in
size
minus the dataoffset
. This attribute can be used to effectively blank the buffer by setting it to zero.
-
offset
¶ The offset from the start of the buffer at which the data actually begins. Defaults to 0. If this is set to a value which would force the current
length
off the end of the buffer’ssize
, thenlength
will be decreased automatically.
-
pts
¶ The presentation timestamp (PTS) of the buffer, as an integer number of microseconds or
MMAL_TIME_UNKNOWN
.
-
-
class
picamera.mmalobj.
MMALQueue
(queue)[source]¶ Represents an MMAL buffer queue. Buffers can be added to the queue with the
put()
method, and retrieved from the queue (with optional wait timeout) with theget()
method.-
get
(block=True, timeout=None)[source]¶ Get the next buffer from the queue. If block is
True
(the default) and timeout isNone
(the default) then the method will block until a buffer is available. Otherwise timeout is the maximum time to wait (in seconds) for a buffer to become available. If a buffer is not available before the timeout expires, the method returnsNone
.Likewise, if block is
False
and no buffer is immediately available thenNone
is returned.
-
put
(buf)[source]¶ Place
MMALBuffer
buf at the back of the queue.
-
put_back
(buf)[source]¶ Place
MMALBuffer
buf at the front of the queue. This is used when a buffer was removed from the queue but needs to be put back at the front where it was originally taken from.
-
-
class
picamera.mmalobj.
MMALPool
(pool)[source]¶ Represents an MMAL pool containing
MMALBuffer
objects. All active ports are associated with a pool of buffers, and a queue. Instances can be treated as a sequence ofMMALBuffer
objects but this is only recommended for debugging purposes; otherwise, use theget_buffer()
,send_buffer()
, andsend_all_buffers()
methods which work with the encapsulatedMMALQueue
.-
get_buffer
(block=True, timeout=None)[source]¶ Get the next buffer from the pool’s queue. See
MMALQueue.get()
for the meaning of the parameters.
-
resize
(new_count, new_size)[source]¶ Resizes the pool to contain new_count buffers with new_size bytes allocated to each buffer.
new_count must be 1 or more (you cannot resize a pool to contain no headers). However, new_size can be 0 which causes all payload buffers to be released.
Warning
If the pool is associated with a port, the port must be disabled when resizing the pool.
-
send_all_buffers
(port, block=True, timeout=None)[source]¶ Send all buffers from the queue to port. block and timeout act as they do in
get_buffer()
. If no buffer is available (for the values of block and timeout,PiCameraMMALError
is raised).
-
send_buffer
(port, block=True, timeout=None)[source]¶ Get a buffer from the pool’s queue and send it to port. block and timeout act as they do in
get_buffer()
. If no buffer is available (for the values of block and timeout,PiCameraMMALError
is raised).
-
-
class
picamera.mmalobj.
MMALPortPool
(port)[source]¶ Bases:
picamera.mmalobj.MMALPool
Construct an MMAL pool for the number and size of buffers required by the
MMALPort
port.-
send_all_buffers
(port=None, block=True, timeout=None)[source]¶ Send all buffers from the pool to port (or the port the pool is associated with by default). block and timeout act as they do in
MMALPool.get_buffer()
.
-
send_buffer
(port=None, block=True, timeout=None)[source]¶ Get a buffer from the pool and send it to port (or the port the pool is associated with by default). block and timeout act as they do in
MMALPool.get_buffer()
.
-
16.6. Python Extensions¶
-
class
picamera.mmalobj.
MMALPythonPort
(owner, port_type, index)[source]¶ Implements ports for Python-based MMAL components.
-
commit
()[source]¶ Commits the port’s configuration and automatically updates the number and size of associated buffers. This is typically called after adjusting the port’s format and/or associated settings (like width and height for video ports).
-
connect
(other, **options)[source]¶ Connect this port to the other
MMALPort
(orMMALPythonPort
). The type and configuration of the connection will be automatically selected.Various connection options can be specified as keyword arguments. These will be passed onto the
MMALConnection
orMMALPythonConnection
constructor that is called (see those classes for an explanation of the available options).
-
copy_from
(source)[source]¶ Copies the port’s
format
from the sourceMMALControlPort
.
-
enable
(callback=None)[source]¶ Enable the port with the specified callback function (this must be
None
for connected ports, and a callable for disconnected ports).The callback function must accept two parameters which will be this
MMALControlPort
(or descendent) and anMMALBuffer
instance. Any return value will be ignored.
-
get_buffer
(block=True, timeout=None)[source]¶ Returns a
MMALBuffer
from the associatedpool
. block and timeout act as they do in the correspondingMMALPool.get_buffer()
.
-
send_buffer
(buf)[source]¶ Send
MMALBuffer
buf to the port.
-
bitrate
¶ Retrieves or sets the bitrate limit for the port’s format.
-
buffer_count
¶ The number of buffers allocated (or to be allocated) to the port. The default is 2 but more may be required in the case of long pipelines with replicated buffers.
-
buffer_size
¶ The size of buffers allocated (or to be allocated) to the port. The size of buffers defaults to a value dictated by the port’s format.
-
capabilities
¶ The capabilities of the port. A bitfield of the following:
- MMAL_PORT_CAPABILITY_PASSTHROUGH
- MMAL_PORT_CAPABILITY_ALLOCATION
- MMAL_PORT_CAPABILITY_SUPPORTS_EVENT_FORMAT_CHANGE
-
connection
¶ If this port is connected to another, this property holds the
MMALConnection
orMMALPythonConnection
object which represents that connection. If this port is not connected, this property isNone
.
-
enabled
¶ Returns a
bool
indicating whether the port is currently enabled. Unlike other classes, this is a read-only property. Useenable()
anddisable()
to modify the value.
-
format
¶ Retrieves or sets the encoding format of the port. Setting this attribute implicitly sets the encoding variant to a sensible value (I420 in the case of OPAQUE).
-
framerate
¶ Retrieves or sets the framerate of the port’s video frames in fps.
-
framesize
¶ Retrieves or sets the size of the source’s video frames as a (width, height) tuple. This attribute implicitly handles scaling the given size up to the block size of the camera (32x16).
-
index
¶ Returns an integer indicating the port’s position within its owning list (inputs, outputs, etc.)
-
supported_formats
¶ Retrieves or sets the set of valid formats for this port. The set must always contain at least one valid format. A single format can be specified; it will be converted implicitly to a singleton set.
If the current port
format
is not a member of the new set, no error is raised. An error will be raised whencommit()
is next called ifformat
is still not a member of the set.
-
type
¶ The type of the port. One of:
- MMAL_PORT_TYPE_OUTPUT
- MMAL_PORT_TYPE_INPUT
- MMAL_PORT_TYPE_CONTROL
- MMAL_PORT_TYPE_CLOCK
-
-
class
picamera.mmalobj.
MMALPythonBaseComponent
[source]¶ Base class for Python-implemented MMAL components. This class provides the
_commit_port()
method used by descendents to control their ports’ behaviour, and theenabled
property. However, it is unlikely that users will want to sub-class this directly. SeeMMALPythonComponent
for a more useful starting point.-
_commit_port
(port)[source]¶ Called by ports when their format is committed. Descendents may override this to reconfigure output ports when input ports are committed, or to raise errors if the new port configuration is unacceptable.
Warning
This method must not reconfigure input ports when called; however it can reconfigure output ports when input ports are committed.
-
close
()[source]¶ Close the component and release all its resources. After this is called, most methods will raise exceptions if called.
-
enable
()[source]¶ Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.
-
control
¶ The
MMALControlPort
control port of the component which can be used to configure most aspects of the component’s behaviour.
-
-
class
picamera.mmalobj.
MMALPythonComponent
(name='py.component', outputs=1)[source]¶ Bases:
picamera.mmalobj.MMALPythonBaseComponent
Provides a Python-based MMAL component with a name, a single input and the specified number of outputs (default 1). The
connect()
anddisconnect()
methods can be used to establish or break a connection from the input port to an upstream component.Typically descendents will override the
_handle_frame()
method to respond to buffers sent to the input port, and will setMMALPythonPort.supported_formats
in the constructor to define the formats that the component will work with.-
_commit_port
(port)[source]¶ Overridden to to copy the input port’s configuration to the output port(s), and to ensure that the output port(s)’ format(s) match the input port’s format.
-
_handle_end_of_stream
(port, buf)[source]¶ Handles end-of-stream notifications passed to the component (where
MMALBuffer.command
is set to MMAL_EVENT_EOS).The default implementation does nothing but return
True
(indicating that processing should halt). Override this in descendents to respond to the end of stream.The port parameter is the port into which the event arrived.
-
_handle_error
(port, buf)[source]¶ Handles error notifications passed to the component (where
MMALBuffer.command
is set to MMAL_EVENT_ERROR).The default implementation does nothing but return
True
(indicating that processing should halt). Override this in descendents to respond to error events.The port parameter is the port into which the event arrived.
-
_handle_format_changed
(port, buf)[source]¶ Handles format change events passed to the component (where
MMALBuffer.command
is set to MMAL_EVENT_FORMAT_CHANGED).The default implementation re-configures the input port of the component and emits the event on all output ports for downstream processing. Override this method if you wish to do something else in response to format change events.
The port parameter is the port into which the event arrived, and buf contains the event itself (a MMAL_EVENT_FORMAT_CHANGED_T structure). Use
mmal_event_format_changed_get
on the buffer’s data to extract the event.
-
_handle_frame
(port, buf)[source]¶ Handles frame data buffers (where
MMALBuffer.command
is set to 0).Typically, if the component has output ports, the method is expected to fetch a buffer from the output port(s), write data into them, and send them back to their respective ports.
Return values are as for normal event handlers (
True
when no more buffers are expected,False
otherwise).
-
_handle_parameter_changed
(port, buf)[source]¶ Handles parameter change events passed to the component (where
MMALBuffer.command
is set to MMAL_EVENT_PARAMETER_CHANGED).The default implementation does nothing but return
False
(indicating that processing should continue). Override this in descendents to respond to parameter changes.The port parameter is the port into which the event arrived, and buf contains the event itself (a MMAL_EVENT_PARAMETER_CHANGED_T structure).
-
close
()[source]¶ Close the component and release all its resources. After this is called, most methods will raise exceptions if called.
-
connect
(source, **options)[source]¶ Connects the input port of this component to the specified source
MMALPort
orMMALPythonPort
. Alternatively, as a convenience (primarily intended for command line experimentation; don’t use this in scripts), source can be another component in which case the first unconnected output port will be selected as source.Keyword arguments will be passed along to the connection constructor. See
MMALConnection
andMMALPythonConnection
for further information.
-
disconnect
()[source]¶ Destroy the connection between this component’s input port and the upstream component.
-
enable
()[source]¶ Enable the component. When a component is enabled it will process data sent to its input port(s), sending the results to buffers on its output port(s). Components may be implicitly enabled by connections.
-
connection
¶ The
MMALConnection
orMMALPythonConnection
object linking this component to the upstream component.
-
-
class
picamera.mmalobj.
MMALPythonConnection
(source, target, formats=default_formats, callback=None)[source]¶ Bases:
picamera.mmalobj.MMALBaseConnection
Represents a connection between an
MMALPythonBaseComponent
and aMMALBaseComponent
or anotherMMALPythonBaseComponent
. The constructor accepts arguments providing the sourceMMALPort
(orMMALPythonPort
) and targetMMALPort
(orMMALPythonPort
).The formats parameter specifies an iterable of formats (in preference order) that the connection may attempt when negotiating formats between the two ports. If this is
None
, or an empty iterable, no negotiation will take place and the source port’s format will simply be copied to the target port. Otherwise, the iterable will be worked through in order until a format acceptable to both ports is discovered.The callback parameter can optionally specify a callable which will be executed for each buffer that traverses the connection (providing an opportunity to manipulate or drop that buffer). If specified, it must be a callable which accepts two parameters: the
MMALPythonConnection
object sending the data, and theMMALBuffer
object containing data. The callable may optionally manipulate theMMALBuffer
and return it to permit it to continue traversing the connection, or returnNone
in which case the buffer will be released.-
default_formats
= (MMAL_ENCODING_I420, MMAL_ENCODING_RGB24, MMAL_ENCODING_BGR24, MMAL_ENCODING_RGBA, MMAL_ENCODING_BGRA)¶ Class attribute defining the default formats used to negotiate connections between Python and and MMAL components, in preference order. Note that OPAQUE is not present in contrast with the default formats in
MMALConnection
.
-
-
class
picamera.mmalobj.
MMALPythonSource
(input)[source]¶ Bases:
picamera.mmalobj.MMALPythonBaseComponent
Provides a source for other
MMALComponent
instances. The specified input is read in chunks the size of the configured output buffer(s) until the input is exhausted. Thewait()
method can be used to block until this occurs. If the output buffer is configured to use a full-frame unencoded format (like I420 or RGB), frame-end flags will be automatically generated by the source. When the input is exhausted an empty buffer with the End Of Stream (EOS) flag will be sent.The component provides all picamera’s usual IO-handling characteristics; if input is a string, a file with that name will be opened as the input and closed implicitly when the component is closed. Otherwise, the input will not be closed implicitly (the component did not open it, so the assumption is that closing input is the caller’s responsibility). If input is an object with a
read
method it is assumed to be a file-like object and is used as is. Otherwise, input is assumed to be a readable object supporting the buffer protocol (which is wrapped in aBufferIO
stream).-
close
()[source]¶ Close the component and release all its resources. After this is called, most methods will raise exceptions if called.
-
-
class
picamera.mmalobj.
MMALPythonTarget
(output, done=1)[source]¶ Bases:
picamera.mmalobj.MMALPythonComponent
Provides a simple component that writes all received buffers to the specified output until a frame with the done flag is seen (defaults to MMAL_BUFFER_HEADER_FLAG_EOS indicating End Of Stream).
The component provides all picamera’s usual IO-handling characteristics; if output is a string, a file with that name will be opened as the output and closed implicitly when the component is closed. Otherwise, the output will not be closed implicitly (the component did not open it, so the assumption is that closing output is the caller’s responsibility). If output is an object with a
write
method it is assumed to be a file-like object and is used as is. Otherwise, output is assumed to be a writeable object supporting the buffer protocol (which is wrapped in aBufferIO
stream).-
close
()[source]¶ Close the component and release all its resources. After this is called, most methods will raise exceptions if called.
-
16.7. Debugging¶
The following functions are useful for quickly dumping the state of a given MMAL pipeline:
-
picamera.mmalobj.
debug_pipeline
(port)[source]¶ Given an
MMALVideoPort
port, this traces all objects in the pipeline feeding it (including components and connections) and yields each object in turn. Hence the generator typically yields something like:MMALVideoPort
(the specified output port)MMALEncoder
(the encoder which owns the output port)MMALVideoPort
(the encoder’s input port)MMALConnection
(the connection between the splitter and encoder)MMALVideoPort
(the splitter’s output port)MMALSplitter
(the splitter on the camera’s video port)MMALVideoPort
(the splitter’s input port)MMALConnection
(the connection between the splitter and camera)MMALVideoPort
(the camera’s video port)MMALCamera
(the camera component)
-
picamera.mmalobj.
print_pipeline
(port)[source]¶ Prints a human readable representation of the pipeline feeding the specified
MMALVideoPort
port.
Note
It is also worth noting that most classes, in particular
MMALVideoPort
and MMALBuffer
have useful repr()
outputs which can be extremely useful with simple print()
calls for
debugging.
16.8. Utility Functions¶
The following functions are provided to ease certain common operations in the
picamera library. Users of mmalobj
may find them handy in various
situations:
-
picamera.mmalobj.
open_stream
(stream, output=True, buffering=65536)[source]¶ This is the core of picamera’s IO-semantics. It returns a tuple of a file-like object and a bool indicating whether the stream requires closing once the caller is finished with it.
- If stream is a string, it is opened as a file object (with mode ‘wb’ if
output is
True
, and the specified amount of bufffering). In this case the function returns(stream, True)
. - If stream is a stream with a
write
method, it is returned as(stream, False)
. - Otherwise stream is assumed to be a writeable buffer and is wrapped
with
BufferIO
. The function returns(stream, True)
.
- If stream is a string, it is opened as a file object (with mode ‘wb’ if
output is
-
picamera.mmalobj.
close_stream
(stream, opened)[source]¶ If opened is
True
, then theclose
method of stream will be called. Otherwise, the function will attempt to call theflush
method on stream (if one exists). This function essentially takes the output ofopen_stream()
and finalizes the result.
-
picamera.mmalobj.
to_resolution
(value)[source]¶ Converts value which may be a (width, height) tuple or a string containing a representation of a resolution (e.g. “1024x768” or “1080p”) to a (width, height) tuple.
-
picamera.mmalobj.
to_rational
(value)[source]¶ Converts value (which can be anything accepted by
to_fraction()
) to an MMAL_RATIONAL_T structure.
-
picamera.mmalobj.
buffer_bytes
(buf)[source]¶ Given an object which implements the buffer protocol, this function returns the size of the object in bytes. The object can be multi-dimensional or include items larger than byte-size.