By Alicia Boya García. validateflow is an upstream plugin for gst-validate that records buffers and events from arbitrary pipelines and validates them against expectation files. It provides a new effective way to test non-regular- playback use cases without extensive coding. Topics covered include: * Overview of existing validating techniques for GStreamer * How validateflow fits and compares to other techniques * Basics of gst-validate * Testing procedure with validateflow * How to fit all together: pipelines.json * Examples of what can be done (c) GStreamer Conference 2019 31 October - 1 November Lyon, France https://gstreamer.freedesktop.org/conference/2019/

1. Introduction to gst-validateflow
Alicia Boya García
Igalia
2019
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 1 / 16

2. About me
I work at Igalia, an open source consultancy.
Most of my work revolves around improving and maintaining
MediaSource Extensions for the GStreamer-based ports of
WebKit (WPE and WebKitGTK).
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 2 / 16

3. Verification tooling GStreamer
Unit tests (meson test):
They are written in C. Lots of control, relatively
straightforward to understand.
Depending on what is being tested they can require high
effort and there is limited framework (GstHarness).
gst-validate:
A C binary for running tests on pipelines and a complex
Python launcher that uses it.
At best, adding a test can feel more like editing config
files.
Complex or non supported use cases can still require C or
Python coding (e.g. to enable new actions in tests).
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 3 / 16

4. Components of a validate test
For every test found, an invocation to gst-validate-1.0 is
made with this information:
A pipeline
Validate config file(s)
Directories providing the available scenarios
Name of the scenario to use (if any)
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 4 / 16

5. Invoking gst-validate
$ gst-validate-launcher ’validate.launch_pipeline.*’
validate is the name of the testsuite.
validate.launch_pipeline.* is a regexp that selects
which tests to run from the set provided by the
testsuite.
Pitfall: It’s a regexp, not a glob!
. means any character.
* means any number of repetitions of the preceding item.
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 5 / 16

6. Invoking gst-validate, under the hood
The testsuites are defined in
gst-integration-testsuites/testsuites/*.py.
If no filter has been specified, the ‘validate‘ testsuite is
run by default.
The testsuite code adds a series of generators, which in
turn emitting a number of tests.
GstValidatePlaybinTestsGenerator
GstValidatePipelineTestsGenerator
...
The testsuite always adds all the tests, but only those
matching the command line arguments are run.
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 6 / 16

7. Playbin tests (file playback tests)
Many validate tests are generated automatically by
combining many of the parameters of the previous slide.
Namely, every file inside medias/defaults with a
matching .media_info gets played in playbin (2 or 3)
with a number of scenarios testing seeks, fast forward
and other use cases.
Monitors watch for error and warning conditions, like
no playback, missing or unordered events.
This allows for lots of useful smoke tests: if someone
breaks playback of a certain kind of file, there is a good
chance gst-validate will flag it!
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 7 / 16

8. Scenarios
Scenarios enumerate actions that will occur during a
validate test.
There are a number of predefined scenarios in
gst-devtools/validate/data/scenarios/.
Most actions are defined in gst-validate-scenario.c.
Validate plugins can define additional actions.
description, seek=true, duration=20, need-clock-sync=true, ignore-eos=true
include,location=includes/default-seek-flags.scenario
seek, name=First-forward-seek, playback-time="5.0",
start="10.0", flags="$(default_flags)"
seek, name=Second-backward-seek, playback-time="15.0",
start="10.0", flags="$(default_flags)"
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 8 / 16

9. Pipeline tests
They can be found in testsuites/pipelines.json:
"flvdemux.audio_only":
{
"pipeline": "uridecodebin
uri=’file://%(config_path)s/../medias/defaults/flv/819290236.flv’
caps=audio/x-raw expose-all-streams=FALSE ! queue !
audioconvert ! audioresample ! %(audiosink)s",
"scenarios": ["play_15s"]
}
These tests are more flexible in that we can run arbitrary
pipelines with arbitrary scenarios.
But validation is still a smoke test in the example above.
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 9 / 16

10. Finally, validateflow
validateflow is a gst-validate plugin that validates the
flow in a certain pad matches a previous execution
known to be good.
All buffers and events flowing are recorded to a text file.
On the first test run it generates an -expected file inside
the Git repo.
Further executions create an -actual temporary file
(inside the build directory) that is compared to the
former.
Any mismatch is marked as a test failure.
When authoring tests it’s important to check that the log
is correct and reproducible.
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 10 / 16

11. A simple validateflow test
"flow_simple_test_custom":
{
"pipeline": "videotestsrc num-buffers=3 ! fakesink",
"config": [
"%(validateflow)s, pad=fakesink0:sink, record-buffers=true,
caps-properties={ width, height };"
]
},
$ gst-validate-launcher
validate.launch_pipeline.pipelines.flow_simple_test_custom
event stream-start: GstEventStreamStart, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1;
event caps: video/x-raw, width=(int)320, height=(int)240;
event segment: format=TIME, start=0:00:00.000000000, offset=0:00:00.000000000, stop=none, time=0:00:00.0
buffer: pts=0:00:00.000000000, dur=0:00:00.033333333, flags=discont
buffer: pts=0:00:00.033333333, dur=0:00:00.033333333
buffer: pts=0:00:00.066666666, dur=0:00:00.033333334
event eos: (no structure)
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 11 / 16

12. Useful scenario actions for validateflow
appsrc-push: Allows to use appsrc elements inside the
pipeline. Instructs it to push a chunk of data from a file
with a given offset and size and waits for the push to
succeed.
appsrc-eos: Marks the appsrc as EOS.
checkpoint: (validateflow-specific) Writes some text in
the actual/expected log. This allows you to know what
buffers and events are recorded before/after what
actions.
stop: Ends the test (otherwise the test ends when there
are no remaining actions and the pipeline is EOS).
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 12 / 16

13. Embedded scenarios
"matroskademux_flush_within_cluster":
{
"pipeline": "appsrc ! matroskademux ! fakesink async=false",
"config": [
"%(validateflow)s, pad=fakesink0:sink, record-buffers=true"
],
"scenarios": [{
"name": "default",
"actions": [
"description, handles-states=false",
"appsrc-push, target-element-name=appsrc0,
file-name="%(medias)s/fragments/feelings_vp9-20130806-242.webm/init.webm"",
"appsrc-push, target-element-name=appsrc0,
file-name="%(medias)s/fragments/feelings_vp9-20130806-242.webm/media1.webm", size=5000",
"flush, target-element-name=appsrc0",
"checkpoint, text="A different cluster is pushed"",
"appsrc-push, target-element-name=appsrc0,
file-name="%(medias)s/fragments/feelings_vp9-20130806-242.webm/media2.webm", size=10000",
"stop"
]
}]
},
By default no actions are run until the pipeline has prerolled
For this reason in pipelines using appsrc you need to either ensure the sink does not wait for pre-roll with
async=false or allow actions to run without waiting for it with handles-state=false.
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 13 / 16

14. Running tests with scenarios
A test can define multiple scenarios, each with a different
name. Or it may use no scenario at all.
The full name of the test is:
validate.launch_pipeline.<test name>.<scenario>
for tests with scenarios.
validate.launch_pipeline.pipelines.<test name>
for tests without scenarios.
Future work
The current naming scheme for pipeline tests is quite
unintuitive. We may consider to change it in the future to
something less surprising.
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 14 / 16

15. Documentation
The GStreamer documentation contains more in-detail
information about validateflow.
https://gstreamer.freedesktop.org/documentation/
gst-devtools/plugins/validateflow.html
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 15 / 16

16. Future work
Turn pipelines.json into pipelines.py, because
authoring JSON is uncomfortable for humans.
Adding support for recording messages with certain filters
(e.g. STREAM_COLLECTION messages from playbin3).
Alicia Boya García (Igalia) Introduction to gst-validateflow 2019 16 / 16