Using GStreamer with Python
You have probably noticed by now that working with GStreamer and Python is quite difficult, mainly because of the lack of real documentation and examples. In this guide I will be showing you how to create a Python program which will allow you to Stream video from the internet, split the audio into different elements which will Play the audio, show a waveform of that audio, and save the audio as an MP3 file. I will also show you tools and resources I found that have been helpful in allowing me to understand how GStreamer works. Please note that in this guide I will be working with GStreamer 0.10.
You may want to have a look through these pages which I have found to be quite useful in learning GStreamer:
- An old but good explination of the workings of GStreamer: http://www.jonobacon.org/2006/08/28/getting-started-with-gstreamer-with-python/
- GStreamer documentation: http://www.freedesktop.org/software/gstreamer-sdk/data/docs/latest/
- Rubenrua’s Python examples. These are by far the most helpful things I’ve found: https://github.com/rubenrua/GstreamerCodeSnippets/tree/master/Python/pygst-sdk-tutorials
- GStreamer Tutorials (Writen in C): http://docs.gstreamer.com/display/GstSDK/Tutorials
As you may already know, GStreamer is a framework for creating, manipulating and playing multimedia. With Gstreamer you will be creating and assembling several objects called Elements which handle the data and pass it off to the next element in line. Sounds pretty simple, right? Well it actually is when you understand how to attach these elements. First, lets use GStreamer in the terminal to play some music. Install GStreamer-0.10-tools:
sudo apt-get install gstreamer0.10-tool
Now in the terminal navigate to an mp3 file on your computer and type in:
gst-launch-0.10 filesrc location="myfile.mp3" ! decodebin ! autoaudiosink
Your music will now be playing. What this is doing is creating a filesrc element which gets the mp3 file. It then passes it to a new decodebin element which converts it from an mp3 file into audio data. It then passes it to a new autoaudiosink file which is intended to play it through your speakers. So as you can see, the first element takes the media, does somthing to it and passes it to the next element. That process repeats. Lets try something else
gst-launch-0.10 audiotestsrc freq=220 ! wavescope shader=0 style=1 ! ffmpegcolorspace ! autovideosink
When you run this, no audio actually plays which is fine. We didn’t tell it to play any audio. What this does is create an audiotestsrc Element which merely emits a tone. We have told it that its frequency will be 220Hz. It passes that audio to a wavescope element which will create a visual representation of the audio. The waveform element passes that media to the ffmpegcolorspace element which does something to it (I just know that without this, it will fail). It then sends that data to autovideosink which outputs the waveform to the monitor.
So, as you can see, this is a pretty neat framework for working with multimedia, and if you encountered the same difficulty I have learning GStreamer, this will be the first proff that GStreamer actually works. Using gstreamer0.10-tool is a great way to learn about GStreamers functionality and what you can do. We can also see what elements we have available, their properties, and what Pads they have available. We will learn more about Pads later. In terminal write:
That will show you all the possible elements you can make with GStreamer. Lets take a closer look at what the wavescope element has to offer. This time, we’ll write to a text file so we can view it easier:
gst-inspect-0.10 wavescope > wavescope.txt
Here we see a very short description of what it does (Sometimes you will notice that an element is obsolete). We also see its Pads and its Element Properties. You can see that shader and style properties we altered.
Other GStreamer concepts
So by now, you should have a general understanding of an element and how they are used. There are a few other major concepts you should know of. First, there are Bins. A Bin is a container for a group of elements. Bins themselves are subclasses of elements and by altering a state of then bin, you can change the state of all elements within the bin. We wont been doing anything intensive with a Bin in this guide.
A Pipeline is a top-level bin which synchronizes the elements contained with it. Unfortunately, at this time, I have little knowledge of what advantages Bins and Pipelines have over each other. We will, however be using both of them within this guide. In the end, they are used to contain multiple elements.
Pads are the piece that interconnects elements. Pads have two properties, their direction and their availability. There are two pad directions, source and sink. Elements receive data with their sink pads and push data to their source pads. In our first terminal example above, we had three elements: filesrc, decodebin, and autoaudiosink. Filesrc receives a file and pushes it to its source pad. This makes it available for decodebin to accept it on its sink pad. When it receives that data it manipulates it and pushes it to its source pad. Now autoaudiosink can accept that data on its sink pad and within that element, it plays it on your computers speakers.
The availability property is far more confusing. There are three types of availability. Always, Sometimes (AKA Dynamic), and Request. Always pads, always exist, Sometimes pads exist only in certain cases, and Request pads exist when you tell them to otherwise they are not present and will not be able to connect to anything. Now, take a look at some of the elements using our gst-inspect-0.10 tool. If you look at audioconvert you will see that it has both source and sink pads and bother have Always as their availability. Decodebin has an Always Sink pad and a Sometimes Source pad. If you check the element, tee will have an On-Request source pad.
Now, we start with a simple project to show you how to use Python and GStreamer together. Finally. We are going to make a simple project where we take an mp3 file move to through an equalizer so we can play with the sound a little bit, and play it through our speakers. In order to do this, we will be using 5 elements.
- filesrc which will get our mp3 file
- mad which decodes the mp3 file into usable audio data
- audioconvert which we will use to connect our decoded audio into the equalizer
- equalizer-3bands which we use to change the audio bands
- autoaudiosink which plays the audio on our speakers
Now, please keep in mind, that I am still new to how GStreamer works. This is the easiest way I could configure the elements while still giving you something to play with (the equalizer). Now, lets take a moment to look at our elements using the gst-inspect-0.10 tool. We are specifically looking at the Pad Templates to see how these elements link together.
Filesrc has a source pad that is Always and has ANY capabilities. It will link to Mad which is an mp3 decoder. Mad has an Always Sink pad and an Always Source pad but its Capabilities are more limited. Here we have audio/x-raw-int with some other parameters. As far as I can tell the rate and width appear to most important in connecting elements. Its width is 32. Mad will link with audioconvert. This element converts audio to different formats but I am using it for its pads. It has an Always Sink pad with many capabilities. Notice that it has x-raw-int with a width of 32. I _believe_ this allows Mad to link to this element. audioconvert also has an Always Source pad with several capabilities. Audioconvert links with equalizer-3bands. It has an Always Sink pad with a x-war-int and width 16 which our audioconvert element can work with. Our equalizer also has an Always Source pad with similar limited capabilities. It links with autoaudiosink which has an Always Sink pad with ANY capabilities.
Now we will begin putting this project together. First we need to import gst and add the Pipeline and each of our five Elements. We will also verify that all elements were created successfully. To create an Element we use the gst.element_factory_make method with takes 1 argument and an optional argument. Adding the options argument is helpful for debugging which we’ll see at the end of this project.
Now we can configure our elements. We will tell audio_source where to find our mp3 file and set some of the bands on our equalizer. If you inspect the equalizer, you will see that in its Element Properties you can change band0, band1, and band2. Their range is from -24 to 12. To edit an elements properties you call its set_property method. Next we will add all our elements to the pipeline. Then we will link the elements together using the gst.element_link_many method. The first argument gets linked to the second, the second to the third, and so on. Order matters here. We encapsulate this in an if statement which will help us catch any errors.
And finally we send a signal to our pipeline to change its state to gst.STATE_PLAYING. Next we capture the pipelines bus and with bus.timed_pop_filtered() we wait until there is an error or EOS signal. We can also check the return value to see any messages. This is a helpful feature because if you have an error and you read the string, it can give you a small insight as to what it wrong. Unfortunately it is cryptic and I haven’t found much documentation as to what the messages mean. I have, however, noticed that it will give you the elements name (The optional argument) where the issue may lie. Lastly, we free the resources.
When you run your project, you should be hearing the audio play. One error I was getting was
<gst.Message GstMessageError, gerror=(GError)NULL, debug=(string)"gstbasetransform.c\(2541\):\ gst_base_transform_handle_buffer\ \(\):\ /GstPipeline:pipeline/GstIirEqualizer3Bands:equalizer:12not\ negotiated"; from equalizer at 0x287da30>
when I had the equalizer and converter switched. I looked into the pad templates for the equalizer and mad and found that they don’t appear to completely match.
If you wish to see the full project go to https://gist.github.com/markwingerd/403306854d457c284d02
Next we will be looking into a few more projects. We will learn how to use the playbin2 element to stream multimedia and after that splitting data so we can do multiple things with it.