Conversation - Background substraction

Here is a description of the different steps to add background substraction on conference : blur, transparent, or image.

In the background subtraction process, a mask (binary image) is computed asynchronously to separate the foreground from the background. Meaning that the foreground stream keeps its original frame rate but the separation has a slower frame rate which depends on the customer’s terminal performances. To ensure that the process can be handled, the computation time of the mask is monitored. If this duration exceeds one second for ten computations then the background subtraction process will be stopped and an error will be raised through apiRTC events. It means that the process stops if the separation’s frame rate falls under one frame per second a total of ten times.

Connect to the conference and select media devices

Check our tutorial on conversation and devices selection to have details about the conversation bases and media devices selection.

Add needed filter parameters on createStream

When you use createStream() function, you can add different options thanks to createStreamOptions parameter.

        var createStreamOptions = {};
        createStreamOptions.audioInputId = selectMic.value;
        createStreamOptions.videoInputId = selectCamera.value;
    

For background filtering, we can choose the kind of filtering between : blur, transparent or image. Here is the code to do it :

        //For Blur
        createStreamOptions.filters = [
            { type: 'backgroundSubtraction', options: { backgroundMode: 'blur' } },
        ];

        //For Transparent
        createStreamOptions.filters = [
            { type: 'backgroundSubtraction', options: { backgroundMode: 'transparent' } },
        ];

        //For Image
        createStreamOptions.filters = [
            { type: 'backgroundSubtraction', options: { backgroundMode: 'image', image: imageData } },
        ];
    

Then simply use createStreamOptions on createStream()

        ua.createStream(createStreamOptions)
        .then(function (stream) {
            [...]
        })
        .catch(function (err) {
            [...]
        });
    

Switch Stream

Use call.replacePublishedStream() function update the published stream after a background modification. Check the code of our sample Github repository for a complete description

        call.replacePublishedStream(null, callbacks);
    

How to get imageData ?

Here is a sample code to get imageData using a canvas :

        var imageData = null;
        const image = new Image();
        image.src = window.location.href + 'img/background_apiRTC.jpg';

        image.onload = () => {
            var canvas = document.createElement('canvas');
            var context = canvas.getContext('2d');
            canvas.width = image.width;
            canvas.height = image.height;
            context.drawImage(image, 0, 0 );
            imageData = context.getImageData(0, 0, 640, 480);
            console.debug("image is loaded");
        };
    

How to catch fired events ?

Here is a sample code to catch events fired by the stream. Events are described here Stream API

        ua.createStream(createStreamOptions)
        .then(function (stream) {
            // backgroundSubstractionStopVisibilityChange event
            stream.on('backgroundSubstractionStopComputeTime', function(e){
                console.log(e);
            })

            // backgroundSubstractionStopVisibilityChange event
            stream.on('backgroundSubstractionStopVisibilityChange', function(e){
                console.log(e);
            })
        })
        .catch(function (err) {
            [...]
        });
    

Live Demo