Developers Guide
Let's start building video communication stuff!
The
UserAgent
is an entry point to ApiRTC CPaaS. This is the first object to instantiate when implementing a front-end application. It represents the local user that will participate in the conversation.The UserAgent can be either anonymous or identified.
Identification is done through a JWT retrieved from an authentication service.
UserAgent
's mediaDeviceChanged
event can be listened to in order to get notified of the list of devices available to the browser:Here is what the
mediaDevices
object looks like:userAgent.on("mediaDeviceChanged", () => {
const mediaDevices = this.userAgent.getUserMediaDevices();
// handle new set of mediaDevices
});
This is useful to propose a list of available media devices to the user.
A
Session
instance represents a connection to the ApiRTC CPaaS. A Session
is configured by an API key and a declared UserAgent
.A session handles all the interactions of participants, including video/audio streams and data exchanges for one
Enterprise
identified by its API key.A
Session
object is to get through the UserAgent.register(options)
method. Some options
(of type RegisterInformation
) controls the authentication mechanisms.Contacts are the participants to a
Session
. They can be authenticated or anonymous.session.getContacts().foreach({username, contactObject} => {
console.log(contactObject.getId());
});
contact.sendData({aProperty:'aValue'})
.then(() => {
console.log("message sent")
}).catch((error: any) => {
console.error("sendData", error)
})
session.on('contactData', contactDataEvent => {
console.log('received data from sender', contactDataEvent.sender, contactDataEvent.content)
})
Each Contact getting into a Session can join presence groups and segment all the connected users into subcategories.
For example: an employee can get into a Session, and join the "Operator" and "Available" groups, while a customer will join the "Customer" group.
To make a user connect within some group, set the
RegisterInformation.groups
in the UserAgent.register(options)
options, or user the Session.joinGroup
method.Joining a group as a participant will activate session's
contactListUpdate
event listening on this group. Alternatively, you can subscribe to the group's events without joining it with the Session.subscribeToGroup
method.If the current participant doesn't subscribe to or join a group, they will not receive event regarding group changes.
The data object associated to
Session.contactListUpdate
event has joinedGroup
and leftGroup
properties to carry information on which Contact
joined of left which group:session.on('contactListUpdate', (updatedContacts: any) => {
for (const group of Object.keys(updatedContacts.joinedGroup)) {
for (const contact of updatedContacts.joinedGroup[group]) {
// ...
}
}
for (const group of Object.keys(updatedContacts.leftGroup)) {
for (const contact of updatedContacts.leftGroup[group]) {
// ...
}
}
})
UserData
is a class that holds a data object to store values associated to a UserAgent
. Make sure to call UserData.setProp(key, value)
to set up a property.Once connected to
Session
, call userData.setToSession()
to make other connected peers notified of UserData
properties change through the Session.contactListUpdate
event.For that purpose, the data object associated to
Session.contactListUpdate
event has a userDataChanged
property which is an array of Contact
s for which UserData
has changed.session.on('contactListUpdate', updatedContacts => {
for (const contact of updatedContacts.userDataChanged) {
// ...
}
}
A conversation is the way to gather participants to exchange medias. It can be text message, audio/video streams, files...
Whenever there are 2 participants or more, a conversation takes place.
To get a
Conversation
instance, the Session's method getOrCreateConversation(name, options)
should be used.The
name
is of string type without any constraint.conversation = session.getOrCreateConversation(name, {
meshModeEnabled: false,
meshOnlyEnabled: false,
moderationEnabled: false,
moderator: false
});
key | description |
---|---|
meshModeEnabled | enables the mesh mode (otherwise SFU star topology is used by default). (default: false ) |
meshOnlyEnabled | forces the mesh mode for the whole Conversation. |
moderationEnabled | enables moderation on the Conversation . This option may change the behavior of the joining process depending on the moderator option value. (default: false ) |
moderator | adds the local user to the list of moderators for the conversation. (default: false ) |
Every participant must enable the
moderationEnabled
option to have consistent moderation apply throughout the Conversation.The mesh mode enables a peer-to-peer connection across participants, without going through a stream routing server (called SFU for Selective Forwarding Unit).
Mesh mode multiplies the stream sent by each participant. As upload bandwidth is often lower than download bandwith, network connection can become shaky as the number of participants grows.
If
meshModeEnabled
is true
when setting the conversation mode, the stream exchanges will remain in P2P until:- the number of participant goes over 4,
- or too many packet loss is detected for one participant.
Then the conversation will automatically switch to star topology mode using the ApiRTC SFUs infrastructure.
Setting both
meshModeEnabled
and meshOnlyEnabled
to true
forces the conversation to remain mesh only, whatever the connection's events.Conversation.join()
makes the local user enter the conversation. Note that this method returns a Promise
and one must wait for it to be fulfilled before doing anything else on the conversation.A good practice is to register all required
Conversation
event listeners before calling the join
method:conversation.on('streamListChanged', streamInfo => {
// Handle the 'streamListChanged' event...
});
// and any other relevant events
// ...
conversation.join()
.then(() => {
// local user successfully joined the conversation.
}).catch(error => {
// error
});
Conversation.leave()
makes the local user leave the conversation. All ongoing streams will be automatically closed.A good practice is to destroy the
Conversation
after leaving it. (Except if you want to be able to join it again afterward)conversation.leave()
.then(() => {
conversation.destroy();
});
Moderation allows a group of users (moderators) to control the conversation's access to other participants.
In order to activate the moderation for a conversation, every party (moderator or not) must explicitly set the
moderationEnabled
option to true
when calling getOrCreateConversation.Additionally, the moderator participant set the
moderator
option to true
as well when calling getOrCreateConversation.If the local user is moderator, then the
join()
will resolve immediately.But if the local user is not moderator, then the
join()
will only resolve when a moderator allows it. In the meanwhile, the user will be put in a waiting room.The waiting room is a presence group associated to the conversation. It allows to identify participants who are currently waiting for a moderation answer.
Events
contactJoinedWaitingRoom
and contactLeftWaitingRoom
will be triggered respectively upon the arrival and departure of a user to/from the waiting room:onversation.on('contactJoinedWaitingRoom', contact => {
// A candidate joined the waiting room.
// Store it into a list and display it in the DOM
// ...
});
conversation.on('contactLeftWaitingRoom', contact => {
// A candidate left the waiting room.
// Remove from list
// ...
});
Then the moderator can allow or deny a contact to enter the conversation:
// Grant...
conversation.allowEntry(contact);
// ... or deny access.
conversation.denyEntry(contact);
Moderators have the ability to eject another participant from the conversation.
conversation.eject(contact, { reason: 'a reason' })
.then(() => {
console.log('ejected', contact);
}).catch((error) => {
console.error('eject error', error);
});
To get notified of a participant ejection, listen on the
participantEjected
event. The event data object wears a self
boolean set to true
if the current local user is the ejected participant.conversation.on('participantEjected', data => {
console.log('on:participantEjected', data);
if (data.self) {
// local user was ejected,
// unpublish streams,
// and destroy the conversation
}
});
ApiRTC platform allows to record a composite video of a conversation. The video will be composed of all streams exchanged in the conversation and will be stored in ApiRTC's database.
To start recording a conversation:
conversation.startRecording().then(recordingInfo => {
console.info('startRecording', recordingInfo);
}).catch((error: any) => {
console.error('startRecording', error);
});
{
"roomName": "Test",
"callId": "COMPOSITE",
"recordType": "composite",
"convId": "2b0839f5-aa1e-4cb2-ba9a-46848a6b",
"mediaId": "1261785",
"mediaURL": "https://dashboard.apirtc.com/media/showVideo/<id>/hash/2c625610-4baa-11ec-a192-538513dee1ef",
"recordedFileName": "vfrP9vWu-3467-composite.mp4",
"audioOnly": false,
"videoOnly": false,
"mode": "complete",
"labelEnabled": false
}
To stop recording a conversation:
conversation.stopRecording().then(recordingInfo => {
console.info('stopRecording', recordingInfo);
}).catch((error: any) => {
console.error('stopRecording', error);
});
Once the recording is stopped, the ApiRTC platform will process it and make it available for download. To get notified when a record is available, listen to the
recordAvailable
event of the Converation's instance:conversation.on('recordingAvailable', recordingInfo => {
console.log("on:recordingAvailable", recordingInfo);
...
});
When the video is available, you can use the
RecordingInfo.mediaURL
to download it.To display which participant is currently talking in a
Conversation
, enable the feature:userAgent.enableActiveSpeakerDetecting(true, { threshold: 50 });
Then, listen on the
audioAmplitude
event:conversation.on('audioAmplitude', amplitudeInfo => {
// handle amplitudeInfo
})
the event data object (
amplitudeInfo
) holds the following information:{
"streamId": "6725958108801516",
"amplitude": 102.36,
"isSpeaking": true
}
When the participant speaks and
amplitude
goes over the threshold
configured during feature enabling, event with isSpeaking
set to true
is fired.When the participant does not speak anymore, the event is fired again with initial
amplitude
value that triggered the event, but this time isSpeaking
is false.Conversation
event callStatsUpdate
provides statistics information on media stream exchanges quality of service.conversation.on('callStatsUpdate', callStats => {
// handle callStats.stats data
});
Depending on whether the stream is sent or received, the event data object (
callStats
) holds the following information:For a local stream, qos info on sent media
"streamId": "7167592935479248",
"stats": {
"audioSent": {
"bitsSentPerSecond": 22044,
"bytesSent": 54006,
"delay": 0,
"kind": "audio",
"mediaType": "audio",
"nackCount": 0,
"packetLossRatio": 0,
"packetsSent": 982,
"packetsSentPerSecond": 50,
"remoteId": "9385e3a0",
"samplingInterval": 10,
"timestamp": 1633005140,
"type": "outbound-rtp"
},
"videoSent": {
"bitrateMean": 490785.10526315786,
"bitrateStdDev": 54128.26265341604,
"bitsSentPerSecond": 517595,
"bytesSent": 1245545,
"delay": 0,
"droppedFrames": 2,
"firCount": 0,
"framerateMean": 30.315789473684212,
"framerateStdDev": 0.749268649265355,
"framesEncoded": 562,
"framesEncodedPerSecond": 30,
"height": 480,
"kind": "video",
"mediaType": "video",
"moyDelay": null,
"nackCount": 2,
"packetLossRatio": 0,
"packetsSent": 1232,
"packetsSentPerSecond": 63,
"pliCount": 4,
"qpSum": 20935,
"remoteId": "526431ec",
"samplingInterval": 10,
"timestamp": 1633005140,
"type": "outbound-rtp",
"width": 640
},
"quality": {
"mosS": "NoStream",
"mosSAV": 3.087473118525441,
"mosSS": 4.409150284259602,
"mosSV": 3.4956463628881274,
"mosV": "NoStream"
}
}
}
For a remote stream, qos info on received media
{
"streamId": "362307064506733",
"stats": {
"audioReceived": {
"bitsReceivedPerSecond": 22044,
"bytesReceived": 109505,
"delay": 0,
"jitter": 0.002,
"kind": "audio",
"mediaType": "audio",
"nackCount": 0,
"packetLossRatio": 0,
"packetsLost": 1,
"packetsLostPerSecond": 0,
"packetsReceived": 1991,
"packetsReceivedPerSecond": 50,
"remoteId": "f14eaf8",
"samplingInterval": 20,
"timestamp": 1633005174,
"type": "inbound-rtp"
},
"videoReceived": {
"bitrateMean": 773107.5384615384,
"bitrateStdDev": 176425.8098193486,
"bitsReceivedPerSecond": 910830,
"bytesReceived": 3874287,
"delay": 0,
"discardedPackets": 0,
"firCount": 0,
"framerateMean": 30.076923076923073,
"framerateStdDev": 0.4220635637221745,
"framesDecoded": 1167,
"framesDecodedPerSecond": 30,
"height": 480,
"jitter": 0.009,
"kind": "video",
"mediaType": "video",
"nackCount": 13,
"packetLossRatio": 0.09086778736937756,
"packetsLost": 3,
"packetsLostPerSecond": 0,
"packetsReceived": 3790,
"packetsReceivedPerSecond": 110,
"pliCount": 1,
"remoteId": "8a312f14",
"samplingInterval": 20,
"timestamp": 1633005174,
"type": "inbound-rtp",
"width": 640
},
"quality": {
"mosAV": 3.4344075003680574,
"mosS": 4.409150284259602,
"mosSS": "NoStream",
"mosSV": "NoStream",
"mosV": 3.860635735132783
}
}
}
The
callStats.streamId
is useful to associate data to corresponding streamreamsAcquiring camera local stream is done through the
UserAgent.createStream(options)
method. The browser asks the user to choose among available media devices. The Promise
resolves with a Stream
instance.userAgent.createStream({
constraints: {
audio: true,
video: true
}
}).then(localStream => {
// ...
}).catch(error => {
// error
});
All possible options for the CreateStream method can be found in the
CreateStreamOptions
reference page.constraints
option is of type MediaStreamConstraints. See the Stream Constraint section for more infos.Acquiring screen sharing local stream is done through a
Stream
static method:// Returns a Promise.<Stream> containing the stream
Stream.createScreensharingStream().then(localStream => {
// ...
}).catch(error => {
// error
});
Publishing a local stream makes it available for remote peer participants to subscribe and view it.
The local user (
UserAgent
) must have joined the conversation before publishing a stream.conversation.publish(localStream).then(publishedStream => {
// local stream is published
}).catch(error => {
// error
});
Conversation.publish(localStream, options)
can optionally take PublishOptions
second parameter object to control publication. Please check reference for details on PublishOptions.Unpublishing a local stream makes it unavailable for remote peer participants to subscribe and stops sending media stream to peers.
conversation.unpublish(localStream);
ApiRTC triggers an event when stream availability changes through the
Conversation.streamListChanged
event.This event is triggered:
- once for each existing stream when the participant joins the Conversation
- every time a new stream is published to or unpublished from the Conversation
The data object carried by Conversation.streamListChanged event is StreamInfo: this is not a Stream yet.
conversation.on('streamListChanged', streamInfo => {
const streamId = String(streamInfo.streamId);
const contactId = String(streamInfo.contact.getId());
if (streamInfo.isRemote === true) {
if (streamInfo.listEventType === 'added') {
// a remote stream was published
...
} else if (streamInfo.listEventType === 'removed') {
// a remote stream is not published anymore
...
}
}
});
The
streamInfo.contact.getId()
and streamInfo.streamId
can be useful to identify which remote peer published their stream.A remote stream is subscribed to using the
Conversation.subscribeToStream(streamId)
method. It takes the id of stream provided in the StreamInfo
data object:conversation.subscribeToStream(streamInfo.streamId);
Be mindful that whithout subscribing to stream's event, you will not be notified of streams updates and termination.
Conversation.subscribeToStream(streamId, options)
can take optionally take SubscribeOptions
as a second parameter to control subscription.Unsubscribing to a remote stream is done by the
Conversation.unsubscribeToStream(streamId)
method.conversation.unsubscribeToStream(streamId);
Once a stream has been subscribed, ApiRTC notifies with an actual
Stream
instance through the streamAdded
event.This event is triggered every time the actual media stream is available to be displayed.
conversation.on('streamAdded', remoteStream => {
// display media stream
...
});
Whenever a
media stream
becomes unavailable, ApiRTC notifies the conversation with a streamRemoved
event.conversation.on('streamRemoved', remoteStream => {
// undisplay media stream
...
});
A
media stream
may encounter technical issues, or meet network optimization requiring to change the actual Stream
instance. In this case streamRemoved
event will be also fired, prior to another streamAdded
event with the new instance.In order to display or remove media element in DOM, you can use our helpers:
Stream.addInDiv()
andStream.removeFromDiv()
to add/remove a<video>
element within an existing<div>
Stream.attachToElement(domElement)
to directly attach to a<video>
element.
We manage some devices specificities in our helpers that can help avoid media plays issue. (for instance with Safari iOS).
// display media stream by attaching to a media element (like <video>)
stream.attachToElement(videoDomElement)
// or insert into a container div
stream.addInDiv('container-id', 'media-element-' + stream.streamId, {}, false)
To control local or remote stream audio/video mute, use the following
Stream
methods:Starting from apiRTC 5.0.1
Older versions
Mute state is managed by enabled/disabled attribute at application level.
// toggle audio
if (stream.isAudioEnabled()) {
stream.disableAudio();
} else {
stream.enableAudio();
}
// toggle video
if (stream.isVideoEnabled()) {
stream.disableVideo();
} else {
stream.enableVideo();
}
// toggle audio
if (stream.isAudioMuted()) {
stream.unmuteAudio();
} else {
stream.muteAudio();
}
// toggle video
if (stream.isVideoMuted()) {
stream.unmuteVideo();
} else {
stream.muteVideo();
}
Constraints are camera properties that can be set: resolution, brightness, contrast, frameRate, saturation, torch, zoom.
Capabilities are supported properties and value ranges. Settings are the current properties values.
ApiRTC allows to access constraints, capabilities, settings on both local and remote streams, using the same methods. This means you can easily control both local and remote devices.
Stream.applyConstraints(constraints)
method returns a Promise resolved when all constraints are applied:stream.applyConstraints({
"audio": {},
"video": {
"frameRate": 10
}
}).then(() => {
... // constraints applied
});
Stream.getConstraints()
returns a Promise with all properties that were modified and their current values:// get stream constraints that were applied and their values
stream.getConstraints()
.then(constraints => {
console.log(constraints) // constraints object
}).catch((error) => {
... // error during process
});
Constraints values depend on the device capabilities. For example, on smartphones with multiple back cameras, sometimes the torch property is only attached to one of the camera.
In addition, supported properties can be queried using
Stream.getCapabilities()
that returns a Promise with accepted values ranges:// get stream capabilities values ranges
stream.getCapabilities()
.then(capabilities => {
console.log(capabilities) // capabilities object
}).catch((error) => {
... // error during process
});
Example of a
capabilities
data object:{
"audio": {
"autoGainControl": [ true, false ],
"channelCount": { "max": 1, "min": 1 },
"deviceId": "...",
"echoCancellation": [ true, false ],
"groupId": "...",
"latency": { "max": 0.085333, "min": 0.002666 },
"noiseSuppression": [ true, false ],
"sampleRate": { "max": 48000, "min": 48000 },
"sampleSize": { "max": 16, "min": 16 }
},
"video": {
"aspectRatio": { "max": 1920, "min": 0.001388888888888889 },
"deviceId": "...",
"frameRate": { "max": 30, "min": 0 },
"groupId": "...",
"height": { "max": 1080, "min": 1 },
"resizeMode": ["none", "crop-and-scale"],
"width": { "max": 1920, "min": 1 }
}
}
In this example
video.frameRate
property may be set between 0 and 30.getCapabilities()
may not work with all browsers. Also, returned capabilities may differ from a device to another.Finally, properties values can be checked with
Stream.getSettings()
that returns in a Promise all current settings:// get stream actual constraints settings
stream.getSettings()
.then(settings => {
console.log(settings) // settings object
}).catch((error) => {
... // error during process
});
Example of a
settings
data object:{
"audio": {
"autoGainControl": true,
"channelCount": 1,
"deviceId": "...",
"echoCancellation": true,
"groupId": "...",
"latency": 0.01,
"noiseSuppression": true,
"sampleRate": 48000,
"sampleSize": 16
},
"video": {
"aspectRatio": 1.333333333333,
"deviceId": "...",
"frameRate": 30,
"groupId": "...",
"height": 480,
"resizeMode": "none",
"width": 640
}
}
In this example
video.frameRate
is a supported property and it's actual value is 30.video.zoom
is not a supported property for this combination of device/camera/navigator as it is not present in the returned object.Noise reduction feature is available on apiRTC 5.0.0
ApiRTC allows to create a stream with a noise reduction filter.
applyAudioProcessor
helper manages the different stream states for you. (ie : switch from noisedReduction to normal mode)To start the noise reduction process on a Stream:
stream.applyAudioProcessor('noiseReduction').then((streamWithEffect) => {
...
})
This method returns a
streamWithEffect
Stream object; it is a encapsulated object of the base stream
with an noise reduction filter applied on it.It means that there both, the base
stream
and the streamWithEffect
stream, are still linked :- If the base
stream
audio is muted thestreamWithEffect
stream audio will be too, - If the base
stream
is released, thestreamWithEffect
stream will be too.
Both streams need to be handled by the application as the noise reduction process is going on.
To stop the noise reduction process:
// stop the noise reduction from base stream
stream.applyAudioProcessor('none').then((streamWithoutEffect) => {
...
})
If an error occurs during applyAudioProcessor() process, apiRTC will reject the promise but will try to restore stream with previous effect.
Additionally, ApiRTC gives you access to
Stream.startNoiseReduction
and Stream.stopNoiseReduction
methods.ApiRTC allows to create a background blurred stream or to add a background image based on an original stream.
applyVideoProcessor
helper manages the different stream states. (ie : switch from blur to background image ...)To start the blur process on a stream:
stream.applyVideoProcessor('blur').then((streamWithEffect) => {
...
})
This method returns a
streamWithEffect
Stream object; it is a encapsulated object of the base stream
with blur filter applied on it.It means that there both, the base
stream
and the streamWithEffect
stream, are still linked :- If the base
stream
audio is muted thestreamWithEffect
stream audio will be too, - If the base
stream
is released, thestreamWithEffect
stream will be too.
Both streams need to be handled by the application as the noise reduction process is going on.
Use the stream with effect as a local stream:
// display streamWithEffect media stream by attaching to a media element (like <video>)
streamWithEffect.attachToElement(videoDomElement)
// publish the streamWithEffect stream
conversation.publish(streamWithEffect).then(() => {
...
});
To stop the blur process:
// stop blur from original stream
stream.applyVideoProcessor('none').then((blurredStream) => {
...
})
Additionally, ApiRTC gives you access to
Stream.blur()
, Stream.unblur()
, Stream.backgroundImage()
, Stream.unBackgroundImage()
.