WebRTC 4. File transfers with Ably
Introduction
WebRTC Datachannels API allows developers to transmit arbitrary data directly between two users.
The idea behind file transfers using WebRTC is to convert files into buffers or DataUrl, transmit them in chunks, then compile them back to the file it originally was, on the receiving end.
In this tutorial, we will implement file transfer with data channels using WebRTC and Ably, that will allow us to share files with multiple people in separate chats.
This is very similar to our initial lesson on data channels.
Step 1 – Set up a free account with Ably
In order to run these tutorials locally, you will need an Ably API key. If you are not already signed up, you should sign up now for a free Ably account. Once you have an Ably account:
- Log into your app dashboard
- Under “Your apps”, click on “Manage app” for any app you wish to use for this tutorial, or create a new one with the “Create New App” button
- Click on the “API Keys” tab
- Copy the secret “API Key” value from your Root key and store it so that you can use it later in this tutorial
Step 2 – Choosing a WebRTC library
Dealing with WebRTC directly might be a bit tedious as it generally involves a lot of lines of code. However, there are many WebRTC libraries available that provide high-level abstractions and expose only a few methods to users, while handling major uplifting themselves. WebRTC has been an emerging standard and is still somewhat in flux, so it’s crucial to make sure that whichever library you choose is up to date and well maintained.
In all the chapters of this tutorial, we will be using simple-peer – a simple WebRTC library for video/voice and data channels.
Step 3 – Designing a simple HTML layout
Create a new file called index.html
and add the following code:
<!DOCTYPE html>
<html>
<head>
<title>Ably WebRTC DataChannel Demo</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css">
</head>
<body>
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
<script src="https://cdn.ably.io/lib/ably.min-1.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/simple-peer/9.1.2/simplepeer.min.js"></script>
<script src="ably-datachannel.js"></script>
<script src="connection-helper.js"></script>
<div class="container-fluid" style="margin-top: 5em;">
<div class="container" id="join">
<h4 id="online">Users online (0) </h4>
<ul id="memberList"></ul>
</div>
<div class="container" id="chat" style="display:none;">
<input type="file" id="file" class="form-control" placeholder="Connecting to the peers, please wait"></input>
<button class="demo-chat-send btn btn-success" id="chatButton" onClick="sendMessage()">Send</button>
<ul id='list'>
</ul>
</div>
</div>
</body>
<style>
small {
border-bottom: 2px solid black;
}
li {
list-style: none;
}
</style>
</html>
What is going on in the code block above?
The code snippet above is a basic HTML declaration in which we did the following:
- Referenced the Bootstrap CSS library
- Referenced the Ably JavaScript library
- Referenced the simple peer JavaScript library.
- Referenced a JavaScript file named
ably-datachannel.js
(which we will create soon). - Referenced a JavaScript file named
connection-helper.js
(which we will create soon). - Declared a
div
with the id ofjoin
which will hold the list of users online. - Declared a
div
with the id ofchat
which holds the input file to select files as well as the send filebutton
.
Step 4 – Defining the connection helper class
In Step 3 above, we Referenced a library called connection-helper.js
. This library helps to manage the simple-peer connections, so as to keep our code organized, as we will be having multiple instances of the connection.
Create a new file called connection-helper.js
and paste the following code:
class Connection {
constructor(remoteClient, AblyRealtime, initiator) {
console.log(`Opening connection to ${remoteClient}`);
this._remoteClient = remoteClient;
this.isConnected = false;
this._p2pConnection = new SimplePeer({
initiator: initiator,
trickle: false,
});
this._p2pConnection.on('signal', this._onSignal.bind(this));
this._p2pConnection.on('error', this._onError.bind(this));
this._p2pConnection.on('connect', this._onConnect.bind(this));
this._p2pConnection.on('close', this._onClose.bind(this));
this._p2pConnection.on('data', this._onData.bind(this));
}
handleSignal(signal) {
this._p2pConnection.signal(signal);
}
send(msg) {
this._p2pConnection.send(msg);
}
_onSignal(signal) {
AblyRealtime.publish(`ablyWebRTC-signal/${this._remoteClient}`, {
user: clientId,
signal: signal,
});
}
_onConnect() {
this.isConnected = true;
console.log('connected to ' + this._remoteClient);
}
_onClose() {
console.log(`connection to ${this._remoteClient} closed`);
delete connections[this._remoteClient];
}
_onData(data) {
receiveMessage(this._remoteClient, data)
}
_onError(error) {
console.log(`an error occurred ${error.toString()}`);
}
}
What is going on in the code block above?
In the code block above, we have defined a connection
class which makes use of the simple-peer
library to manage connections.
In our constructor, we accept 3 parameters:
-
remoteClient
: this refers to the other party we want to connect to. -
AblyRealtime
: this refers to an instance of an Ably channel -
initiator
: this is a Boolean parameter that states whether this peer is the one initiating the connection or not.
Next, we move ahead to initiate the simple-peer
library and bind some methods to events exposed by simple-peer
such as signal
, error
, connect
, close
and data
.
Let us now understand the methods defined in the class:
-
handleSignal
: this method is called when a signal has been sent via the realtime channel. This method passes the signal to the current peer connection. -
send
: this method is used to send messages to the other peer. The method, in turn, calls the send method of thesimple-peer
instance. -
_onSignal
: this method is called by thesimple-peer
library when it wants to send a signal to the other peer. In this method, we make use of the Ably realtime channel to publish the signal to the other peer. -
_onConnect
: this method sets the class propertyisConnected
to true. This is an indicator that the peers have been connected. -
_onClose
: this method deletes the current connection instance from an object calledconnections
, which we will define later on in theably-datachannel.js
file. -
_onData
: this method is called when a new message has been sent to the other peer. Here, it calls another method calledreceiveMessage
, which we will later define in theably-datachannel.js
file. -
_onError
: should an error occur in our connection, this method will be called. Currently, we just log out the details of the error.
Step 5 – Displaying online users
Before we can send messages via WebRTC, we need to verify that the other peer we want to connect to is online, as WebRTC would not connect with offline peers.
Create a file called ably-datachannel.js
and add the following code:
var messageList = {}
var membersList = []
var connections = {}
var currentChat
var apiKey = 'XXX_ABLY_API_KEY'
var clientId = 'client-' + Math.random().toString(36).substr(2, 16)
var realtime = new Ably.Realtime({ key: apiKey, clientId: clientId })
var AblyRealtime = realtime.channels.get('ChatChannel')
AblyRealtime.presence.subscribe('enter', function(member) {
AblyRealtime.presence.get((err, members) => {
membersList = members
renderMembers()
})
})
AblyRealtime.presence.subscribe('leave', member => {
delete(connections[member.client_id])
if (member.client_id === currentChat) {
currentChat = undefined
document.getElementById('chat').style.display = 'none'
}
AblyRealtime.presence.get((err, members) => {
membersList = members
renderMembers()
})
})
AblyRealtime.presence.enter()
function renderMembers() {
var list = document.getElementById('memberList')
var online = document.getElementById('online')
online.innerHTML = 'Users online (' + (membersList.length === 0 ? 0 : membersList.length - 1) + ')'
var html = ''
if (membersList.length === 1) {
html += '<li> No member online </li>'
list.innerHTML = html
return
}
for (var index = 0; index < membersList.length; index++) {
var element = membersList[index]
if (element.clientId !== clientId) {
html += '<li><small>' + element.clientId + ' <button class="btn btn-xs btn-success" onclick=prepareChat("' + element.clientId + '")>chat now</button></small></li>'
}
}
list.innerHTML = html
}
What is going on in the code block above?
Before we move forward, please ensure that you have set the value of apikey
to your private Ably API key. The other variables are explained below:
-
messageList
: this is an object which will contain keys of each client you are currently chatting with, or that has sent a message, with an array of what the messages are. -
membersList
: this is an array of all currently online members you can chat with. -
connections
: this is an object which will contain keys of each client you have a current chat with, or that has sent you a message, with their current connection object. -
apiKey
: this is your API key for Ably as generated in step 1 -
clientId
: this is a unique identification of the current person who wants to connect to both Ably and WebRTC. In your application, you might need to get this key from a database or some other authentication methods. -
realtime
: an instance of Ably -
AblyRealtime
: an instance of an Ably channel.
First, we need to make an enter
subscription, so that we are notified every time a new member joins our channel. We do so by using AblyRealtime.presence
. In the callback, we get a list of all current members and add it to our member’s list. After this, we call on the render members method.
Also, as we subscribed to to Enter
events, we subscribed to Leave
events, so we can keep track of members who have left the channel.
We need to call the AblyRealtime.presence.enter()
method so Ably is aware that we want to enter the presence channel and keep track of the new client.
Lastly, we have our renderMembers
method, which loops through the membersList
, and appends them as list items to the ul
tag with the id of memberList
.
Step 6 – Sending files
We have been able to identify who is online and ready to receive instant files using WebRTC data channels, now we move to the part where we send files. Add the following to the end of your ably-datachannel.js
file.
function PrepareChat(client_id) {
if (client_id === clientId) return
// Create a new connection
currentChat = client_id
if (!connections[client_id]) {
connections[client_id] = new Connection(client_id, AblyRealtime, true)
}
document.getElementById('chat').style.display = 'block'
render()
}
AblyRealtime.subscribe(`rtc-signal/${clientId}`, msg => {
if (!connections[msg.data.user]) {
connections[msg.data.user] = new Connection(msg.data.user, AblyRealtime, false)
}
connections[msg.data.user].handleSignal(msg.data.signal)
})
function sendMessage() {
if (!messageList[currentChat]) {
messageList[currentChat] = {}
}
var MAX_FSIZE = 2.0
var fileInput = document.getElementById('file')
var file = fileInput.files[0];
var fileReader = new FileReader();
if (file) {
var mbSize = file.size / (1024 * 1024);
if (mbSize > MAX_FSIZE) {
alert("Your file is too big, sorry.");
// Reset file input
return;
}
fileReader.onload = (e) => { onReadAsDataURL(e, null, file) }
fileReader.readAsDataURL(file)
render()
}
}
What is going on in the code block above?
To share files with a user/member, we need to click the chat now
button next to their name. This button, in turn, calls the prepareChat
method, which we have just defined.
The prepareChat
method receives a single argument, which is the id of client you want to send files to. We check that the user is not trying to send files to his own instance. Although it isn’t possible for the user to send files to his own instance, we do this as an extra layer of security. At this point, if the user does not have an active connection with the other peer, it means he is the one initiating the chat/file transfer, so we create a new connection with that client_id
, passing the initiator status to be true. After this, we call the render
method to render the files that might have been sent by the other peer.
For the two pairs to connect, the other peer has to be signaled of the incoming connection. This is where our subscription comes in. We subscribe to an event specifically for the current client’s id, which we defined the event as ablyWebRTC-signal/${clientId}
. Here, we check if the current user has a connection instance of the current client trying to connect, if not, we initiate a new connection for the other party, and surely the initiator flag is set to false. We then call on the handleSignal
method of the connection class, to handle the signal coming in and connect the two pairs.
Now, we can go ahead to send files.
In our sendMessage
method, we use the currentChat
variable to figure out who it is that we are trying to send the file to. We also check that the file about to be sent is not larger than our file limit. Here, it is set as 2mb, feel free to increase yours. We then initiate a new fileReader
instance. Because the fileReader
is an asynchronous API, we have to wait till it’s loaded before we can do anything with it. Here, we defined a new function for it to fire when it’s onload
event is ready. We passed the event, a null and the file we want to send to a function called onReadAsDataURL
. We then ask the fileReader
to read the file as a DataUrl.
Note: Files can be read as ArrayBuffers
, DataUrls
, BinaryStrings
, and Text
. The same concept of sending the file in chunks follows for the other read types. In this lesson, we are using the DataUrl type.
What happens in the onReadAsDataURL method?
function onReadAsDataURL(event, text, file) {
var chunkLength = 1024;
var data = {}; // data object to transmit over data channel
data.name = file.name
data.size = file.size
if (event) {
text = event.target.result; // on first invocationgettrue;we
data.stringSize = text.length
messageList[currentChat][file.name] = {
completed: true,
sender: 'me',
downloadLink: text,
chunks: []
}
render()
}
if (text.length > chunkLength) {
data.message = text.slice(0, chunkLength); // getting chunk using predefined chunk length
} else {
data.message = text;
data.last = true;
}
data.sender = clientId
connections[currentChat].send(JSON.stringify(data));
var remainingDataURL = text.slice(data.message.length);
if (remainingDataURL.length) {
onReadAsDataURL(null, remainingDataURL, file); // continue transmitting
}
}
The method above accepts three parameters which are:
-
event
: the instance of the event when thefileReader
was initially loaded. This would be passed the first timefileReader
initiates the method. -
text
: this refers to the remaining part of the data that was chunked. This would be passed so far there is still data pertaining to the file being sent. -
file
: this refers to an instance of the file being sent.
We first define the limit for chunks which we will be sending in batches. Here, we want to send only 1024 bytes at a time.
Next, we declare an object called data, which will contain metadata of the file we want to send.
The if(event)
block caters for instances when it is initiated by the fileReader
itself, and not the remaining part of the file. In this block, we set the text
variable to the result of the fileReader
, which is a DataUrl string. We then push a new object key, with the name of the file we want to send to our file conversation with the other peer.
Next, we check if the length of the DataUrl is greater than the defined chunk size. If true, we slice the text, else, we send the text, with an extra property telling the other peer this is the last part of the data. This is useful to know when all parts of the file have been received.
Step 7 – Receiving files
Receiving the files involves storing the chunks in an array, then joining the chunks together when all parts have been received. Let’s take a look at the receiveMessage
function below:
function receiveMessage(client_id, data) {
if (!messageList[client_id]) {
messageList[client_id] = {}
}
data = JSON.parse(data)
if (!messageList[client_id][data.name]) {
messageList[client_id][data.name] = {
sender: data.sender,
completed: false,
downloadLink: '',
stringSize: data.stringSize,
chunks: []
}
}
messageList[client_id][data.name].chunks.push(data.message)
if (data.last) {
var file = messageList[client_id][data.name].chunks.join('')
messageList[client_id][data.name].downloadLink = file;
messageList[client_id][data.name].completed = true;
}
renderMembers()
render()
}
function render() {
if (!messageList[currentChat]) {
return
}
var list = document.getElementById('list')
var html = ''
if (Object.keys(messageList[currentChat]).length === 0) {
html += '<li>No files available here</li>'
list.innerHTML = html
return
}
Object.keys(messageList[currentChat]).forEach((key) => {
var element = messageList[currentChat][key]
var downloadPercent = (element.chunks.join('').length / element.stringSize) * 100
var downloadLink = element.completed ? '<a target =_blank download=' + key + ' href=' + element.downloadLink + ' > download now </a>' : 'receiving ' + downloadPercent + '%'
html += '<li><small> ' + element.sender + ' : ' + key + '</small> ' + downloadLink + ' </li>'
});
list.innerHTML = html
renderMembers()
}
What is going on in the code block above?
In the receiveMessage
method, we receive the file here with the id of the user who sent it, then create a new key with the name of the file we want to receive in the object that belongs to the sender. Next, we set the properties of completed
to false, the downloadLink
to empty. The size of the string we are expecting in total is received as stringSize
, then we defined a chunks array where the chunks would be stored.
When we receive the property called last
, we join the array as one, and then set it as the downloadLink
, while setting the completed property to false.
The downloadLink
property herein would be a valid DataUrl
Finally, we defined the much loved render
` method, which loops through all our messages and shows the percentage received for non-completed files and the download link for completed files.
At this point, if we open the index.html
file in two separate browser windows we should be able to send and receive messages as seen below:
Step 8 – Adding audio notifications
In our current app, we have a new badge on the edge of the client’s id, which shows that someone has sent us a new message, however, that requires that the user must be on the page and be actively looking for the new status before he notices that a new message has been sent to him by someone who is not his current chat. We can improve this by adding audio notifications.
Let’s add the following at the beginning of our ably-datachannel.js
file:
var audio = new Audio('https://notificationsounds.com/soundfiles/a86c450b76fb8c371afead6410d55534/file-sounds-1108-slow-spring-board.mp3');
Here, we have created a new Audio element with an mp3 sound. This sound can be any sound you like. I used a random link I saw off the internet here.
Next, let us update the receiveMessage
method to be able to play the notification
function receiveMessage(client_id, data) {
if (!messageList[client_id]) {
messageList[client_id] = {}
}
data = JSON.parse(data)
if (!messageList[client_id][data.name]) {
audio.play()
messageList[client_id][data.name] = {
sender: data.sender,
completed: false,
downloadLink: '',
stringSize: data.stringSize,
chunks: []
}
}
messageList[client_id][data.name].chunks.push(data.message)
if (data.last) {
var file = messageList[client_id][data.name].chunks.join('')
messageList[client_id][data.name].downloadLink = file;
messageList[client_id][data.name].completed = true;
audio.play()
}
renderMembers()
render()
}
Testing our app
To test the messaging system we have just built, serve your index.html
file as a static file via any server of your choice.
Using Node.js, install the http-server
package:
npm install http-server -g
Once the http-server
package is done installing, at the root of your working directory, run:
http-server
Navigate to http://127.0.0.1:8081
to view the demo.
Live demo
To try this example yourself, Open this demo in a new browser window to see WebRTC file transfer in action.
Step 9 – Conclusion
In this tutorial, we have seen how to use WebRTC data channels in conjunction with Ably to create realtime file share. Do you have any thoughts on making this lesson better? Let us know
Next: Chapter 4 – Less code, more efficient screen sharing with Ably