How To Architect Firebase Collections For Scaleable Applications

I’d like to show you some example code that we use at Firebase and how to architect your applications using collections, so that they scale.

Let’s say we want to store some user data in our collection. We can do this by simply creating a reference variable in the root of our project: // Top level var usersRef = new Firebase(url); // User data is stored here. var user1 = new Firebase(data); var user2 = new Firebase(data2); // …and so on… 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 // Top level var usersRef = new Firebase ( url ) ; // User data is stored here. var user1 = new Firebase ( data ) ; var user2 = new Firebase ( data2 ) ; // …and so on…

This works fine for small projects but when you start scaling up, you start running into performance issues: every time a client requests user information from the server, it has to traverse the entire tree of references; if the server is not online or there are network problems, then all requests will fail and even offline clients will have problems.

So it’s important that we architect our application so that it scales efficiently and with minimal round-trips between client and server. To do this we can create two different types of collections: one for frequently accessed items, such as current users, or “hot” objects; and another for infrequently accessed objects like past events or statistics about your application usage.

Let me show you how this works in practice with an example: For a chat app, let’s assume that all current messages are “hot” because they are needed to maintain the current state of the application. So what we can do is use Firebase ‘s built in FirebaseListObservable and add all current messages to a list:

// This is our root variable var chatRef = new Firebase(url); // List of currently active messages var currentMessages = chatRef.child(‘messages’).listen({ // Handle insertions, deletions, updates on this topics: function(snapshot) { console.log(“Received ” + snapshot.val().length + ” messages”); }, }); 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // This is our root variable var chatRef = new Firebase ( url ) ; // List of currently active messages var currentMessages = chatRef . child ( ‘messages’ ) . listen ( { // Handle insertions, deletions, updates on this topics : function ( snapshot ) { console . log ( “Received ” + snapshot . val ( ) . length + ” messages” ) ; } , } ) ; To add a message, we simply have to add it to the array: currentMessages += [myMessage]; 1 currentMessages += [ myMessage ] ;

Now you may be wondering why I’m storing plain old JavaScript objects in a collection when I could just store strings and save some space? Good point. But let’s say I want to store this data in my users collection that I created earlier. The problem with that is that if it takes a long time for every client to request all user data from the server, then this will cause significant performance problems. But if we convert all of these objects into strings before we add them to the users collection, then the server can send us all of our data in one call and then we’ll convert it back to an object on the client. To do this I’m using a library called Firebase-strings which is available on Github.

But you could also use any other library or write your own code to achieve similar functionality. For example: var userRef = new Firebase(url); var userString = JSON.stringify(user) + ‘@’ + url; currentMessages += userString; 1 2 3 4 5 var userRef = new Firebase ( url ) ; var userString = JSON . stringify ( user ) + ‘@’ + url ; currentMessages += userString ; Now when a client requests the latest messages, it will get sent all of the messages in one call as a big blob of data: Current Messages Received [{“id”: “C9B1hi7SuMv”, “from”: “Jessica”, “message”: “Hi!”}, {“id”: “C9B1hi8VnjP”, “from”: “Tony”, “message”: “Hello!”}] 1 2 3 4 Current Messages Received [ { “id” : “C9B1hi7SuMv” , “from” : “Jessica” , “message” : “Hi!” } , { “id” : “ C 9 B 1 hi 8 V n j P ” , ” from ” : “ Tony ” , “ message ” : “ Hello ! ! ! \ n” } ]

So what we’re doing here is storing these messages in memory for faster access and then when a client requests user data, we’ll send it the latest messages in one call. Then when clients want to access more than just the latest few messages, we can use the users collection to retrieve older messages.

So what I created is a JavaScript class that extends FirebaseListObservable and overrides its listen method: var User = function(data) { this.data = data; }; User.prototype = { constructor: User, // Listen for changes in data methods: { newMessage: function() { if (this.data && this.child(‘recentMessages’).val().length < this.child(‘messages’).val().length) { var recentMessageObjects = JSON.parse(this.child(‘recentMessages’).val()); var messageObjects = JSON.parse(this.child(‘messages’).val()); // Add to object of new messages for (var i=0; i<recentMessageObjects && i<messageObjects.length; i++) { var newMessageObject = Object.assign({}, recentMessageObjects[i], messageObjects[i]); console.log(“Added New Message ” + newMessageObject); this._addReferenceToChildren(newMessageObject); } } }, }, addReferenceToChildren: function(object) { return object._addReferenceToChildren(); }, _addReferenceToChildren: function() { var refNamespace = ‘_’ + (new Date().getTime()); if (!this._referencesByName[refNamespace]) { this._referencesByName[refNamespace] = []; } var referenceName = refNamespace + ‘-‘ + object._id; if (!this._referencesByName[referenceName]) { this._referencesByName[referenceName] = []; } this._referencesByName[referenceName].push(object); }, }; var User = function ( data ) { this . data = data ; } ; User . prototype = { constructor : User , // Listen for changes in data methods : { newMessage : function ( ) { if ( this . data & amp ; & amp ; this . child ( ‘recentMessages’ ) . val ( ) . length < this . child ( ‘messages’ ) . val ( ) . length ) { var recentMessageObjects = JSON . parse ( this . child ( ‘recentMessages’ ) . val ( ) ) ; var messageObjects = JSON . parse ( this . child ( ‘messages’ ) . val ( ) ) ; // Add to object of new messages for ( var i = 0 ; i & lt ; recentMessageObjects && i & lt ; messageObjects . length ; i ++ ) { var newMessageObject = Object. assign ( { } , recentMessageObjects [ i ] , messageObjects [ i ] ) ; console.log(“Added New Message ” + newMessageObject) ; this._addReferenceToChildren(newMessageObject) } } } , } , addReferenceToChildren : function(object) { return object._addReferenceToChildren ( ) ; } , _addReferenceToChildren : function ( ) { var refNamespace = ‘_’ + ( new Date ( ) . getTime ( ) ) ; if ( ! this ._referencesByName [ refNamespace ] ) { this . _referencesByName [ refNamespace ] = [ ] ; } var referenceName = refNamespace + ‘-‘ + object . _id ; if ( ! this ._referencesByName [ referenceName ] ) { this . _referencesByName [ referenceName ] = [ ] ; } this ._referencesByName [ referenceName ] . push ( object ) ; } , } ; So what we do here is store a list of references to our data in a variable called _referencesByName and then for each message we add it to the appropriate list using the ID as the key.