Summoning builders…Compiling dreams…Polishing pixels…Aligning the stars to your roadmap…Brewing the backlog…Forging gold from ideas…Charting the constellations…Waking the render spirits…

    Insights/Cloud & Infrastructure/How To Architect Firebase Collections For Scaleable Applications

    How To Architect Firebase Collections For Scaleable Applications

    Marco Lopes
    November 05, 2019
    1.6k views
    Cloud & Infrastructure
    app
    development
    firebase
    How To Architect Firebase Collections For Scaleable Applications

    Scaling a Firebase application starts with a deceptively simple question: What will each screen need to read, and who should be allowed to read it? The answer matters more than whether your collections look like a tidy representation of your business.

    A small application can often get away with loading a large dataset and filtering it on the client. As that dataset grows, the same approach increases latency, bandwidth, and database costs. A scalable design fetches a bounded amount of relevant data, supports predictable queries, and keeps authorization understandable.

    This guide uses Firebase capabilities and SDK conventions available in 2019; tools and interfaces may have changed since then.

    We’ll focus on Cloud Firestore, Firebase’s document database, and use a project-management application as an example. Firebase Realtime Database has a different data model and different querying constraints; advice about its JSON tree should not be applied directly to Firestore collections.

    Understand what Firestore actually reads

    Firestore organizes data into collections and documents. Documents contain fields and can have subcollections:

    users/{userId}
    projects/{projectId}
    projects/{projectId}/tasks/{taskId}
    

    Reading users/alice does not traverse every user or download an entire tree. Likewise, reading a project document does not retrieve its task subcollection. You explicitly request the documents or query results you need.

    Creating a reference is also not a network operation:

    // Assumes the Firebase JavaScript SDK has been initialized.
    const db = firebase.firestore();
    const userRef = db.collection('users').doc(userId);
    
    // This requests the document through the SDK.
    const snapshot = await userRef.get();
    

    The scaling problem is therefore not the existence of a root-level reference. It is usually an unbounded query, an oversized document, excessive listeners, write contention, or an access pattern the schema does not support efficiently.

    Firestore queries use indexes rather than scanning every document in a collection. A large collection is not inherently a problem; returning too many documents from it is.

    1. Design around access patterns

    Before creating collections, list the reads and writes behind your main screens.

    For a project-management application, that might include:

    • Display one user’s public profile.
    • List the projects owned by the signed-in user.
    • Show the next 25 open tasks in a project.
    • Update a task’s status.
    • Display a project summary without downloading all tasks.
    • Browse completed work separately from active work.

    For each operation, identify its filters, sort order, expected result size, and authorization boundary. Also ask whether it needs live updates or whether a one-time fetch is sufficient.

    This gives you a practical schema test: Can each common screen load with a small, predictable number of reads?

    Do not aim for the fewest collections. Aim for clear access paths. A single enormous document may look simple, but it creates problems when unrelated data must be read or updated together.

    2. Choose document boundaries deliberately

    A useful starting structure is:

    users/{userId}
    userSettings/{userId}
    projects/{projectId}
    projects/{projectId}/tasks/{taskId}
    projects/{projectId}/summaries/current
    

    A project document might contain an owner ID, name, status, and timestamps. Each task gets its own document containing its title, status, assignee ID, and creation time.

    Keep documents bounded

    Firestore documents have a maximum size of 1 MiB. More importantly, documents are the normal unit of client reads: retrieving a document retrieves its fields, not just the handful a screen displays.

    Avoid embedding an ever-growing task array inside a project document. Independent task documents allow pagination, smaller updates, and more focused listeners. They also prevent every task edit from becoming a write to the same parent document.

    Small, naturally bounded fields, such as a short list of labels, can remain arrays. The distinction is whether the data has a predictable limit and belongs in the same read and write lifecycle.

    Separate public and private data

    If other users may read a profile, keep private settings in a different document. Security rules cannot selectively hide fields within a document that a client is allowed to read.

    Document boundaries are therefore security boundaries as well as performance boundaries.

    3. Use subcollections for scope, not automatic cleanup

    Subcollections work well when data is normally accessed within a parent context. Project tasks are a natural example:

    const tasksRef = db
      .collection('projects')
      .doc(projectId)
      .collection('tasks');
    
    await tasksRef.add({
      title: 'Review onboarding flow',
      status: 'open',
      assigneeId: userId,
      createdAt: firebase.firestore.FieldValue.serverTimestamp()
    });
    

    The automatically generated document ID avoids the concentrated key patterns that sequential IDs can create under heavy writes.

    A top-level tasks collection with a projectId field is also valid. It can simplify global task queries. Subcollections make project-scoped paths explicit; top-level collections make cross-project access more direct.

    Firestore introduced collection group queries in 2019, allowing queries across subcollections sharing a collection ID. These require suitable indexes and security rules, so plan cross-project access rather than assuming nesting makes it impossible, or automatically safe.

    Important pitfall: deleting a project document does not delete its task documents. Build an explicit, authorized cleanup process, typically on a trusted backend, and handle large deletions in batches.

    4. Make every list query bounded and indexed

    A scalable task screen should ask for the tasks it can actually display:

    const firstPageQuery = tasksRef
      .where('status', '==', 'open')
      .orderBy('createdAt', 'desc')
      .limit(25);
    
    const firstPage = await firstPageQuery.get();
    const lastDocument = firstPage.docs[firstPage.docs.length - 1];
    
    if (lastDocument) {
      const nextPage = await tasksRef
        .where('status', '==', 'open')
        .orderBy('createdAt', 'desc')
        .startAfter(lastDocument)
        .limit(25)
        .get();
    
      // Render or store nextPage.docs.
    }
    

    This filter-and-sort combination may require a composite index. Firestore reports missing indexes and provides a link to create them. Keep index configuration in source control and deploy it alongside application changes rather than treating console setup as an undocumented production step.

    Prefer cursor pagination over offsets. Skipped documents still create unnecessary work and can incur read charges with offset-based queries.

    Practical details matter:

    • Ensure every task has the field used for ordering; documents missing it will not appear in that ordered query.
    • Define how the interface handles tasks moving between pages when their values change.
    • Avoid subscribing to an entire collection just to calculate a count.
    • Test realistic result sizes, not only a handful of development records.

    5. Denormalize selectively to reduce extra reads

    Firestore does not provide relational joins. A task list containing an assignee ID for every row may otherwise require additional profile reads to display names and avatars.

    For frequently displayed information, consider storing a small snapshot on the task:

    {
      title: 'Review onboarding flow',
      assigneeId: 'user_123',
      assigneeDisplayName: 'Design reviewer',
      status: 'open'
    }
    

    This trades extra storage and update work for simpler reads. It is useful when the duplicated information changes infrequently or when slightly stale display data is acceptable.

    Keep the canonical value in the user document. Decide whether copied names should reflect current profiles or preserve historical attribution. If current values are required, define a backend propagation process and make retries safe.

    Never treat copied display fields as authorization evidence. Permissions should depend on trusted identity and ownership or membership data, not editable labels.

    For project summaries, maintain selected counters or totals instead of downloading all tasks. In the 2019 feature set, do not assume a modern server-side aggregation API is available. Use controlled writes or backend-maintained summaries where needed.

    6. Separate hot and cold data only when it helps

    Active and historical data often have different access patterns. An active project may need live task updates; an archived project may only need occasional reads.

    That does not automatically require separate collections. A status filter with an appropriate index can keep the active query small even when historical documents remain in the same collection.

    Separate collections become useful when they clarify different schemas, retention policies, permissions, or backend processing. They can also help avoid accidentally attaching live listeners to historical records.

    Do not assume moving data into an archive collection creates a cheaper storage tier. Any savings come from changing application behavior, for example, reading historical data less often, not from the collection’s name.

    Similarly, split large, rarely used details from frequently requested metadata when those details would otherwise inflate every document read.

    7. Plan write consistency and contention

    Independent task documents spread writes better than a shared project document updated for every action. However, a summary counter can reintroduce a hotspot if every task change updates that same summary.

    For modest activity, an atomic increment may be sufficient. For high write rates, consider distributed counters or asynchronous aggregation, accepting that summaries may lag behind the underlying records.

    Use the appropriate consistency mechanism:

    • Batched writes commit related writes atomically when no read-dependent decision is needed.
    • Transactions read current state and commit dependent changes, retrying when concurrent changes conflict.
    • Backend jobs handle larger fan-outs and workflows that cannot fit into one atomic operation.

    In 2019, a batched write supports up to 500 write operations. Transactions require connectivity, while client batched writes can be queued offline. Do not put external side effects inside a transaction callback, because that callback may run more than once.

    Cloud Functions event handlers should also tolerate retries. A naive “increment on every invocation” implementation can double-count if the same event is processed again.

    8. Treat rules, listeners, and offline behavior as architecture

    Security rules are not filters. A query must satisfy the rules’ constraints; Firestore does not fetch everything and silently remove unauthorized documents.

    For owner-only projects, the application might query:

    const projectsQuery = db.collection('projects')
      .where('ownerId', '==', firebase.auth().currentUser.uid)
      .limit(20);
    

    The corresponding rules must enforce ownership, validate creation, and prevent clients from changing protected fields. Subcollections also need appropriate rules; a parent document rule does not automatically grant access to its descendants. Trusted server SDKs bypass Firestore Security Rules, so protect backend credentials and validate server-side inputs separately.

    Live listeners should have equally deliberate lifetimes:

    const unsubscribe = firstPageQuery.onSnapshot(
      snapshot => renderTasks(snapshot.docs),
      error => showTaskError(error)
    );
    
    // Call when the screen is no longer active.
    unsubscribe();
    

    Offline support does not mean every document is available without a network. Mobile clients support persistent caching by default; web persistence must be enabled explicitly in the 2019 SDK and has browser limitations. Clients can work with cached data and queue supported writes, but uncached reads and transactions need connectivity to succeed as expected.

    Test permission failures, reconnects, stale summaries, and interrupted cleanup jobs, not just successful online reads. Monitor read volume and listener behavior as usage grows.

    Takeaway

    Scalable Firestore architecture is about bounded work: small documents, focused queries, controlled listeners, and writes that avoid shared hotspots. Start with access patterns, make authorization explicit, and denormalize only where the read benefit justifies the consistency cost. Collections provide the structure; disciplined access patterns make that structure scale.

    Conversation (0)

    No comments yet. Start the conversation.

    Join the conversation

    Create a free member account to comment.