Get a count of total documents with MongoDB when using limit


I am interested in optimizing a "pagination" solution I'm working on with MongoDB. My problem is straight forward. I usually limit the number of documents returned using the limit() functionality. This forces me to issue a redundant query without the limit() function in order for me to also capture the total number of documents in the query so I can pass to that to the client letting them know they'll have to issue an additional request(s) to retrieve the rest of the documents.

Is there a way to condense this into 1 query? Get the total number of documents but at the same time only retrieve a subset using limit()? Is there a different way to think about this problem than I am approaching it?

No, there is no other way. Two queries - one for count - one with limit. Or you have to use a different database. Apache Solr for instance works like you want. Every query there is limited and returns totalCount.


Mongodb 3.4 has introduced $facet aggregation

which processes multiple aggregation pipelines within a single stage on the same set of input documents.

Using $facet and $group you can find documents with $limit and can get total count.

You can use below aggregation in mongodb 3.4

db.collection.aggregate([
  { "$facet": {
    "totalData": [
      { "$match": { }},
      { "$skip": 10 },
      { "$limit": 10 }
    ],
    "totalCount": [
      { "$group": {
        "_id": null,
        "count": { "$sum": 1 }
      }}
    ]
  }}
])

Even you can use $count aggregation which has been introduced in mongodb 3.6.

You can use below aggregation in mongodb 3.6

db.collection.aggregate([
  { "$facet": {
    "totalData": [
      { "$match": { }},
      { "$skip": 10 },
      { "$limit": 10 }
    ],
    "totalCount": [
      { "$count": "count" }
    ]
  }}
])

Times have changed, and I believe you can achieve what the OP is asking by using aggregation with $sort, $group and $project. For my system, I needed to also grab some user info from my users collection. Hopefully this can answer any questions around that as well. Below is an aggregation pipe. The last three objects (sort, group and project) are what handle getting the total count, then providing pagination capabilities.

db.posts.aggregate([
  { $match: { public: true },
  { $lookup: {
    from: 'users',
    localField: 'userId',
    foreignField: 'userId',
    as: 'userInfo'
  } },
  { $project: {
    postId: 1,
    title: 1,
    description: 1
    updated: 1,
    userInfo: {
      $let: {
        vars: {
          firstUser: {
            $arrayElemAt: ['$userInfo', 0]
          }
        },
        in: {
          username: '$$firstUser.username'
        }
      }
    }
  } },
  { $sort: { updated: -1 } },
  { $group: {
    _id: null,
    postCount: { $sum: 1 },
    posts: {
      $push: '$$ROOT'
    }
  } },
  { $project: {
    _id: 0,
    postCount: 1,
    posts: {
      $slice: [
        '$posts',
        currentPage ? (currentPage - 1) * RESULTS_PER_PAGE : 0,
        RESULTS_PER_PAGE
      ]
    }
  } }
])

there is a way in Mongodb 3.4: $facet

you can do

db.collection.aggregate([
  {
    $facet: {
      data: [{ $match: {} }],
      total: { $count: 'total' }
    }
  }
])

then you will be able to run two aggregate at the same time


It all depends on the pagination experience you need as to whether or not you need to do two queries.

Do you need to list every single page or even a range of pages? Does anyone even go to page 1051 - conceptually what does that actually mean?

Theres been lots of UX on patterns of pagination - Avoid the pains of pagination covers various types of pagination and their scenarios and many don't need a count query to know if theres a next page. For example if you display 10 items on a page and you limit to 13 - you'll know if theres another page..


MongoDB allows you to use cursor.count() even when you pass limit() or skip().

Lets say you have a db.collection with 10 items.

You can do:

async function getQuery() {
  let query = await db.collection.find({}).skip(5).limit(5); // returns last 5 items in db
  let countTotal = await query.count() // returns 10-- will not take `skip` or `limit` into consideration
  let countWithConstraints = await query.count(true) // returns 5 -- will take into consideration `skip` and `limit`
  return { query, countTotal } 
}

By default, the count() method ignores the effects of the cursor.skip() and cursor.limit() (MongoDB docs)

As the count method excludes the effects of limit and skip, you can use cursor.count() to get the total count

 const cursor = await database.collection(collectionName).find(query).skip(offset).limit(limit)
 return {
    data: await cursor.toArray(),
    count: await cursor.count() // this will give count of all the documents before .skip() and limit()
 };

You can do this in one query. First you run a count and within that run the limit() function.

In Node.js and Express.js, you will have to use it like this to be able to use the "count" function along with the toArray's "result".

var curFind = db.collection('tasks').find({query});

Then you can run two functions after it like this (one nested in the other)

curFind.count(function (e, count) {

// Use count here

    curFind.skip(0).limit(10).toArray(function(err, result) {

    // Use result here and count here

    });

});

It is possible to get the total result size without the effect of limit() using count() as answered here: Limiting results in MongoDB but still getting the full count?

According to the documentation you can even control whether limit/pagination is taken into account when calling count(): https://docs.mongodb.com/manual/reference/method/cursor.count/#cursor.count

Edit: in contrast to what is written elsewhere - the docs clearly state that "The operation does not perform the query but instead counts the results that would be returned by the query". Which - from my understanding - means that only one query is executed.

Example:

> db.createCollection("test")
{ "ok" : 1 }

> db.test.insert([{name: "first"}, {name: "second"}, {name: "third"}, 
{name: "forth"}, {name: "fifth"}])
BulkWriteResult({
    "writeErrors" : [ ],
    "writeConcernErrors" : [ ],
    "nInserted" : 5,
    "nUpserted" : 0,
    "nMatched" : 0,
    "nModified" : 0,
    "nRemoved" : 0,
    "upserted" : [ ]
})

> db.test.find()
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c8"), "name" : "forth" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c9"), "name" : "fifth" }

> db.test.count()
5

> var result = db.test.find().limit(3)
> result
{ "_id" : ObjectId("58ff00918f5e60ff211521c5"), "name" : "first" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c6"), "name" : "second" }
{ "_id" : ObjectId("58ff00918f5e60ff211521c7"), "name" : "third" }

> result.count()
5 (total result size of the query without limit)

> result.count(1)
3 (result size with limit(3) taken into account)

Try as bellow:

cursor.count(false, function(err, total){ console.log("total", total) })

core.db.users.find(query, {}, {skip:0, limit:1}, function(err, cursor){
    if(err)
        return callback(err);

    cursor.toArray(function(err, items){
        if(err)
            return callback(err);

        cursor.count(false, function(err, total){
            if(err)
                return callback(err);

            console.log("cursor", total)

            callback(null, {items: items, total:total})
        })
    })
 })

Thought of providing a caution while using the aggregate for the pagenation. Its better to use two queries for this if the API is used frequently to fetch data by the users. This is atleast 50 times faster than getting the data using aggregate on a production server when more users are accessing the system online. The aggregate and $facet are more suited for Dashboard , reports and cron jobs that are called less frequently.