How long should you lag your replica? Thats up to you, but as a general rule of thumb 8 hours would leave you reasonable time to detect a data problem and take corrective action.

MongoDB now has this capability with the 1.7.x versions of MongoDB. For now, you will have to use the nightly builds in order to have the capabilities. But on release of 1.7 it will be generally available. Here is how it works.

Setup a replica set like normal. But be sure to specify a slave with some amount of lag. It’s important to make sure you set priority=0 on this slave so it never automatically becomes master. Thus, it makes sense to always have at least 1 primary and 2 replicas in a lagged replica configuration. 1 primary, 1 replica for failover, then a lagged replica to ensure data safety.

In the above example, here is the config:

> c={_id:"sfly",
         members:[
             {_id:0,host:"host_a:27017"},
             {_id:1,host:"host_b:27017"},
             {_id:2,host:"host_c:27017",priority:0,slaveDelay:120},
             {_id:3,host:"host_d:27017",arbiterOnly:true}]
}
>
{
	"_id" : "sfly",
	"members" : [
		{
			"_id" : 0,
			"host" : "host_a:27017"
		},
		{
			"_id" : 1,
			"host" : "host_b:27017"
		},
		{
			"_id" : 2,
			"host" : "host_c:27017",
			"priority" : 0,
			"slaveDelay" : 120
		},
		{
			"_id" : 3,
			"host" : "host_d:27017",
			"arbiterOnly" : true
		}
	]
}
> rs.initiate(c);
{
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}
> rs.conf()
{
	"_id" : "sfly",
	"version" : 1,
	"members" : [
		{
			"_id" : 0,
			"host" : "host_a:27017"
		},
		{
			"_id" : 1,
			"host" : "host_b:27017"
		},
		{
			"_id" : 2,
			"host" : "host_c:27017",
			"priority" : 0,
                        "slaveDelay" : 120
		},
		{
			"_id" : 3,
			"host" : "host_d:27017",
			"arbiterOnly" : true
		}
	]
}

The idea presented in the blog is to utilize the MongoDB findAndModify command to pluck sequences from the DB using the atomic nature of the command.

counter=db.command("findandmodify", "seq", query={"_id":"users"},update={"$inc":{"seq":1}})
f={"_id":counter['value']['seq'],"data":"somedata"}
c.insert(f)

When using this technique each insert would require both the insert as well as the findAndModify command which is a query plus an update. So now you have to perform 3 operations where it used to be one. Not only that, but there are 3 more logical I/O’s due to the query, and those might be physical I/O’s. This pattern is easily seen with the mongostat utility.

Maybe you still meet your performance goals. But then again maybe not.

I did some testing to play with the various options. I compared a complete insert cycle with a unique key. The test is a simple python program that performs inserts using pymongo. The program is a single process and I ran 3 concurrent processes just so simulate a bit of concurrency. The save uses safe_mode=False. I tested the findAndModify approach to the native BSON objectId approach vs Python UUID generation approach.

The results are:

So clearly if the problem being solved can be achieved using the native BSON objectId type it should be. This is the fastest way to save data into MongoDB in a concurrent application.

f={"data","somedata"}    # let MongoDB generate objectId for _id
c.insert(f)

That said, what if auto-increment / concurrent unique key generator is still required? One option would be to use a relational store with a native sequence generation facility like PostgreSQL. PostgreSQL, in my testing, achieved 389,000 keys/sec when fetching from a single sequence using about 30 processes. Thus fetching sequences clearly outpaces the ability for MongoDB to insert them. Something like the following is possible:

cur.execute("nextval('users_seq')")
s=cur.fetchone()
f={"_id":s[0],"data":"somedata"}
c.insert(f)

The stack used in this test is:
- Sun X2270 dual quad core AMD 2376, 24GB RAM, 2 100GB SATA Drives, software RAID.
- MongoDB 1.5.7
- PostgreSQL 8.4.2
- Python 2.6.4
- Pymongo 1.7
- Linux Centos 2.6.18-128.el5 x86_64

Sure there are options like cacti out there, but this is just a simple raw utility vs something designed for a larger environment. Simple.

Feedback would be awesome.

http://github.com/kgorman/mongo_graph

So how does one manage this technique in MongoDB? If you’re not familiar MongoDB is a fairly new database that is non-relational and schema-less. However, data density and clustering is still important. Anytime you can reduce the amount of logical or physical I/O to return a query that is a good thing.

With SERVER-1054, the MongoDB folks implemented one key feature that helps one manage data clustering in MongoDB; the ability to show what file/offset a given document lives in. This allows the inspection of the location, and action to be taken. Think of it as a measure of how fragmented your data is inside blocks.

> var arr=db.photos.find({}, {'$diskLoc': 1}).showDiskLoc()
> for(var i=0; i<arr.length(); i++) {
    printjson(arr[i].$diskLoc.offset/512)
  }
3241650.34375
3241650.4453125
>

In the above example, both of these rows live in block # 3241650. Thus the data is dense.

If a given set of documents are typically queried together, then they should be as dense as possible. Additionally if they are in contiguous blocks that is good too.

In a traditional RDBMS data store there are various techniques to re-cluster the data by a given key to re-arrange the data densely. For instance using CREATE TABLE AS SELECT * FROM foo ORDER BY mykey. However, it’s mostly a one time affair because future inserts may not be dense.

In MongoDB depending on the design, that may or may not be required. A design pattern called embedding can alleviate many of the typical problems associated with data clustering and AUTOMATICALLY keep your collection dense. Thus further making the MongoDB seem much faster than a traditional RDBMS.

Let me give an example to illustrate. Let’s give the following relational data model:

postgres=# \d photos
         TABLE "public.photos"
  COLUMN   |     Type      | Modifiers 
-----------+---------------+-----------
 id        | integer       | 
 user_id   | integer       | 
 file_path | character(42) |

And the typical access path is:

  SELECT * FROM photos WHERE user_id = 10;

Then one can expect results (worse case) where all the results are in different blocks. Thus at least 3 I/O operations to return this query. If they were dense, they would be all in one block.

postgres=# select ctid, * from photos where user_id=10;
 ctid  | id | user_id |                 file_path                  
-------+----+---------+--------------------------------------------
 (0,1) |  1 |      10 | /home/foo/1.jpg                           
 (22,2) | 24 |      10 | /home/foo/2.jpg                           
 (334,3) | 23 |      10 | /home/foo/3.jpg

In MongoDB the following model can be used to *always* keep the data dense and tightly clustered.

photos=
{ "_id" : ObjectId("4c252807164314895e44fb6d"),
  "user_id" : 10,
  "paths" : ['/home/foo/1.jpg','/home/foo/2.jpg','/home/foo/3.jpg']
}

And a query would be:

db.photos.find({"owner":10})

The data payload is exactly 1 I/O. As the embedded document grows over the block size it would start to span multiple blocks. So this is an additional design consideration. Keep embedded documents less than the block size or you may not be able to see additional benefits.

Embedding may not always be possible. But if one is aware of the potential I/O savings when performing the design then it’s just another data point to making a more intelligent and fast performing data store.

MongoDB does not yet have the simple capability to rebuild a collection and re-order the data in a simple operation. This is the technique used on the RDBMS side pretty commonly and shown in my previous post examples. So design with that in mind.

I try not to blog about blogging, but I couldn’t help it in this case. The upgrade is pretty compelling, and I thought I would share my thoughts.

- Stop I/O temporarily
- Snapshot the filesystem (OS snapshot, rsync, whatever)
- Release I/O
- Backup any logs needed to recover point in time

This technique works for many different data stores. In the article I only show Oracle. But many other databases have the same capabilities for backups. Here are some examples:

PostgreSQL:

SELECT pg_start_backup('label');
-- snapshot the DB here
SELECT pg_stop_backup();
-- backup wal logs here

You can find all the details of this kind of backup in the PostgreSQL docs.

MongoDB:

 > use admin
switched to db admin
> db.runCommand({fsync:1,lock:1})
{
	"info" : "now locked against writes",
	"ok" : 1
}
// snapshot the DB here
> db.$cmd.sys.unlock.findOne();
{ "ok" : 1, "info" : "unlock requested" }

You can find the docs on this procedure on the MongoDB site.

I thought I would include the original article here even though it’s going on 8 years old!

OS Snapshots for Backup;
Utilizing operating system snapshots for quick and painless Oracle database backup and restore. from VOL. 16, No. 2 · MAY, 2002 of the NOCOUG Journal

Here are my slides from my presentation: