kennygorman.com

Starting with version 2.1, MongoDB has a new utility called Mongoperf. Mongoperf is a great little utility for quickly testing the I/O performance of your system. I thought I would go over it a little bit.

Mongoperf can be found in the bin directory of your 2.1+ MongoDB distribution. The utility is invoked with options to tell it how to perform a sample I/O run, and it gives output indicating the performance of your disk I/O subsystem. The utility generates random I/O over a single file. One great aspect of this utility is it accesses the I/O subsystem very much like MongoDB itself does. It uses the same memory mapped files interface just like MongoDB itself. Running the utility with –help outputs the various options for running. The utility takes a JSON document as input. Typically I like to store the various options in a .js file and emit that into stdin of the utility as such:

$>echo "{nThreads:2,w:true,r:false,mmf:true}" > perfsession1.js
$>
$>mongoperf < perfsession1.js

The most powerful usage of this tool is to bypass the Linux page cache, and perform Direct I/O. Running in this manner shows true gauge on how fast your storage subsystem is. This is done via the {mmf:false} assignment in the config document. When mmf is false, then Mongoperf performs direct I/O via the O_DIRECT flag. So no system memory is used to cache file buffers. That said, anything below the linux filesystem could be cached, including controller and drive caches. For example:

$>echo "{nThreads:2,w:true,r:false,mmf:false,fileSizeMB:10000}" > perfsession1.js
$>
$>mongoperf < perfsession1.js

The output of the utility shows the throughput as the threads ramp up to the nThreads limit you specify and then will just run at that level of concurrency forever. The output is in Disk Operations per Second. For example:

mongoperf < perfsession1.js
mongoperf
use -h for help
options:
{ nThreads: 2, w: true, r: false, mmf: true, fileSizeMB: 256 }
creating test file size:256MB ...
testing...
new thread, total running : 1
9922222 ops/sec 
11197085 ops/sec 
11130190 ops/sec 
11308340 ops/sec 
11068894 ops/sec 
11084711 ops/sec 
11447503 ops/sec 
11233556 ops/sec 
new thread, total running : 2
12925223 ops/sec 
12142161 ops/sec 
12290461 ops/sec 
12476609 ops/sec 
12392879 ops/sec 
11772720 ops/sec 
12308988 ops/sec

There are a couple things to note about this utility. For one, be sure to specify a reasonably large file for fileSizeMB. Maybe like 10000. This ensures you are wider than a single stripe if using RAID, this also ensures that you have enough data on disk that random I/O is truly random. Let the test run long enough that the system normalizes (caches get populated, etc). Try for 3-5 minutes or so as a general guideline. Also try different nThreads values to get a feel for the level of disk concurrency your system can tolerate. Also, another great test is to test the read and write options together, and separate to get a feel for how well your system does on each on of these types of workload. For instance, you can test the write cache on a controller quite effectively using this approach.

In the future I think having an option for creating multiple files to I/O against would be a great option and allow for a bit more realistic workload. Perhaps an option to output the results in BSON/JSON would be cool too.

What a madhouse!  The 2011 MongoSV conference was a blast, it didn’t hurt there were over 1000 attendees.  I really enjoyed some excellent talks, and good discussions with colleges and 10gen employees.  Plus, the opportunities in the evenings for a couple cold beverages were fantastic.

I have a couple take-aways from this years conference:

• The new aggregation features of MongoDB are going to be great, I am excited to start playing with them.
• 10gen is listening to us about the locking behavior of MongoDB, and the roadmap is promising.
• Lots of people use AWS for MongoDB.  More than I would have thought considering EBS is horrid.  Most everyone thought AWS and EBS was painful, but the pain was outweighed by the perceived benefits.
• Everyone agrees schema design is key.  The money quote from the conference was:

Schema free != Design free –@nathenharvey @nathanharvey 

The day before this years MongoSV Conference a few folks, chosen by 10gen, and labelled the Masters of MongoDB gathered to discuss the MongoDB platform, swap war stories, share gripes, and overall give feedback on the direction and roadmap of MongoDB to the 10gen folks.

First of all, I am incredibly honored to be considered in such a group of high caliber folks. But beyond that I was struck by the level of commitment that 10gen has to making sure the roadmap is proper and that they are listening very closely to the community. I feel like with such a commitment the areas for improvement in MongoDB will get good attention and focus. I think this is a huge component for consideration when looking at MongoDB. The community is very very strong.  I made lots of good contacts and was able to connect with friends.

Thanks 10gen for putting on such a good event, and above all, listening to your customers.

I had a great time presenting at MongoSV last year.  This year I am presenting a talk on MongoDB performance tuning, similar to what I presented at MongoSF but with some updated items around scalability, tuning, and hardware!  My college Luciano Resende is also speaking.  I hope to give some insight into some of the work we have done at Shutterfly around performance tuning and scalability.  I also hope to speak in more detail about our implementation of the Facebook Flashcache technology.  If you haven’t signed up, you can register here for the conference. Here is a video 10gen put out for the conference:

*changed the video to the updated version from 10gen