Category Archives: Database Engineering

MongoDB: Lagged Replica with Replica Sets

Posted on August 24, 2010 by kgorman

In an enterprise database architecture, it’s very common to create a standby or replica database with a ‘lag’ in it’s state relative to the primary. Operations applied to the primary are not seen on the replica for some amount of … Continue reading →

Posted in Data Architecture, Database Engineering, Mongodb | Tagged , | View Comments

Why Not Auto Increment in MongoDB

Posted on August 4, 2010 by kgorman

I came across this blog post with a nice pattern for auto-increment in MongoDB. It’s a great post, but there is something to think about beyond how to logically perform the operation; performance. The idea presented in the blog is … Continue reading →

Posted in Data Architecture, Database Engineering, Mongodb, Python | Tagged , , , , | View Comments

Data clustering in MongoDB using embedded docs

Posted on June 25, 2010 by kgorman

I wrote a while ago about how to cluster data to save cash. This post was geared towards relational stores. But in reality, the technique is applicable to any block store on disk. To recap, the premise is simple. When … Continue reading →

Posted in Data Architecture, Database Engineering, Mongodb | Tagged , , , | View Comments

Wayback Machine: snapshots still valid technique

Posted on May 10, 2010 by kgorman

I came across this old article I wrote for the NOCOUG newsletter in 2002 about using OS snapshots for backups. This technique is still very much a valid and widely used technique to perform backups. The idea is simple: – … Continue reading →

Posted in Database Engineering, Mongodb, Oracle, PostgreSQL | Tagged , , , | View Comments

MongoSF Slides

Posted on May 3, 2010 by kgorman

I had a great time at the MongoSF Conference on Friday. There were a ton of great presentations, and lots and lots of excitement. A big thanks to 10gen for inviting me to speak. I had a great time and … Continue reading →

Posted in Data Architecture, Database Engineering, Mongodb | Tagged , | View Comments

Dropping ACID

Posted on March 15, 2010 by kgorman

The de-facto durability story in MongoDB is essentially… there is none. Or at least single server durability. OMFG! No ACID WTF! &^%#^#?! For the next generation of internet scale, downtime intolerant systems, ACID may not be a desirable property. Traditional … Continue reading →

Posted in Data Architecture, Database Engineering, Mongodb, MySQL, Oracle | Tagged , , , , , , , , | View Comments

Hello Shutterfly

Posted on December 7, 2009 by kgorman

I am very excited to start a new position at Shutterfly.com. Shutterfly is a well known internet property with a massivly growing customer base and thus new and interesting challenges in how to store, share, and organize data. What fantastic … Continue reading →

Posted in Database Engineering | Tagged , | View Comments

pg_reorg 1.0.4

Posted on June 25, 2009 by kgorman

At Hi5, we currently use pg_reorg1.0.3 in order to organize data in a clustered fashion. I posted previously about the strategy. Our version is slightly modified, the modifications I made to the C code essentially allow pg_reorg to spin/wait for … Continue reading →

Posted in Database Engineering, PostgreSQL | Tagged , , | View Comments

Fusion-io SSD

Posted on May 27, 2009 by kgorman

I got the opportunity to test out some of the new Fusion-io Solid State ioDrive, and I thought I would post some results. Fusion-io has created a SSD product called ioDrive that is based on PCIe cards vs replacing SAS … Continue reading →

Posted in Database Engineering, PostgreSQL | Tagged , , , , | View Comments

Cluster data, save cash

Posted on February 13, 2009 by kgorman

Since the economy is not exactly rocking these days, I suspect there are a lot of companies out there trying to save a buck or two on infrastructure. Databases are not exactly cheap, so anything that an engineer or DBA … Continue reading →

Posted in Database Engineering, PostgreSQL, Python | View Comments