New Relic Blog

How do you move more than 6000 customers in 2 hours? (Hint: plan ahead)

As many of you know, we undertook a major data center move last November. The move went quite smoothly, and we thought a blog post or two about how we did it might be useful to the community. We cajoled our IT Director Bayard Carlin into writing two posts that discuss the planning and execution of our effort to move live production performance data for 6000+ customers. This post covers the efforts that went into planning the move and the second post will detail events as they unfolded the night of the move. Without further ado…

Egads, I wish I could sleep but how am I going to move the data? (Part 1)

by Bayard Carlin

First, a huge thanks to those involved

The move was a pretty complex project with a lot of planning, evaluation and testing. We did a lot of rehearsing before the actual cutover. Many thanks to the folks in our engineering department who contributed to the project. We also also want to thank the folks at Percona as well as Tyler Poland at Engine Yard. They played a big part in our success.

The challenge

One of the most difficult problems to solve was how to move several terabytes of hot data across the country without significant downtime or gaps in data collection. This was keeping me up at night. The official company target was a 6 hour maintenance window. I wanted to be collecting customer data within one hour. We almost made it–I think the actual Collector outage was about one hour and 10 minutes. It took us about 2 hours to get the UI up. After the jump, learn how Bayard and team planned the move of over 6000 customer accounts to a new datacenter