Egads, I wish I could sleep but how am I going to move the data? (Part 1)

by Bayard Carlin

First, a huge thanks to those involved

The move was a pretty complex project with a lot of planning, evaluation and testing. We did a lot of rehearsing before the actual cutover. Many thanks to the folks in our engineering department who contributed to the project. We also also want to thank the folks at Percona as well as Tyler Poland at Engine Yard. They played a big part in our success.

The challenge

One of the most difficult problems to solve was how to move several terabytes of hot data across the country without significant downtime or gaps in data collection. This was keeping me up at night. The official company target was a 6 hour maintenance window. I wanted to be collecting customer data within one hour. We almost made it–I think the actual Collector outage was about one hour and 10 minutes. It took us about 2 hours to get the UI up.

We had two viable options to move the data across: move customers programmatically a batch at a time or use MySQL replication. Some of us were a bit skeptical that we could replicate across the Internet because of the amount of rows we write (1.4 billion a day). Moving the customer data programmatically was pretty complex and fraught with its own issues.

I am an Ops guy, I understand replication, I have used it before and I trust it. Replication would be cleaner as we could do a knife-switch cutover, so it was my preferred method.

The general steps for replication are: Get an image of the database, start a mysql instance from that image and enable replication. Most times mysqldump/restore is used to get the image.

A bit about the New Relic database architecture

New Relic uses MySQL 5.0. We have sharded database architecture for our metric data and a central database for all the account data. Customers are mapped to specific shards, which are around 500 GB each. As mentioned above, we insert around 1.4 billion rows a day across all our shards. We have a multi-tenant application and each customer gets their own set of tables. We have *lots* of tables, at least 120,000 per shard.

The tables that contain data for the last hour are getting written to. These tables are extremely hot and tend to be large. Every hour we roll them over, create a new set and drop the tables that are no longer necessary.  The tables containing historical data are static unless we are processing them.  Every night we also purge tables that have expired.

These factors make it very difficult to use normal methodologies to back up and restore the databases. We delete tables every hour and mysqldump takes greater than an hour to run, plus mysqldump will fail if tables are dropped during a backup.

Another issue that prevents us from doing a normal dump is that the metric tables that are getting written to are way too hot to lock. I tried Xtrabackup but it failed because we were dropping tables. We have a way to back up our shards but that is out of scope for this posting. Essentially we don’t back up the hot tables and have a wrapper that keeps mysqldump from borking when tables are dropped. Our backups unfortunately are not suitable for replication.

Replicating the accounts DB was a no brainer. I used Xtrabackup to dump the DB, copied the file across and executed xtrabackup with the –prepare option. The Xtrabackup output gave me the correct binlog and log position and I was off to the races. Xtrabackup is orders of magnitude faster than mysqldump. You can find information on Xtrabackup here.

Proof of concept

Our first step was a proof of concept to see if replication was actually possible. We decided to test things out with one shard.

In most cases mysqldump/restore is used for replication but as explained above this would not work for us. In the end we decided that LVM snapshots were the best option and we would rsync the data across to the new data center. If you do a snapshot of a running database you need do a flush tables with read lock before you snap or you have to dump/restore prior to replication. For a variety of reasons the flush tables may have been problematic with our app and we didn’t want to risk it. I also didn’t want to take the time for a dump and restore so I stopped the DBs prior to the snapshot. Having a sharded architecture made it possible for us to do this without affecting all our customers at once. The outages were generally sub 5 minutes per shard, the first snaps (ahem) took longer because we hit some snags.

Here is an outline of the steps we used. This is not meant to be a complete tutorial on replication. I would strongly recommend reading up on replication or use a consulting service like Percona to help you out. Having a data center migration go sideways would not be career enhancing. Docs on replication can be found here.

Some prereqs

Set up ssh keys on the master and replica hosts. This is necessary for passwordless rsync and since we are replicating over the Internet we need an ssh tunnel pass data back and forth securely.

Make sure the MySQL root passwords match on the master and slave (this is done for you if you rsync the entire MySQL directory across)

Edit the my.cnf on the master to enable binlogging and set a unique server ID. If binlogging is already enabled and you are not replicating to another DB you will want to delete all the binlogs after shutting mysql down and prior to doing the snapshot.

We do not use localhost or bind our mysql instances to the server’s IP address, we have a unique IP per instance. The first thing we needed to do is add the correct permissions for root so the remote DB can be administered.

On the master in the mysql client
#grant all on *.* to 'root'@'<hostname_of_replica>' identified by '<somepassword>' with grant option; flush privileges;

Then stop mysql
#/etc/init.d/mysql stop

Do the snap
#lvcreate -L <size> -s /dev/<original-device> -n <new-device>

Mount it
# mount -o nouuid,ro /dev/<new-device> /mnt/backup

Check the snapshot. You will want to have an idea of whether the correct files are there so you need to do a compare. Since MySQL is down we wanted to make it quick.
#rsync -avz --dry-run <source_directory> /mnt/backup

If there is no output you are golden.

Start mysql again
#/etc/init.d/mysql start

Now do the rsync. We used screen as the rsyncs took a long time. We also wanted to have notifications when the syncs were finished. I also wanted to look at how things were going so I logged the screen output.
# screen -S rsync -L
#rsync -avz s/mnt/backup <destination_host>:<destination_directory> && echo "Snapshot copy done" | mail -s "`hostname` snapshot copy is done" <your_email@your_domain.com>

If bandwidth or disk IO is an issue you can use the --bwlimit flag.

Now it is time to eat dinner, go to a movie and get a good night’s sleep. With 500 GB and 120,000 files over the Internet this is going to take awhile.

Next we want to allow the slave to replicate from the master. On the master in the mysql client:
#grant replication client, replication slave on *.* to <'replication_user>'@'<hostname_of_replica>' identified by '<replication_password>';

When the rsync is finished log into the slave and execute:
/etc/init.d/mysql start && tail -f <path_to_logs>/mysql.err

Assuming there are no errors in the log you are ready to start replication. Set up the bidirectional SSH tunnel. We set up the tunnel on the remote server and decides to use port 13306 on both ends. It is a good thing to script this into a loop
ssh -N -v -L13306:127.0.0.1:3306 -R13306:127.0.0.1:3306 <remote_host>

On the slave in the MySQL client: (You need to use the same <replication_user> and <replication_password> as you did with the master.)
#change master to master_host=<hostname_of_master>', master_port=13306, master_user='<replication_user>', master_password='<replication_password>', master_log_file='<name_of_first_binlog>', master_log_pos= <correct_log_position>;
#start slave;

Also, you will want to make sure nothing is writing data to the slave or replication will get borked. Yes, I speak from experience.
#set global read_only = on;

Now you can see how replication is doing:
#show slave status \G;

For an exact explanation of the output please see MySQL documentation here.

The initial test went well. It took about 25 hours to sync all the data across. Having a large number of small files did not help us any. I brought up the new instance and enabled replication without issue. I think it took about 8 hours for all the changes to catch up.

Rinse, lather, repeat

We were very encouraged with the results. We kept replication going for a couple of weeks without issue. Replication for the most part kept current and when it fell behind it caught up rather quickly. Once we proved this method would work we set it up on the rest of our shards. We did 2 a night. We kept replication running for a week or so prior to the cutover to make sure everything was stable.

Nic Benders one of our Dev/Ops guys cooked up a script that would check replication status across a number of databases. A slightly modified (to protect the innocent) version is at the bottom of the post. We later wrapped this into a ruby script that sent alerts if seconds > 0 for a couple of iterations and put it into a cron job. We wanted to know about replication errors as soon as possible because redoing everything would slip the date.

The steps for the cutover were all scripted but the process looked like this::

Shut down the app.

Make sure replication was up to date on all the shards:
# show slave status \G

Shut down the master.
#/etc/init.d/mysql stop

On the accounts database and each of the slaves:

Turn off replication.
# stop slave;

Turn off mysql and do a snapshot just in case we need to fail back. If we have an image we can start replication right where it left off. Karsten Self another Dev/Ops guy scripted this with a bunch of trimmings as we had some unique issues we needed to solve.
/etc/init.d/mysql stop
#lvcreate -L <size> -s /dev/<original-device> -n <new-device>
# mount -o nouuid,ro /dev/<new-device> /mnt/backup
#rsync -avz --dry-run <source_directory> /mnt/backup
.
Make the DB read/write.
# set global read_only = off;

Make the DB it’s own master.
# change master to master_host=' ',

Tighten up security a bit by removing the permissions for the old master:
#revoke all on *.* from 'root'@'<hostname_of_old_master>'; flush privileges;

Now the data was production ready on all the new servers.

Be sure to stay tuned for the next exciting episode where we discuss creating the playbook and the actual execution of the switchover.

Voila!

The next steps were bringing up the different tiers of the application.

All in all I was delighted with the results.  The cutover was smooth easy and efficient.

Nic’s check replication across multiple hosts script:
#!/bin/bash

for db in A B C; do
case "$db" in
"A")
hostname="<hostname A>"
;;
"B")
hostname="<hostname B>"
;;
"C")
hostname="<hostname C >"
;;
esac
printf "%-16s " $hostname
status=''
status=`mysql -h $hostname -u <replication_user> --password='<replication_password>' mysql -e 'show slave status' -E 2>/dev/null | grep Seconds_Behind_Master | awk '{sub(/^[ \t]+/, "")};1'`
if [ "$?" == "0" ]; then
echo -en "$status"
seconds=0
seconds=`echo $status | awk '{print $2}'`
if [ -z $seconds ]; then
echo
elif [ "$seconds" == "NULL" ]; then
echo -e "\t--:-- (hh:mm)"
elif [ $seconds -lt 60 ]; then
echo -e "\tCURRENT"
elif [ $seconds -gt 0 ]; then
minutes=$(($seconds / 60))
hours=$(($seconds / 3600))
minutes=$(($minutes - $(($hours * 60))))
printf "\t%d:%02d (hh:mm)\n" $hours $minutes
else
echo -e "\t???"
fi
else
echo
fi

echo
done

PHP Fog provides simple one-click installations of some of the most popular PHP applications and frameworks, including Drupal, WordPress, Zend and more. PHP Fog handles deployment, failover, database maintenance, scaling, and all the other plumbing that can take an army of programmers and systems administrators to handle. For organizations deploying apps on PHP Fog’s robust platform, RPM is a valuable enhancement that can help them better ensure the success of their online apps.

PHP Fog founder Lucas Carlson said today “More than ever before, organizations are building and deploying PHP web applications that deliver essential online services and must meet very high standards for performance and availability. As these companies turn to PHP Fog for reliable and scalable hosting services, the addition of RPM’s monitoring and troubleshooting capabilities offers unbeatable advantages for ensuring their applications are meeting business and performance goals.”

PHP Fog customers, start using RPM today!
For organizations deploying applications on PHP Fog’s platform, taking advantage of free New Relic RPM Bronze is simple. Customers can simply log onto phpfog.com, deploy any PHP application and a New Relic account will be automatically created for them. The implementation process takes about 15 seconds. Customers are encouraged to contact PHP Fog through the live chat or support desk for additional details.

The essence of Apdex is in measuring customer happiness. How frequently are my customers happy with the performance of my application? How frequently are they tolerating slow performance and how often are they frustrated? These levels: satisfied, tolerating and frustrated, are derived from a performance goal that you set. To learn more, check out How Apdex Reveals User Frustration with Your Web Experience.

So what should my Apdex goal be?

Your Apdex goal represents the ideal response time for your application. When customers are using your application, they don’t care what your technology stack is. They don’t care what language you write code in, and they certainly don’t care which database you use. What matters is perceived value per transaction. Broadly speaking, the leader board for a mobile game needs to be much faster than submitting an order for a flying car online. Ideally, your application is among the top performers in your category.

Setting Apdex too low

It can be tempting to want to put a low Apdex goal in place, and work for several releases to speed up the app to see if you can make your goal. If you found out that a competitor had really low response times, for example, it might be tempting to use Apdex as a way to benchmark against the goal of catching up. The problem with setting Apdex too low in comparison to your current average response time is that the Apdex breakdown loses a lot of value in the mean time.

Let’s say, for example, your average response time was 1,000ms. If you set your Apdex goal to be 500ms (0.5s), you’ll get an overwhelming majority of requests in the bottom tiers and it won’t help you see performance outliers.

Setting Apdex too high

Following our earlier example, let’s say we set our Apdex goal to 2.0 seconds with an average response time of 1 second. With this configuration, we would see an artificially high and consistent Apdex score. Setting Apdex too high or too low can both result in hiding your performance outliers.

The ideal Apdex score for your app is…

If you set your Apdex goal to match your average response time, you’ll get a lot more value out of it. The benefit of Apdex in practice is that it highlights discrepancies in response time that are otherwise hidden by an average.

1. Set your Apdex goal near your average response time

As an example, this customer has an average response time of 140ms and has set their Apdex goal to 200ms (0.2s).

2. Focus on application performance, as indicated by customer happiness

By setting a relevant and achievable goal, this customer immediately sees fluctuations in customer satisfaction that are otherwise hard to spot. If customers experience an increasing number of slower than ideal responses, you’ll see this reflected immediately in the Apdex chart, while the average response time may not change much.

3. Lower your Apdex score to your new, faster response time average

As you improve the performance of your application, lower your Apdex score so it stays relevant.

4. Profit!

While we’ve heard many customer stories of how increasing application performance increases sales, we’ve yet to hear the opposite. Feel free to enjoy those additional profits while your customers enjoy your improved performance! It’s on us!

This is certainly true in the case of PHP, which first appeared in 1995. The creation of pioneering young developer Rasmus Lerdorf, it leaned heavily on syntax from C, Java and Perl. While his larger vision was to produce dynamic web pages quicker and easier, the initial intent was simply to track hits to his online resume. He dubbed the original exploration “Personal Home Page Tools,” then expanded its functionality to include database interaction and some application development capabilities and released it for public use. By 1997, PHP/FI was being used by about 50,000 domains – around 1% of total internet properties at that time. And that’s when the popularity of this new development language outpost really exploded.

Zeev Suraski and Andi Gutmans came along and completely rewrote the language to better suit the e-commerce app they were building and in 1998 they created PHP 3.0, the closest predecessor to the modern version. Word spread quickly, and it grew to encompass installations on hundreds of thousands of websites and up to 10% of web servers. Gutmans and Suraski decided to rewrite PHP’s core entirely and their creation became known as the Zend Engine, and now, essentially, there was new sheriff in town. PHP 4.0, built on Zend and released in 2000, is still the most commonly used version of the language. PHP 5.0 came out in 2004 with features that give developers shorter, cleaner, more flexible code to work with.

PHP is now used on 75% of all web servers and is the foundation of some of the best-known apps and frameworks, including Facebook, WordPress, Wikipedia, Drupal and Digg . And that’s why New Relic got into the PHP app management business. We had heard “I wish there was New Relic for PHP” enough times to realize that 1) our customers and their peers are building more business-critical PHP-based apps than ever before and 2) there was real and growing need for 24×7 monitoring and troubleshooting.

Social gaming apps are a perfect example of trend. PHP is used extensively by Facebook and iPhone game developers. The success of a game is unpredictable: it can be a flop or an instant hit. What is predictable is that if the game takes off and you’re not prepared to scale rapidly, it’s a good idea to have monitoring in place so you can diagnose issues at the earliest moment and take immediate corrective measures.

Many of the leading PHP platform providers are offering application management to their customers as an added value. As we mentioned in a previous post, this strategy lets the vendor work more collaboratively with customers as they share visibility into apps deployed on their platform, and gives a new level of control to the customer who is ultimately responsible for application performance. New Relic is teaming with several top platform providers to integrate RPM into their solutions. Here’s just a short list of our PHP partners:

• Acquia enables organizations to build and deploy social publishing websites quickly and easily by leveraging the Drupal platform.
• cloudControl is a leading PHP cloud hosting platform provider based in Potsdam with a managed solution that eliminates the administrative hassles.
• Joyent provides cloud computing software and service designed specifically to enable Service Providers and Enterprises to deliver cloud services to clients or within enterprises.
• Morphlabs provides a cloud computing platform that converts commodity hardware into a cloud environment, allowing companies to easily create their own private cloud.
• Network Redux develops, deploys and manages custom cloud-computing solutions and web-hosting environments, including virtual servers and complex enterprise platforms.
• PHP Fog offers platforms for PHP-based applications – like WordPress, Drupal, Shopify, Zend Framework and more – that help companies take full advantage of cloud computing.

We will continue working with new partners to ensure on-demand APM is soon accessible to PHP developers. Visit us here to learn more about New Relic RPM for PHP.

The Engine Yard platform-as-a-service offerings provide a curated and optimized Ruby on Rails stack, a set of best-of-breed deployment and management tools, and access to Engine Yard support and professional services. New Relic RPM is the leading on-demand application performance management service for Ruby on Rails. It allows teams to monitor applications in production, proactively identify performance issues, and drill down for root-cause diagnosis.

Matt Greer, CEO of Armadealo, a leading provider of location-based marketing services, talked about the combined advantages of the Engine yard and New Relic solutions. “Our customers are shoppers who want instant access to discount and coupon information on their iPhones, so we need to deliver quick service and 100% uptime. We rely on Engine Yard because they are experts in deploying Ruby on Rails applications, and we have confidence that we can scale quickly to respond to large traffic spikes, up to 300%, on weekends and holidays. We chose New Relic because we can optimize our application response times during development and then monitor performance after we deploy to make sure we’re providing the best user experience. Using the combination of Engine Yard AppCloud and New Relic RPM frees me to focus on growing my business and serving customers.”

Tom Mornini, Engine Yard co-founder and CIO said today that “Over 1,800 Engine Yard customers have chosen to build their mission-critical applications in Ruby on Rails and deploy to the cloud. As fast-growing web companies and enterprises continue to build innovative products and services, they want to provide maximum uptime and the best user experience for their customers. New Relic and Engine Yard are delivering an enterprise-class platform and service that allows companies to easily deploy their apps and get a critical view into production application performance.”

Engine Yard customers get started with RPM today!

New Relic RPM installs in about 2 minutes and is easy to get up and running. Engine Yard AppCloud and xCloud customers can easily start using RPM Bronze by contacting their Engine Yard sales representative for details. Additionally, RPM Silver and Gold subscriptions are available to Engine Yard customers at special discounted pricing. For more information about using New Relic RPM on Engine Yard, please visit http://pages.engineyard.com/NewRelicOffer.html.

New Availability Monitoring Features

Availability monitoring has several new features. In configuring Ping URLs, you can now specify a substring to match in the response content. If the string is not found, a failure is recorded. You can also disable the SSL certificate validation for a site if you are using a self signed certificate or a non-host matching certificate.

REST API now uses an API key

Our REST API now uses a generated API key for authorization. Previously, the account license key was used. For accounts which are currently using their license key for API access, the API key will be set initially to the license key so your applications will continue to work without modification. However, for extra security you can now go in and generate a new API key to use, revoking the use of the license key in API calls.

You can now temporarily disable pinging on your site using REST calls to disable and re-enable pinging during maintenance windows. Details of the API are specified on the Availability Monitoring configuration page.

REST API on Github

Our REST API is now fully documented with Active Resource examples in the New Relic API gem on github.

Unsubscribe links on weekly email reports and incident emails

We have simplified your ability to unsubscribe from certain emails from New Relic by adding unsubscribe links at the bottom of several types of email messages.

Three years ago when we started the company, we felt that broad adoption of Platform as a Service offerings would include services beyond just a basic platform for customers deploying business-critical apps. We believed the Platform as a Service vendors would start adding value higher and higher in the stack. First a strong platform, then easy deployment, then tools to make the deployed apps the most successful.  This is proving more true than even we believed it would be.

Flashback warning: You will now be transported back in time to the era before cloud computing. Please do not adjust your computer screen.

Remember when the “Cloud” was just beginning to take off?  Seems like ancient history now…the earth’s crust was cooling and dinosaurs were roaming about. Back then, in reality just a couple of years ago, there were far fewer cloud platform vendors in the market then than there are today, with most of them supporting Ruby on Rails. At that time, we began working with folks like Heroku and Engine Yard, organizations that were pioneering the new Cloud Platform concept. They quickly expressed a desire to provide their customers with real-time app monitoring. They saw the fundamental utility of allowing their support teams to share visibility inside the customer apps running on their platform when potential performance-related issues popped up. This way,  support staff and the customer could efficiently troubleshoot the app together. Not only that, but by giving their customers production monitoring capabilities, visibility and control for apps deployed on their platform transferred back to the rightful owner: the customer.

This concludes the flashback portion of this post. Please remain calm and continue reading.

Fast forward to the present–we are working with most of today’s leading PaaS (and several IaaS) providers, many of whom are standardizing on New Relic as their monitoring tool of choice for their support staff and for their customers. Most are building New Relic RPM into their platform as a critical feature, and providing it for free for all their customers. The list of New Relic partners reads like a who’s who of Platform as a Service companies and includes Blue Box Group, Brightbox, Engine Yard, Eucalyptus, GigaSpaces, GoGrid, GridGain, Heroku, Joyent, Morphlabs, Network Redux, PHP Fog, RightScale, SpeedyRails, Stax Networks (Cloudbees), and Webbynode (And we have forged relationships with IaaS providers as well including Amazon AWS and MSFT Windows Azure, Rackspace, and VMware).

Benefits of of adding app management to cloud platforms

Real-time visibility into apps deployed on these platforms has several advantages:

• Reduces support calls by empowering customers with application management capabilities
 to understand and tune their own app.
• When support calls do occur, it enables support staff with shared realtime visibility into the customer’s app
, making their support calls fast, efficient, clear and effective,  thus driving down the cost of supporting the customer.
• Provides visibility that proves the performance of the Platform, and helps the customer optimize their app to make best use of the Platform Resources, which we all know often have different requirements than “in-house” hardware.
• Helps drive overall customer application success, making the customer a happy camper, which in turn, drives PaaS provider’s success as customer’s traffic increases and they increase deployments.
• Adds functionality to the Platform, enabling the PaaS provider to continue to add features higher and higher in the stack, without necessarily hiring a team of people to learn the complicated Application Management space.

As the sun continues to rise on the Cloud (wait, what?), we all believe that more and more organizations will migrate their business-critical apps to the Cloud.  The cloud and the platforms on which companies are deploying their apps, provide tremendous advantages. Adding shared insight into the health and availability of these apps gives PaaS providers and their customers a common language for collaboratively managing and maintaining successful apps.  In the immortal words of Microsoft (and we can’t believe we’re saying this): “To The Cloud!”

Anti-Pattern: Monitoring a subset of hosts

We occasionally work with customers who are monitoring a subset of hosts in production, rather than all of their production hosts. How do we know they’re not monitoring all of their hosts? Several specific issues tend to crop up when you only manage parts of your application.

Missing Transactions

Murphy’s law says that critical business transactions are bound to happen on the unmanaged hosts. Errors, slow requests and poorly performing queries are all invisible when they happen on unmanaged hosts. By monitoring all hosts, you’ll never miss another poorly performing transaction.

Seeing only parts of the whole picture

By only monitoring a subset of hosts, virtually all of the metrics you see in New Relic would be skewed. Instead of seeing the throughput of your entire application, you see the throughput of activity on just a few hosts. Even with “fair” load balancing across a number of hosts, distribution is not typically equal. We consistently see a level of variability in response time and throughout across hosts supporting a given application.

Scalability and Optimization features work best with a complete picture

Our Gold level of service includes capacity and scalability reporting for optimizing application performance. These features are incredibly valuable in helping you scale an application over time and proactively manage performance. These features work best, however, with a complete picture of your application’s performance. It’s very difficult to extrapolate capacity analysis, for example, with only a limited amount of information.

Gathering performance metrics in pre-production environments

We believe managing performance in production is essential. Real performance data comes from real customers using your application every day. For highly performance-sensitive environments with very large amounts of traffic, we often see customers also deploying to QA and or load testing environments before releasing to production. In these environments they may run synthetic load tests, specific stress tests or other forms of quality assurance that aim to find the “low hanging fruit” for functional and performance related issues.

We recommend that customers include the name of the non-production environment in the application name, so they are able to track the performance of these environments separately. For example, for a production application named “My Application”, you might configure the application name in your load testing environment to “My Application (load testing)”.

Spread the love of New Relic everywhere!

Having performance management available across your entire application is the best way to mitigate application problems while optimizing for long-term scalability. Sign up for New Relic today or upgrade your existing service to cover your entire environment. You’ll be glad you did!

Does it really matter which transaction is the “slowest”?

Maybe not. Every app tends to have a few administrative pages that are usually reserved for internal use. These tend to be reporting type pages that may take quite a long time to process. While these pages might be the slowest, they are not the most impactful to your customers.

Which transaction is the slowest for the majority of users?

Our Web Transaction Analysis feature displays transactions sorted by the most time consuming by default. A transaction’s time consumption is measured by the product of its response time and the number of times it is called within a period of time.

Here we see that this app spends 14% of its time in the Admin::OrdersController#index transaction. This may not be the most frequently called transaction, nor the slowest transaction. By considering both factors, however, we can see that improving its performance would be the best place to start for this app.

Your web transactions can also be sorted in other ways. We currently offer sorting by “Slowest Average Response Time”, “Apdex most dissatisfying”, “Highest Throughput” and “Standard Deviation”.

Transaction Details

For any web transaction, we get an overview and detailed breakdown of how that transaction performs.

We see that his transaction has a reasonable Apdex score of 0.77, with an average response time of 293 ms.

We can also see how this transaction has performed within the current time window (set to the last 30 minutes). While throughput stayed relatively consistent, response time shows nearly a tripling from fastest to slowest. This is reflected in the Apdex score, which dips into the “poor” region briefly.

As we continue down the page, we see a breakdown of performance for this transaction type. We can quickly see how much time is being spent in each component or service used in this transaction. In this case, 61% of our time is being spent processing the Rails controller action itself, while around 30% of the time is spent in the database.

Review

With New Relic, it’s incredibly easy to find performance bottlenecks in your web applications. We’ll guide you right to the slowest parts of your app, so you can improve response time making every customer a happier customer.

The fallacies are:

1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn’t change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

TL;DR

The Fallacies of Distributed Computing applies directly to web applications, and is especially important when developing applications hosted in the cloud. Read on to see just how relevant they are today.

What is a Distributed System?

A collection of independent computers that appears to its users as a single coherent system
  - Tanenbaum and Steen: Distributed Systems: Principles and Paradigms

You know you have a distributed system when the crash of a computer you’ve never
heard of stops you from getting any work done.
  - Leslie Lamport : Security Engineering: A Guide to Building Dependable Distributed Systems

A distributed system is an application that executes a collection of protocols to coordinate the actions of multiple processes on a network, such that all components cooperate together to perform a single or small set of related tasks.
  - Introduction to Distributed System Design – Google Code University

Web Apps Are Distributed Systems

Web applications are composed of multiple systems working closely together. Proxy servers, cache servers, load balancers, web servers, application servers and databases are all likely to be involved in processing a single web request. Many web applications also leverage other web-APIs. An application may show your friends on Facebook, allow you to post to Twitter, or store some files on Amazon’s S3 storage service.

Web Apps in The Cloud

Ignore the fallacies at your own peril. For a little while, you might pretend that some of the fallacies don’t apply to you when you host applications in datacenters with your own hardware. Hosting applications in the cloud provides no such grace period. Failure will happen. Hosting in the cloud, while being the easiest way to scale a web application, means facing up to these fallacies immediately.

When Netflix moved to the cloud, they were quick to realize that they could not ignore the fallacies:

“I knew to expect higher rates of individual instance failure in AWS, but I hadn’t thought through some of these sorts of implications.”

“AWS networking has more variable latency. We’ve had to be much more structured about “over the wire” interactions, even as we’ve transitioned to a more highly distributed architecture.”

“Co-tenancy can introduce variance in throughput at any level of the stack. You’ve got to either be willing to abandon any specific subtask, or manage your resources within AWS to avoid co-tenancy where you must.”

“We’re designing each distributed system to expect and tolerate failure from other systems on which it depends.”

   – John Ciancutti, Vice President of Personalization Technology at Netflix, Inc

The Cloud vs. The Fallacies

The Fallacies of Distributed Computing are even more important to consider when building web applications in the cloud. While it seems obvious that the cloud highlights these fallacies more than ever (as Netflix noted recently), not everyone agrees. Tim Bray is currently an Android Developer Advocate at Google, former Director of Web Technologies at Sun and editor of the W3C XML specification. I was surprised to find a post from 2009, “The Web vs. the Fallacies”, where Tim suggests that the Fallacies are much less important to systems built on the web. I was even more surprised that he still agrees with that assertion when I asked him. Let’s see if I can change Tim’s mind.

Disputing The Fallacies: The Cloud Era

1. The network is reliable

When you consume web APIs from Facebook, Twitter or Amazon, “the network” is the internet. How reliable, within a given performance window, is the aggregate of all the web services you interact with? Tim Bray suggests “if a GET gets a network blowup, just do it again”. Sure, we all do that. If a page loads really slowly and appears to hang, we just refresh. Do you do that with each of your services and API calls?

The network is not reliable. When you add up all the points of failure for all the various services and APIs you use, the odds of failure are not just high, they are a given. Instead, build an app that can function at reduced capacity when a given service is offline.

2. Latency is zero

Try as we might, there just isn’t a way to improve the speed of light. Serving customers half-way around the globe affects performance in big ways. Moving services closer to your customers via cloud availability zones and content delivery networks can help tremendously. Ignore latency online and you’re likely throwing away customers from half the globe.

3. Bandwidth is infinite

Not long ago everyone had high-speed internet access at home. (Not everyone, of course. The digital divide is real.) Wireless hotspots filled every home and apartment and the web felt fast. A few years ago this started to change. Starting with the iPhone, web applications were being used not just over high-speed internet connections, but over phone networks. Using web apps on mobile devices and tablets reminds us that bandwidth problems did not disappear with the dial-up modem. Stay mindful of how much data you’re shipping across the wire and all your users will be grateful. For a great read on how The Fallacies apply to mobile development, check out this great series of posts by our friends at Carbon Five on iPhone Distributed Computing Fallacies.

4. The network is secure

Network security can be a full-time job for a prominent web business. Transport layer security is great, except when we leak security tokens over insecure connections. Firesheep brought some necessary attention last year to the prevalence of private security tokens littering insecure connections, ripe for snooping by the guy in the corner of your favorite coffee shop sipping chai. The network is not secure, and the more we move our lives online the higher the risk of exposure. Keep a security mindset when developing your apps and keep your users safe.

5. Topology doesn’t change

One of the biggest benefits of moving applications to the cloud is the ability to change topology at will. Add new app servers. Upgrade (and possibly change the architecture of) the CPU on your database server in the middle of the day. Add reverse proxy servers, cache servers, change CDN providers, migrate availability zones. Topology changes all the time. Depending on a static infrastructure design not only limits your ability to respond to change, it increases the likelihood of site-wide outages.

6. There is one administrator

Of course there isn’t just one administrator. Even with applications hosted in your own private datacenter, your applications are likely interacting with systems outside your administrative control. These systems may have performance, availability or security issues that you have no direct influence over. Staying mindful that these systems are beyond your control can help you ensure they have minimal impact on your services when they are unresponsive.

7. Transport cost is zero

Not only is transport cost not zero, it’s priced like any other commodity and can be purchased per transaction, per gigabyte, per compute hour, etc. Cloud storage costs (and transport of that storage) are a major component of application hosting costs, just ask anyone doing video streaming online how close to zero their transport costs are.

8. The network is homogeneous

This fallacy was added to the original seven by James Gosling, creator of Java, in 1997. The best part of this fallacy is that it might actually be true. Almost. Between REST and JSON APIs, have you ever had to think about the implementation of an external service, beyond satisfying your curiosity? You’re much more likely to see libraries producing invalid messages than you are operating systems playing a role.

Building performant distributed web applications is hard

The Fallacies were coined 17 years ago, but they apply just as well today. Back then folks might have been fooled into actually believing them, but we’re smarter than that, right? Knowing what gotchas await our web applications, most notably those hosted in the cloud, can help us build better, faster and more performant applications. Isn’t that what we all want?