To avoid a single point of failure you needed to have redundancy.

There are several power lines between a power station and a customer. When an overload is detected by a circuit breaker, power can be rerouted along another line. Each individual line is normally run at reduced capacity so it can handle the increased load.

Redundancy, however, introduces problems. Business applications have state. One electron is as good as another and can be easily rerouted; application state is tied to particular users, sessions, or transactions. I need to access parts of my current order. If the network connection I am on gets broken, substituting someone else's order does not work. Parallel processing of mathematical algorithms (such as MapReduce, or multi-threaded versions of Linq) cannot solve this problem.

You need to remember information about your users, or what products are on order. Some of this data changes rarely if at all; some of this data changes dynamically. Some of this data is very valuable, some of it is not. In a non-cloud application you might use data base mirroring. But you pay a performance penalty for a database mirror because the transaction must commit on both the original and the mirror. Imagine the cost (latency and throughput) of trying to keep redundant databases in different geographic areas in sync. Hence, you must deal with state explicitly.

You have to reduce the parts of your application that handle state to a minimum. Loss of a component means the loss of application state that it holds.

This is classic advice for scaling systems, but is more critical in the cloud. In the past we created stateless middle tier components for easier scalability, and relied on the data tier to handle scale. You then rely on clusters or technology built into the database (transaction logs, etc.). While stateless components are still an excellent idea, we cannot always rely on the database to scale, and the possibility of database failure is a large risk.

The business layer and the domain layer should be stateless, and to the greatest extent possible the clients should hold whatever state they can. Stateless services can be added or removed easily to handle changes in demand or the failure of a service instance, since you have decoupled functionality. Try to build services that are as atomic as possible because that makes it easier to scale or recover from failure by using redundancy. Here, atomic does not mean small.

Suppose you put customer information in the same service as your catalog service. If the customer information service goes down, so does the catalog. You should be able to check in people to your flight even if you cannot give the prices for tomorrow's flights.

There are other techniques you can use. For example, if the bill paying service is not available, you can usually use a transactional queue, and just retry the payment until the service is available. That is one of the reasons why banks, for example, say that they need 24 hours for an electronic payment that could be processed immediately. The odds that the payment service will be down for 24 hours are miniscule.

Also don't forget infrastructure pieces as to where logs are stored. You should probably back them up frequently.

So what about where state has to be stored, say in the data tier? This is a complicated problem which I have discussed this before in both a blog post and a presentation . It basically boils down to the question: what is the acceptable level of data loss? As it turns out, in many applications you do not have to be absolutely consistent in all places, and you can relax consistency constraints to get scalability and reliability. Now you may think this is ridiculous, data can never be lost. But think about how business is actually done.

Airline reservation systems separate out the flight query database, from the transactional database where flights are booked. As a result, occasionally a flight or a price you thought was available is not. But if they did not do that, the performance of making reservations would be very poor. More business would be lost under strict consistency than under relaxed consistency constraints. Why does Amazon use an email system for notifying you of your book order? To scale the user interface and to avoid performance problems due to abandoned shopping carts. You have to ask the question: What is the cost of an apology for the data loss?

What kind of availability are your customers actually willing to pay for, as opposed to what they say they want? Given a choice, do they really want absolute consistency in all cases? If you used replication over a short period of time to another data center, and a data center was destroyed by a hurricane, and a certain amount of data was lost, how terrible would that be?

Consider grouping your components in units of failure. If you have components of type α, β, and γ, you could put all α on a single host, all β on a separate host, or all γ on a separate host. Or you could put groups of α, β, γ on separate hosts. The latter would cause a complete failure on a loss assuming affinity to a given machine. The former would only result in a failure of α functionality.

What do you do if your components fail? Do you have to reroute traffic to another data store, or data center? Do you need to add more instances of a component? You have to monitor your components and understand why they fail. Under certain circumstances, you can degrade performance. Caching might help in keeping your application running under failure.

Degrade gracefully and predictably. Know what you can live without.

Designing for Failure has been around a lot longer than cloud computing. As we have discussed in several other blog posts, cloud computing, as opposed to hosting in the cloud, is about the ability to acquire or release computing resources as necessary. Acquiring more resources allows you to keep up with demand, or to compensate for a failed instance of a resource.

You must examine every source of dependency in your application: third party libraries, hardware, software interfaces between parts of your own application, TCPIP ports, DNS servers, message queues, database drivers, database size, latencies, to name just a few. These include third party services such as credit card processors, fraud detection services, and geocoding services.

You also have to examine your queries because small queries can become large overnight as you scale. This is why search providers limit the result set that they return. See what kind of joins your ORM is producing when it handles inheritance. Look at the number of objects coming back from a DCOM or RMI call.

Any one of these could fail, or cause latency. As we discussed in a previous post, any potential long latency has to be treated as a potential failure. You need to avoid single points of failure because they are potential bottlenecks or failure points.

Acquiring more resources costs more money. So every strategy is a tradeoff between keeping the application responding (available and scalable) and how much it costs. This is driven of course, by what customers are willingness to pay. Every strategy has to undergo a cost benefit analysis.

The more you approach 100% availability, however, the more the law of diminishing return sets in. I cannot here tell you which problems you should solve, and which you can safely ignore. It depends on your application and your customers. I can tell you that every component is a potential source of failure.

Avoid single points of failure. Accept the fact that you have to build a distributed system.

You have to make sure your cloud application is not brittle. Make your components more resistant to failure. Bridges can withstand more traffic than their largest anticipated load. Since you can add and remove resources in a cloud computing environment, your margin of safety can expand or contract as your load expands or contracts. Nonetheless, adding and removing resources is not instantaneous. You have to make sure that your system can handle a "normal load".

How do you determine your margin of safety?

Look at every resource you use in the system: database sizes, bandwidth, virtual memory, CPU, network latencies, and the response times of your software and your third party components. See how they respond under various types of commands, reports, and queries over time. Because of the economic costs, and possible performance hits with handling failure, you want to ensure your application in its normal state of operations can handle the load. You might want to factor in some likely scenarios, for instance, and make the resources required larger than might be ordinarily needed. Make sure all errors are handled, even unlikely ones. Return clear error codes that indicate what the problem is to the best of your ability. When problems occur, you might degrade performance rather than eliminate functionality. Determine what functionality is essential and what is not. During the Amazon outage last year Netflix turned off personalized movie lists, but you could still get lists of movies and play them.

Use a Margin of Safety when determining the resources needed for your application.