Huy learned recently that the customer wanted to increase the performance of the system – three years after the initial system had been deployed.

First, the customer replaced all of the old flash solid state drives (SSDs) in the system to higher capacity SSDs during the day, watching the amount of cache increase, even as the system continued operations with no disruption to any applications. That evening, Huy arrived and pulled out the old controller and popped in a new, higher performance CS420 controller. He then failed back to the upgraded controller, and replaced the second older controller with a newer CS420 controller. No downtime or disruption to any application, and the performance was now three times higher than with the CS220! The system was now brand new, ready to chug along for several more years.

Having been in the storage industry for a long time, I know that non-disruptive software upgrades are marketed far more often than they are truly realized in customer data centers. Non-disruptive controller upgrades and cache upgrades are non-existent within mid-range enterprise storage systems. This is one of the many capabilities of our architecture that is allowing us to earn customer appreciation and build a strong customer community!

The biggest challenge I have encountered, not only here but everywhere, is that you always had to sacrifice storage for performance. For Service Providers, ensuring solid performance has depended on costly investments in excess disk capacity and expanded footprint just to offset the performance required for their customer environments to run smoothly.  For example, if your storage requirement was just one terabyte of disc space but your I/O requirement was 1000 IOPS, you would be forced to buy a SAN (storage area network) with roughly 4 terabytes rather than only one terabyte to get the random IOPs mandated by your I/O requirement (assuming Raid 50, using 147GB drive with 7200RPM).

My introduction to Nimble Storage began with the novel concept that there was no longer any need to sacrifice storage for performance.

Over the course of my career, I had purchased quite a few SANs and had established relationships with many resellers, so I reached out to one and said, “I’m in the market again. I need to buy storage.” He introduced me to Nimble saying, “You should check this out…they have a very good platform.  Since you have a lot of experience with storage and performance, then you can see whether you like this: put them through the ringer and see whether they can perform in your environment.”

The more I read about Nimble Storage and the more I tested the units, the more I liked it. An engineer at heart, I recognized that Nimble decouples storage performance from backend disks and spindles. This is one of the biggest reasons why we went with Nimble Storage.

It has been a successful rollout and all of our shared cloud platforms, today, are on Nimble. We get the performance we need without having to waste precious budget monies on excess storage or added data center footprint. And we just keep growing with it. This was not possible, in the past. The Nimble Storage performance and compression benefits include:

• 20+ times higher IOPs
• Data Center Space Savings: 5:1 reduction (performance)
• Data Center Space Savings: 2:1 reduction (compression)

Over the past four years we have seen a shift in our industry toward more hosted managed services and cloud services, attracting clients who are not necessarily quite as tech savvy—in other words, attracting business clients.  Before, our business relied chiefly on companies that manage other clients, but now such clients are starting to come directly to us.

Today our business is based on what is really the core of software services: hosted Exchange, hosted desktops, virtual servers and virtual desktops. (We also continue to provide co-location and web services, our bread and butter in our earlier history.)

Nimble hosts our virtual infrastructure, both servers and VDI, plus all the applications that we have deployed, today, and even some legacy applications that were migrated into Nimble. Altogether, Nimble is hosting all of our platforms today: the web hosting platform, Exchange, SharePoint, cloud servers and cloud VDIs.

Remember, we are a service provider, so our ability to offer some of these services is because of Nimble. We could not offer them previously, or at least not with the same level of performance. Four years ago we were strictly a network, bandwidth and co-location facilitator: we also did a lot of web hosting.  Since that time, we have focused more on storage, particularly the advantages that we get from the Nimble Storage platform in compression and performance.

We have also been offering Business Continuity products and services, which we never did before and which has quickly become one of our flagship services. Nimble Storage is used for BC (business continuity)/DR (disaster recovery) from a compute and storage perspective, replicating VMs and data. Because of lower TCO thanks to Nimble, we can offer DR to customers at a fraction of the cost of traditional solutions.

Helping the City of Bradenton to Leverage Nimble and VMware for DR

As an example, we just deployed a DR solution for the City of Bradenton in Florida, which came to us when it discovered that it could take a full two weeks to recover its data its legacy DR system.

The process to back up was also cumbersome, as the City was forced to back up data through an agent to a different storage system, then export that data into media shipped off to a provider at a remote location. The process to recover the data was equally arduous, as the City had to send personnel to get the physical data from the remote location and then drive it somewhere where they could mount it.

With our solution, the City of Bradenton was able to leverage its existing VMware/Nimble Storage environment: we leveraged the replication that happens within these Nimble arrays—taking its backup data through the snapshots Nimble already provides and smoothly transporting it over VPNs to our Nimble arrays and VMware virtual machines here in Miami. It’s good to hear from a customer, as we did from the City, “It looks like it’s going great, just as expected.”

Today, should the City have a DR scenario, it can access its data here very simply through VPNs and be up and running on the same day: its recovery time objective (RTO) has been cut from weeks to just hours.  We also save the City precious budget monies on its contract for recovery, as our scenario can be roughly 40 percent lower than what it was paying.

From the standpoint of BC/DR, the easiest point of entry and where we can really save the client money is when it uses VMware and currently has its data in a Nimble Storage platform.  It also allows regular DR testing:  we do quarterly testing for these clients (or less frequently, if they prefer).

As our CEO, Anton Resnick, says, “We like to think that we are a little bit more responsible, and accountable, and we like to do things right. It is particularly satisfying for us to be able to revolutionize a customer’s DR plan, transforming it from one that calls for three to four days of outage to one that calls for only a couple of hours, if that.”

Finally, from the management perspective, Nimble Storage is very good: the most efficient of all the other SAN vendors I have seen.  And to have the performance baseline included in the management feature is definitely a plus.

One cool thing about this result is that our cost-effective, high-performance hybrid was used for both redo logs and data. This allows DBAs to snapshot and replicate consistent database images for DR (disaster recovery) and rollback.

Here’s Tom, demonstrating mind-boggling techniques for DBAs everywhere.

For this test run, Tom used a single two-socket, sixteen core blade server, and a single CS440G over 2X10gigE links using DM multipath for Oracle ASM. He’s already lined up a bunch more configs to test, and the array still has plenty of headroom to play with. So we’ll see what Tom comes up with in the coming weeks.

Tom’s a big fan of real-world benchmarks – that’s why he insisted on maintaining great latencies (3ms log file sync through AWR) and following Nimble best practices during these test runs. As a result, we feel comfortable Oracle DBAs can trust Nimble to deliver amazing performance in their production environments.

We used some cool new InfoSight-based performance optimization tools to help us identify and clear out a few bottlenecks. These tools will be available directly to customers in an InfoSight release due out later in the year… in the meanime, we are already using them in support to help our customers get the most from their Nimble gear.

We will continue to update this blog with more details and results from the Oracle side, as well as more information on what we’re cooking up for performance junkies in upcoming versions of InfoSight.

Until then, here’s some Hammerora screen candy from the middle of one of the test runs, courtesy of Tom – the 15-minute average for the full test run was ~1.03M TPM.

Here’s some Hammerora screen candy from the middle of one of the test runs, courtesy of Tom.
The 15-minute average for the full test run was ~1.03M TPM.

Hammerora Settings:

• No think & key time
• 200,000 transactions per virtual user
• 105 virtual users
• 15-minute test run with 2 min ramp up time
• 100 warehouses

Results:

Vuser 1:Rampup 1 minutes complete …
Vuser 1:Rampup 2 minutes complete …
Vuser 1:Rampup complete, Taking start AWR snapshot.
Vuser 1:Start Snapshot 83 taken at 27 APR 2013 13:56 of instance PERFASMDB (1) of database PERFASMD (1272977337)
Vuser 1:Timing test period of 15 in minutes
Vuser 1:1 …,
Vuser 1:2 …,
Vuser 1:3 …,
Vuser 1:4 …,
Vuser 1:5 …,
Vuser 1:6 …,
Vuser 1:7 …,
Vuser 1:8 …,
Vuser 1:9 …,
Vuser 1:10 …,
Vuser 1:11 …,
Vuser 1:12 …,
Vuser 1:13 …,
Vuser 1:14 …,
Vuser 1:15 …,
Vuser 1:Test complete, Taking end AWR snapshot.
Vuser 1:End Snapshot 84 taken at 27 APR 2013 14:11 of instance PERFASMDB (1) of database PERFASMD (1272977337)
Vuser 1:Test complete: view report from SNAPID 83 to 84
Vuser 1:105 Virtual Users configured
Vuser 1:TEST RESULT : System achieved 1030899 Oracle TPM at 343103 NOPM
Vuser 1:Checkpoint
Vuser 1:Checkpoint Complete

This result shows that our purpose-built hybrid systems can perform as well as or better than flash-only solutions at a small fraction of the price. With high performance and high capacity together, you really can have your cake and eat it too!

Our decision to use Nimble Storage really started a few years ago when we issued an RFP for a new storage vendor. We were an EqualLogic shop, but were not happy with its snapshot capabilities. At the time, my predecessor had read about the Nimble Storage solution in an analyst research article, so he reached out to Nimble to have it participate in the RFP. After a rigorous evaluation process and detailed analysis, Nimble Storage emerged as the winner.

Since that time we have deployed several different applications on our Nimble Storage array, really hammering its products – the results have brought us confidence in working with them. We are using them for most of our production environment. We have also discovered its snapshot and replication capabilities work as advertised. We are now using it to replicate our data for disaster recovery.

We have also been investing in deploying Citrix because of our challenges in managing computers and applications on a campus with:

• 1,200 employees (staff and faculty)
• 20,000 to 30,000 students per annum
• 20+ computer labs with about a hundred different software applications
• 2,000 workstations (including 800 in computer labs)

A key mandate for us is consistency across-the-board in our lab capabilities: students must be able to move from one lab to the other and still have access to all the same software. This means that if one lab is updated, all the other labs must be updated, too. And, as each new semester approaches, instructors ask for new software in the labs. Such continual updates meant that we had to update each of 800 computers, individually – a very manual, time consuming, tedious process and not something we could do overnight. And, if something was not installed correctly or the instructor decided to make a change, IT staff was forced to go back to each and every one of those machines to update or fix it.

The new library at Langara College is LEED-certified (Leadership in Energy & Environmental Design), and uses innovative wind towers instead of traditional heating and cooling.

As for faculty and staff, it was the same story. Each and every computer was tied to a user. The College does a lot of shuffling of offices and instructors from one area to another, which meant their computers had to go with them, creating a tremendous amount of work for the IT staff: on occasion tying up our IT service desk people for weeks at a time, just moving computers around. Not a very efficient use of time!

At a Synergy conference we were warned by presenters that the SAN (storage area network) was going to be one of the biggest challenges when implementing a Citrix solution, especially when deploying XenDesktop. They all talked “IOPS” (input / output operations per second) and how you need to design the SAN properly. We took these warnings to heart and through an extensive exercise determined that we had an extremely high IOPS requirement.

As a part of the evaluation process, we discovered that the solutions from incumbent storage companies ranged from $500,000 to $1,000,000 to deliver the IOPs we needed for our Citrix environment.

Working at a college, we must stretch our budgets as far as we can so we went back to the team at Nimble and asked, “Is this something that your array can handle?” We were confident because of the way Nimble Storage has architected the CS-Series, but needed to be sure.

Nimble Storage was the only vendor willing to work with us to really test our numbers: shipping us two arrays, enabling us to build out a PoC (Proof of Concept) environment for Citrix, and proving the arrays could handle the workload cost-effectively. Our Citrix consultant remarked that the numbers coming from the Nimble Storage solution were quite impressive.

Now, with Nimble Storage and Citrix, we are transitioning to a virtualized environment that decouples everything from the physical workstations: a very different delivery model from what we had before and one which saves many staff hours that used to go into moving, updating and managing machines. All labs and all computers can quickly and easily be updated simultaneously. Users are no longer tied to physical machines: they can go to any room where there is a thin client and access their desktops. They can even access their desktops from home.

With full deployment of virtualization right down to the desktop level, I foresee my IT resources will be freed up to do more, with my staff being able to help students and employees at a much higher level, rather than just fixing computers.

Overall, we have found the Nimble Storage team to be very responsive and very willing to help us achieve what we needed to achieve – the folks at Nimble, whether technicians or VPs, are truly exceptional. Over the years, the bond has strengthened between the College and Nimble, making it into a true partnership.

Different Customers, But Facing Some Common Challenges

Each of the three customers was different but had some common characteristics in terms of challenges that they faced.

1. The application hosting services arm of a large technology consulting organization that delivers hosted Microsoft applications to end customers leveraging their expertise, ability to deliver high availability and manage deployments at scale. For example, they host over a million customer mailboxes as a managed service.
2. A regional bank with over a hundred branches in the mid-west, deploying Microsoft SQL databases, Exchange, Citrix and Citrix XenApp servers on virtualized infrastructure using VMware  with vCenter Site Recovery Manager (SRM).
3. A leading provider of hybrid cloud services that offers public and private cloud service to mid-market customers using a VMware-based platform and leveraging their own unified management portal as a differentiator.

All three customers evaluated us as a result of some key challenges that they faced with their incumbent solutions. The table below is an extract of some of the problems as presented by the customers:

They Put Us Through an Extensive Testing Process

All three customers evaluated us thoroughly before making the switch.

• The application hosting company evaluated over 6 different storage platforms – three very large storage vendors, two start-ups offering flash-only arrays, and one other hybrid storage start-up. They went through an extensive 10,000 mailbox Exchange test conducted over several weeks.
• The regional bank tested us for their database environment and then migrated all of their Citrix XenApp Servers on to the Nimble Storage arrays during the proof-of-concept process given the performance headroom on the array.
• The cloud service provider tested us for a large telecom client hosting large databases with heavy input/output (IO) loads.

Compelling Benefits

Thank you to each of the three customers for choosing us: (i) The Application hosting company has deployed 7 Nimble arrays for a total capacity of ~280TB across two locations; (ii) The regional bank has deployed ~70TB across a production and disaster recovery (DR) site; and (iii) The cloud service provider has deployed over 300TB across 9 Nimble arrays.

The best part of the presentation at kickoff was to hear each of the customers present some very compelling evidence of the benefits that they saw from deploying Nimble Storage.

1. Significant capital cost and operating cost reductions. All three customers commented on the dramatically fewer resources needed with Nimble compared to all of the alternatives.

• In their initial deployment, the application hosting company replaced the incumbent vendor’s storage system made up of 300 drives with two of our arrays comprising 8 SSDs (solid state drives) and 24 high-density HDDs (hard disk drives). They saved $10,500 a month just on data center and rack space charges. The capital cost reductions were even more significant. The accompanying picture is the slide that the customer presented showing two racks of competitors’ products being replaced by 6U of Nimble storage arrays.

• The regional bank save significantly on rack density and capital costs. Compression alone was able to reduce the data footprint of their databases by a factor of 3.
• The capital cost of the combined Cisco + VMware + Nimble solution was 50% lower than the alternative, while providing additional savings through compression and lower monthly rental charges.

2. Performance. As an example, the regional bank saw much improved ESX boot times, and was able to host not just the databases that saw poor performance but was also able to migrate Citrix XenApp servers given the high performance. The cloud service provider was able to eliminate all of the extensive tuning that they had to do in the past to achieve the performance needed for their clients.
3. Data Protection. The regional banking customer and the Application hosting company have both implemented a data protection scheme that leverages local snapshots and remote replication coupled with VMware SRM. The Application hosting company is now in the process of integrating Nimble snapshots into CommVault’s backup application. Every one of their customers is now able to create almost instantaneous backups and is able to recover rapidly in the event of a problem.
4. Simplicity of Management. In every instance, our customers commented on how much simpler it was to manage a Nimble environment – both in terms of initial deployment as well as on a day-to-day basis.
5. Dramatically Better Support Experience. One attribute of Nimble that every customer commented on very positively was his experience of dealing with our support organization. The attributes that stood out about our support were as follows:

• Through our remote support automation, we have the information we need to solve all the problems fast
• We care deeply enough to solve issues, even when they are outside our domain
• We do not let issues fester and we marshal the internal resources needed to address a problem quickly.

One of the main goals of any sales kick-off is to motivate the sales organization and arm the sales teams with confidence in the strength of our solutions. Nothing could have accomplished this goal as well as the customer presentations did, when they articulated their challenges, their decision process and why they chose Nimble Storage.

Boulevard Brewing Company is the Midwest’s largest craft brewer. We are the 10^th largest craft brewery and 18^th largest brewer in the US. We are available in 29 states and have grown every year since we were founded in 1989.

Before we bought Nimble, we were out of space. We only had 3TB of data, but it was taking 7.85 TB of disk. Because of replication and snapshots, it was wasting a lot of disk space. Once we switched to Nimble, our 3TB of data now only take 2TB and our snapshots are done every hour and only take 50GB of disk.

During our evaluation process, we reviewed our current storage solution, Equallogic. We also looked at EMC and NetApp. We were looking for low cost and high performance – other vendors could provide one, but not both criteria.

Here are the three top reasons we picked Nimble Storage:

1. Price
2. Performance
3. Disk savings with dedupe and compression

We have three ESX hosts in our primary site. We had an EQ PS5000 which had 56 VMs running on the 3 hosts. We have a DR site with 3 more hosts and another EQ PS5000 using SRM. Now we have 2 Nimble CS240s in addition and all primary workload has been moved to the Nimble array.

We bought CS240s because I didn’t want to run out of disk space. We use Panologic zero clients in our manufacturing environment but haven’t deployed any other VDI because of performance and space issues. With Nimble we have the space and performance to move forward with more VDI.

Our experience with Nimble has been great. They have responded quickly. I would love to get the SRA done for SRM but that’s my only complaint.

McElvain Energy, Inc., is a Denver-based oil and gas exploration and production company active in the San Juan Basin, the Permian Basin, the Denver Basin, the Central Kansas Uplift, and the North Dakota Bakken.

Our prior fibre-channel SAN was not supported by a key software application and was nearing full capacity.  In addition, it was excessively complicated to manage and did not offer flexibility to adapt to changing business needs, resulting in wasted capacity and administration time.

After determining our basic needs, we summarily reviewed the market and chose three contenders based upon their representation of different approaches to the storage question.  Dell Equalogic represented a simple, proven IP-based platform, Hitachi represented an established, high-performance, scalable IP-or-fibre player, and Nimble took the position of an up-and-coming, revolutionary approach to storage.

We chose Nimble because:

1. Nimble’s technology, culture, reputation, and market position were in line with our expectations
2. The simplicity and flexibility of the product met or exceeded those of its competitors
3. Nimble offered, and has delivered, no-apologies performance, head and shoulders above not only our prior SAN but also all competitors.

Our Nimble CS220 supports a two-host Vsphere 5 cluster, three virtual machines within the cluster (using direct IP connections to the CS220), a hardware Microsoft Exchange 2007 server, and a hardware Microsoft SQL 2008 R2 server.  Approximately 90 user accounts are active on the system.

In addition, simpler and more flexible management of the array, coupled with its Vsphere integration, has decreased the amount of effort that is necessary to deploy, upgrade, and extend virtual machines to meet business needs.

We are continually impressed by our CS220 array and have benefitted from its simplicity, flexibility, and performance.  In addition, Nimble’s support and engineering staff have responded promptly and effectively to our questions and requests.

Ace Relocation is a family owned and operated moving company that was started in 1968. Ace performs over 14,000 relocations per year, is the fifth largest agency for Atlas Van Lines, and holds a seat on the Atlas board of directors. Ace has nine offices throughout the country with the corporate office located in San Diego. In addition to local equipment at each office, we have the majority of our IT infrastructure at an offsite collocation facility.

Prior to purchasing our Nimble storage array, we had all of our servers running in VMware on an HP SAN. We were running at the capacity of the HP array for almost a year and required an upgrade to something larger. We also had plans to deploy thin clients to our end-users, but all the testing we ran with the thin clients on the HP SAN ended up showing very poor performance.

At the time when we began to evaluate storage vendors, we only looked at Dell EqualLogic, HP, and Compellent. At a demo of one of the products we were considering, a local partner introduced us to Nimble as well. The main two benchmarks we were looking at in making our decision was cost and performance. This is mostly because the budget was limited and good performance was critical in order to run those VMware thin clients on a WAN connection across the country.

We chose Nimble Storage because of its:

1. Performance with the SSD and compression
2. Initial cost with deployment
3. Ease of management along with 24×7 technical support

The local team for Nimble was very helpful in setting up the array at our data center. In about 60 minutes, we had the array in the rack, created a few volumes, and performed a system update.

We purchased a CS-220. Currently we support about 200 users, 8 virtual servers, 5 thin clients (increasing gradually), and primary storage for 1.3 million documents. Among the volumes hosted on the Nimble CS-220, we utilize the snapshot feature on all of them and are able to store several weeks of snapshots without impacting significantly performance or capacity. In addition to the automatic protection templates on the Nimble, we do perform a weekly backup of our servers.

By choosing the Nimble array, we were able to begin deploying thin clients to our remote locations. Without the faster SSD that the Nimble offers, it was nearly impossible to have virtual desktops. The latency reduced by at least half on the virtual desktops after we moved to the Nimble compared to our old HP SAN. The simplified interface of Nimble also allowed for quick learning versus relying on a 3^rd party for support and training in the beginning.

Our department feels well suited with the Nimble array for our organization. We have had a very beneficial overall experience with the performance, reliability, and technical support. Often with other vendors, performing system updates (firmware & BIOS) can be extremely time consuming, with scheduled downtime and large downloads. However, with the Nimble array, the system updates are almost seamless to install given the intuitive interface and features. Our CS-220 array has been an important tool and a valuable investment.

I have worked for the City of Del Mar since 2003. The City of Del Mar is a residential community in San Diego County with a population of approximately 4,500. Del Mar is a Charter City with five City Council members elected at large, and operates under a council-manager form of government.

Prior to implementing the Nimble SAN, the City’s data was housed on a variety of physical servers as local storage with no snapshot or management features. The main fileserver and SQL Database server were quickly running out of storage space. The only option for data recovery was to restore from tape.

In 2011, the City undertook a Datacenter Virtualization/Storage Consolidation project through the  Energy Efficiency and Conservation Block Grant (EECBG) Program, funded by the American Recovery and Reinvestment Act (Recovery Act) of 2009. The goal of the project was to reduce energy used in the datacenter through server virtualization and storage consolidation.  The City issued a Request for Proposals for the purchase of a SAN and integration of the SAN into its Citrix XenServer environment.  The City required a solution was easy to install and manage, did not require additional software or add-ins to perform backups, offered superior performance, and would provide enough capacity for both data and snapshots.

The City selected the Nimble CS210 because of the unique and innovative storage architecture that offered the best value and long term performance. Specifically, compression technology, fault tolerant hardware, and built-in disaster recovery features offered the City the best solution to protect its information assets. In addition, the pro-active support and easy installation and administration were attractive for a small I.T department.

The city of Del Mar purchased a single CS210 and integrated with three XenServer 6.0 hosts.  The majority of Del Mar’s physical servers and some VMware ESX 3.0 VMs including Exchange 2007, SQL 2005/2008, Sharepoint 2007, XenAPP 5.0, were migrated moved to XenServer/Nimble environment.  The Nimble installation was performed by a local engineer and was painless, lasting just a few hours including implementation and training for IT Staff.  Our IT Staff was able to quickly learn how to create volumes and attach to Xen VMs, create protection policies, and fully administer the device to complete the server virtualization and data migration process.

Business benefits include reduction of physical hardware in data center through storage consolidation which has reduced power and cooling requirements; enhanced support by having the remote heartbeat function – support has been aware of issues before IT on a couple of occasions. Management is greatly simplified – less hardware to manage and online software updates means we don’t have to schedule firmware updates after hours.

The experience with Nimble is one of the best relationships that I’ve ever had with a product and vendor overall.  The local engineer stops by to check on us and makes a personal phone call to check on open support cases. The product is rock solid and support is extremely responsive and helpful.

Greenway Medical is part of the healthcare Industry. We make EHR solutions for converting/replacing traditional paper medical records to high tech electronic tools.

One of our significant storage challenges is the complexity of backups. Nimble’s lightweight snapshots allow us to have many backup windows.  Nimble caching and performance has allowed us to move away from the traditional expensive 15K drives as well as power and rack space savings in our datacenter providers.

We were previously a heavy Dell Equallogic shop, which we enjoyed, but we are always open to new solutions.  Ultimately the backups, more efficient snapshots, and great performance lead us to give Nimble a try.

Our top 3 reasons to pick Nimble Storage were:

1. Performance
2. Backups
3. Ease of Use

We purchased three 240 10gb models, which will eventually run up to a 1000 virtual machines in our ESX clusters.

The 2-2.5x compression has helped us greatly to reduce storage cost, also we are enjoying the easy firmware/software updates.  It is also great we can do with 3U of disk space what we were doing with 10-15U’s of competing products.

We are a somewhat new customer but our experience has been great, we have seen low latency, easy backups and great support.  We haven’t had any outages or failures at all.