All Storage Arrays Will Be Hybrid Storage Arrays!

The Register recently wrote about us (http://www.theregister.co.uk/2012/09/11/nimble_forty_million/), and made the point that all storage arrays would become hybrid storage arrays. I find myself agreeing wholeheartedly. Nimble Storage was founded on this premise because we believe that hybrid storage arrays simultaneously optimize performance and useable capacity. Further, intelligent hybrid storage arrays have a cost structure that when combined with efficient, pointer-based snapshots, allows for thousands of snapshots to be collocated with primary application data for near-instantaneous data recovery.

Hybrid Storage Arrays Simultaneously Optimize Performance and Capacity

One of the core founding beliefs that Nimble Storage started with is that flash SSDs are dramatically disruptive – an order of magnitude better at random reads; good at random writes but with the caveat that random writes come at the cost of flash endurance. They are, however, not so good at other things. They are not as good on sequential IO per $, and they are particularly bad on cost of capacity ($/GB) when compared to high density, low-cost HDDs.

Another factor that influences our belief in hybrid storage arrays is the diversity of workloads that are deployed on networked storage systems. When we look at our own experience across 1,000+ deployments, most of our arrays are used to consolidate storage for databases, file servers, Microsoft Exchange mailboxes, SharePoint documents and so on – all on the same storage platform. Some of these workloads are performance-hungry and sensitive to latency, and would benefit from a significant portion or even all of the workload being contained on flash SSDs. Other workloads do not need the performance that flash can deliver, and would rather be optimized for lower cost of capacity.

Given that both flash SSDs and high-density, low-cost HDDs are good at some attributes and poor at others, our belief is that the ideal storage array should leverage the complementary strengths of both media types. Further, when considering the wide variety of workloads in an enterprise and the relative benefits of flash-accelerated performance across those workloads, we believe that a hybrid storage array that is flexible in how it handles various workloads is the ideal solution.

In fact, we believe that hybrid storage arrays represent a major transformation of the traditional, modular storage arrays that have come to become the mainstay of networked storage deployments in most enterprises!

Hybrid Storage Platforms Enable A Transformation in Data Protection

Since the early 1990s, NetApp has been the pioneer of using snapshots for data protection. Most storage systems, to this day, support a small number of snapshots either because array performance degrades when you have too many snapshots and/or the amount of space needed for each incremental snapshot is significant. In contrast, NetApp’s WAFL file system enabled dozens of snapshots that could reside alongside the primary application instance without any performance penalty. The incremental space consumption associated with each snapshot in WAFL was also relatively small in that only changed, incremental 4KB blocks were stored in snapshots.

This advantage results in most NetApp customers leveraging snapshots for rapid recovery in day-to-day operations – recovering in seconds instead of hours or days.  Despite major data recovery benefits, many NetApp customers often end up storing only a small number of snapshots, relying instead on a separate backup system for backups beyond a few days worth of protection. This is because of cost concerns – the snapshots live on the same media as primary application data – expensive, high-RPM drives for even modestly performance-hungry applications!

In sharp contrast, Data Domain leveraged storage systems using low-cost SATA drives combined with data de-duplication and compression to be a cost-effective and dramatically simpler alternative to tapes for backups.  Tape elimination drove rapid growth even though the ability to recover data only improved marginally.

Ideally, you would have both – the low cost of de-duplicated, SATA based arrays and the rapid recovery provided by snapshots and replication. A well-designed hybrid storage array can do just that – it would leverage low-cost, high-density HDDs and employ in-line data compression, while well-designed snapshots would eliminate duplicate blocks across thousands of backup versions and also enable recovery in seconds.

In fact, we believe that hybrid storage arrays will enable a transformation in data protection approaches as most operational backup and recovery will rely on snapshots and replication.

Gartner endorses this view in saying that “a major advantage of snapshots over backup software is that they scale very well.  No data movement means near instantaneous application recoveries.”  The fact that major backup vendors like CommVault are cataloging array-based snapshots and treating them as first class citizens alongside tape backups is indicative that this shift is already underway.

Are All Hybrid Storage Arrays Created Equal?

The founding premise that Nimble started with is now becoming widely accepted, as industry observers generally believe that the bulk of mid-range storage systems will evolve towards hybrid storage systems that leverage flash and HDDs.  In my next blog, I will talk about another core belief that we started with – that all hybrid storage arrays are NOT created equal, and designing the ideal hybrid storage array requires a ground-up design!