Case Study: Graniterock

For Graniterock, which literally paves the freeways and airport runways that help bolster American commerce, data storage serves as a source of critical real-time intelligence. Concrete trucks form in a queue that requires precise orchestration to keep things moving on construction of a major freeway overpass or airport runway. That interplay – without fail – requires that data be accessed and transmitted between a storage server and a foreman's laptop in real time.

The choreography required for such an operation was not always a given. Graniterock faced the challenge of maintaining multiple, disparate storage platforms that were nearing capacity and lagging in performance. But fortuitous timing would come into play with the introduction of the CS240 converged storage array from Nimble Storage.

"We had multiple storage appliances—a SAN from EMC, a two-terabyte NetApp NAS, a four-terabyte Dell MD3000/1000i, a two-terabyte Dell JBOD, two Data Domain appliances and six Buffalo TeraStations," said Ken Schipper, operations manager for Graniterock. "Data was scattered over all of these devices, all with different management systems, all with different capabilities, and—worst of all—unable to meet our storage capacity or performance needs."

That's when Graniterock adopted the CS240. According to Schipper, the company might have standardized on the SAN from its existing vendor but chose to test a beta version of the Nimble array. "We learned quickly that the Nimble array could meet all of our needs. In our initial analysis, we thought we might keep some of the existing devices. But some were going out of warranty or near end-of-life. As we started using the Nimble array for more and more applications, it kept proving that it could exceed any performance highs we had ever achieved—and significantly."

"Today, Nimble is our one-stop storage shop. And we have moved virtually our entire storage operations to Nimble."

Overcoming Capacity Barriers of the Past

Having experienced a number of hardware data failures a year before adopting Nimble, Rick Boston, network engineer for Graniterock, said, "Only fully redundant systems would be placed at the center of Graniterock's operations."

"In the past, redundancy had always come at the cost of capacity loss. Our systems were configured in either RAID 5 or RAID 10, depending on how critical the data was. With RAID 10 you cut down capacity pretty dramatically. And we were getting to the point where our primary enterprise device storage was full."

In addition, Boston said that he and Schipper faced data protection issues. "We were really fighting a constant battle to maintain our primary and backup storage devices and keep them talking. It was taking hours and hours of time."

Today, with a new Nimble-based storage regimen, the ratio of usable to total capacity is high and capacity losses are a thing of the past, said Boston. "In evaluating the move to Exchange 2010, we also moved our Exchange systems to the Nimble arrays. They replicate to each other. We split the active and passive databases on the Exchange system between the two to give us a good measure of redundancy. VMware, our 85 to 90 virtual machines and Exchange—they're all on the Nimble arrays—and we're seeing much, much better performance than we had across all the systems that we discarded."

In its data center, Graniterock's Exchange implementation maintains an active database on one Nimble array and a passive database on the other, each replicating to the other. "At any one time, if we were to lose a mailbox server and a Nimble array, we would still have two active up-to-date production databases on the system. It's all part of a nice interplay between Exchange and Nimble." Boston says that they plan to add a third Nimble array and locate it at a disaster recovery site. The two onsite arrays would both replicate to the third as part of the company's disaster recovery plans.

"We probably have capacity to last into the next millennium," said Boston with a wry smile. "But, of course, somehow over the next five years, we'll fill it," added Schipper.

Performance at an All-Time High

Raw storage performance was the key concern of Boston and Schipper in the move away from disparate storage systems and towards Nimble. But performance can lag in an environment requiring frequent, routine system backup, together with even more frequent snapshots.

"We snapshot everything multiple times a day," said Schipper. "We snapshot our VMware volumes, our Exchange databases and our Exchange server partitions. We have literally taken every type of data storage we have and moved it to the Nimble array."

He said that Graniterock had experienced several storage issues with backups in the past. "We said 'Well, what if we could leverage the snapshot features of the Nimble array? What affect is that going to have storage-wise and performance-wise?' And we decided to try it, and literally, there was no perceptible impact. We can truly look at it today as a background process with no effects on performance. And the storage compression is pretty amazing."

He added: "The I/O is so efficient that those mail servers are hardly working. They're at about 5 to 10 percent. That's a real life-saver. With all of the intelligent devices in use – smartphones and iPads—our people are sending emails over these devices to correspond with customers in real-time."

Closing in on 100 Virtual Machines

For Graniterock, virtualization is key to a host of advantages, but always with the proviso that those advantages will not come at the cost of performance. Today, 85 to 90 virtual machines, previously on nine servers, run on the two Nimble arrays.

And what about the move of the VMs to the Nimble arrays? "We just add the iSCSI interfaces and build partitions on the Nimble array, set up the iSCSI groups, connect up with our VMware host and migrate each server from existing storage," said Schipper. "I'm pleased to say we've never lost a server." Today, he said, he and Boston are working on distributing the VMs more symmetrically across the nine servers.

On the move of the VMs to Nimble, Schipper called the performance boost "amazing." "Problems we had in the past, mostly related to I/Os that were backing up, we couldn't really troubleshoot. Today, what a difference: Those problems have all disappeared, and our VM performance is so much higher than ever in the past."

No More Command Line Interfaces

As if to demonstrate the importance of an effective user interface to storage, Schipper recounts events of the recent past: "We eliminated five backup and storage devices over a matter of weeks. Most of the storage devices we were dealing with had command line interfaces that made daily operations and management difficult."

"Now, everything that we need to do operationally we can access through Nimble's command GUI: the status of every operation, the provisioning of devices, you name it. It was a dramatic change. We can create a new volume in literally minutes. Contrast that with our past regimen, which required three days of planning, downtime on a weekend, and a couple of hours to execute. Or, for a major reconfiguration, the process could take many hours – and then we crossed our fingers."

"That problem has vanished. If we need a new volume, we simply create one. I can do it in minutes – and just possibly in under one minute! We attach our virtual servers to a Nimble array, and the implementation can start literally within a few hours of saying 'yes, let's make it happen.'" As for his daily operations, Schipper says that, with just one screen, he can quickly view a summary of system performance, error messages and exact activity on each volume. "It's all a matter of gaining the confidence that I won't be surprised. And, pleasantly, I've had no surprises."

Doing Business in Real Time

To demonstrate the real-time nature of business at Graniterock, Schipper points to the company's concrete management system. "We have five concrete plants, and all of the concrete truck mixing and dispatching is run from our concrete management system, which runs on the Nimble array. Concrete trucks are loaded, they're mixed, they go on the road, and they have a time of expiration on that concrete."

"It's busy, it's exciting, and it has to be real time. You can't have any delays in getting data to those folks out in the field. Ultimately it's going to cost somebody. It's like a highly choreographed dance," he added. "Drivers show up precisely when site workers are ready to pour. Site workers can't be kept idle, but neither can they tolerate having trucks line up and sit idle. Construction schedules are the lifeblood of the operation, and today they run like clockwork on the Nimble array."

Graniterock prospers with the growth of heavy highway and runway construction. The company has built overpasses in the San Francisco Bay area, including major Highway 85 and Highway 101 interchanges, and performs much of the concrete construction at San Jose's Mineta International Airport.

"In addition, our system is used in estimating required materials, equipment and manpower. Every field supervisor has a laptop that is connected to a back-end system running our serversand storage. We're only as good as our storage and our network."

Upgrades Without Downtime

Schipper and Boston agree that performance, manageability and reliability are the top three benefits they've observed in the move to Nimble. "During a recent HVAC failure the monitored room temperature reached 105 degrees Fahrenheit, and we measured 134 degrees on the Nimble motherboard," said Boston. "It didn't wince. Performance has been fabulous, and we've had no failures." And as for manageability? "An OS upgrade was once a major effort involving downtime. And you never knew what was going to happen. But with Nimble, updates are automated with the aid of its redundant controllers. The Nimble process upgrades one controller, switches over, and upgrades the other controller. It's just a really smooth, well-designed process."

That's quite a shift from the past, he said, when he and Boston would schedule an upgrade several months in advance and would endure hours of system downtime on weekends. "Now I say, 'OK, we're going to do a firmware upgrade, and I think I'll do it Sunday morning. See you on Monday.' With no interruptions, the job is done. Exchange keeps running, the databases keep running, and the VMware systems keep running."

For managing the Nimble arrays, Boston estimates that he spends no more than 10 percent of his time. "That's by comparison with the past, when we needed one person at 100 percent—with the possible half-time addition of another staffer." Schipper puts it into perspective: "We used to require a full-time expert in storage, but today that person can wear five other hats."

"I can't think of any aspect of storage in which I'd rate Nimble below A-plus."