With 5,000 employees and 280 branch offices, Associated Bank is a prime example of a medium-sized business with big data demands.
After acquiring another bank in 2006, its storage area network (SAN) grew from 17TB to 300TB in less than a year. Since then, due to more applications coming online and federal regulations requiring more data retention, the SAN has grown to 900TB, or 52 times its original size.
Dan Marbes, a systems engineer at the Greenbay, Wisc.-based bank, decided to try solid-state drives (SSD) to increase the performance on I/O-hungry applications, while reducing his spindle footprint.
He bought three SSDs to serve as top-tier storage for business intelligence (BI) applications on his SAN. The flash storage outperformed 60 15,000rpm Fibre Channel disk drives when it came to small-block reads.
However, when Marbes used the SSDs for large-block random reads and any writes, "the 60 15K spindles crushed the SSDs," he said, demonstrating that flash deployments should be strategically targeted at specific applications.
SSDs use non-volatile NAND flash memory chips, which are cheaper than DRAM chips but are still as much as 18 times more expensive than 15,000rpm Fibre Channel or serial SCSI (SAS) drives, according to Gregory Wong, an analyst with market research firm Forward Insights. But prices continue to fall, mostly due to consumer adoption of solid-state drive and NAND flash card technology, which ultimately affects the cost of data center-class products.
According to the market research firm IDC, the global outlook for client-side SSDs -- those used in consumer products such as laptops and tablets -- is expected to grow 10-fold, from 11 million units in 2011 to 100 million units in 2015. The use of NAND in 256GB SSDs is forecast to more than double from 19% of all NAND used in SSDs in 2011 to 42% in 2015. Further, demand for 512GB SSDs is expected to grow from a 0.3% portion in 2011 to 8% in 2015, also underscoring the growing interest in higher density SSDs.
Wong expects mass adoption of SSDs by consumers won't occur until the price has reached about $1 per gigabyte, sometime in the second half of 2012. The more consumer NAND flash that is purchased, the lower the overall price of flash products, even in the data center.
Currently, the price for NAND flash in a SSD form factor is about $9 per gigabyte for high-end, single-level cell (SLC) flash and $3 per gigabyte or multi-level cell (MLC) flash. A new class of MLC, called enterprise MLC or eMLC, can withstand up to 30 times more writes over consumer-grade MLC flash, but it also comes with a price premium of about 20%, Wong said.
By comparison, a Fibre Channel or SAS drive costs between 50 and 60 cents per gigabyte.
Many array manufacturers, such as EMC and Hitachi Data Systems, offer an SSD option, which is ability to fill array drive slots with SSDs in 2.5-in or 3.5-in hard drive form factors. Some vendors also offer automated tiering software that migrates the most frequently accessed data to the SSDs, while keeping less critical data on lower-cost hard drives.
When it comes to PCIe NAND flash cards, like those sold by Fusion-io, Texas Memory Systems, Micron or Virident Systems, which can be used in all-flash arrays or in application servers themselves, prices can go through the roof, but so does performance due to the higher speed interconnect and the proximity of the flash storage to the server processors.
Even so, not everyone "gets" why flash-based PCIe cards are so expensive. Chris Rima, supervisor of infrastructure systems for the Information Services Department at Tuscon Electric Power, a subsidiary of UniSource Energy Corp., paid only half of the retail price for flash cards to accelerate the performance in his company's NetApp into 3170 NAS arrays. That means he paid $30,000 for each card.
Tony Edlebrock, senior systems administrator with Tuscon Electric, said NetApp's flash cards, called Performance Acceleration Modules (PAM), boosted the performance his PeopleSoft and Oracle customer care and billing reporting system to the point that it cut the nightly batch process in half, from from between 8 and 12 hours to 4 to 6 hours.
"The nice thing about flash cache is you just put them in and they begin to improve performance everywhere. Protocol latency went down and throughput shot up," Rima said. "I just don't get why they're so expensive."
According to NetApp, using its PAM flash cards in its NAS arrays makes it possible to reduce the number of disk drives by as much as 75% while achieving the same or better performance.
Flash strategies, use cases
However, because of the exorbitantly high cost of some PCIe-based flash cards, their purchase should be based on multiple business use cases, not a single task, Rima said.
Tuscon Electric Power uses its six flash cards on the front end all of its PeopleSoft and Oracle databases. The cards also act as front-end cache for two-thirds of Unisource's 500 VMware virtual machines (VMs), GIS mapping systems as well as databases that manage power outages. The cards have hold between 40TB and 60TB per NAS cluster.
When the online auction site had problems meeting the I/O storage demands of business units starved for more virtual machine (VM) deployments, its quality assurance division had an idea: swap out hard disk drive-based arrays with SSD storage.
After replacing 100TB of storage in a year, eBay experienced a 50% reduction in standard storage rack space, a 78% drop in power consumption and a five-fold boost in I/O performance. That speed boost now allows eBay to bring a new VM online in five minutes, compared with 45 minutes previously.
The online auction site's Quality Assurance Division had been using both network-attached storage (NAS) and a storage area network (SAN) prior to deploying modular SSD arrays from Nimbus Data Systems. It began by deploying a single 2U (3.5-in high) Nimbus S-class SSD system and HALO storage operating system a year ago.
Today, eBay has 12 SSD arrays with capacity that rivals the amount of traditional hard drive storage used, said Michael Craft, eBay's manager of QA Systems Administration.
Rima said Tuscon Electric is looking to move more of its data onto high-density serial ATA (SATA) hard disks, using NAND flash as a cache on the front end to increase performance. That would easily make the business case for executives to approve the purchase of more solid-state storage, he said.
"If you ... add in the cost of data center space, then you could justify an ROI within 18 months easily because instead of 300GB Fibre Channel disk you can use 2TB SATA disk for a portion of those environments. So you have fewer spindles but the same performance."
Satiating virtualization and the boot storm
One data center advancement driving the adoption of solid-state storage is virtualization. As server density and utilization rates climb, I/O bandwidth bottlenecks, Wong said.
Virtual desktop infrastructures, where hundreds or thousands of machines are all booting from a virtualized server farm, come up against I/O constraints. So, instead of utilizing traditional spinning disks, data center mangers are rolling out flash as boot drives in their servers. The SSDs hold the basic operating systems and meta file data.
Rich Raether, manager of network engineering at the law firm of Quarles & Brady, currently oversees 450 Windows 7 desktops that run from 19 VMs. The firm rolled out its VDI infrastructure at the beginning of this year.
Quarles & Brady has a total of 350 servers, about 260 of which are virtualized through VMware in support of nine U.S. offices and one Shanghai, China location. The virtual machines are all running off of networked storage from Dell EqualLogic storage arrays that include PS5000XV, PS6000S, PS6000SVS and PS6010XVS models.
Quarles & Brady's data center contains a total of 450TB of disk drive capacity, all of which resides on the EqualLogic array except for 35TB on an EMC Celerra NAS filer. The EqualLogic arrays utilize a combination of 15,000rpm SAS drives and SSDs. About 1TB of the capacity is made up by SSDs.
The law firm has five PS5010XVX arrays, three in its Milwaukee data center and two in its Phoenix data center, all of which support its VDI infrastructure.
The lawfirm's EqualLogic XVS arrays are capable of tiering data within the array based on usage patterns. "Hot data" is moved to the SSD drives during the morning "boot storm" and then back to SAS as the day progresses and utilization slows.
The VDI infrastructure runs over 10GbE switches from Cisco. While the law firm also has 10GbE Juniper switches, Raether said he found out the hard way they weren't qualified with the Dell EqualLogic arrays.
Before rolling out SSDs for his VDI infrastructure about six months ago, Raether said he had a number of vendors come in with various solid state and hard drive systems so that he could benchmark their performance in his environment.
The firm performed benchmark testing using large amounts of RAM in servers, all-SSD arrays, the EqualLogic hard drive and SSD-hybrid arrays, and straight-SAS arrays.
"The best performance we received in our VDI environment was to load up a server with RAM and do cache to RAM within the server itself. It outperformed everything. Nothing was even close," Raether said.
The configuration that came in second was using server flash acceleration cards from Fusion-io, which offered almost twice the speed as his other disk arrays. The third best performer were the hybrid arrays.
"It becomes cost prohibitive to really jack these servers up with a lot of memory. And, the Fusion-io cards weren't inexpensive either. So probably the best bang for the buck for us were these hybrid arrays," he said.
Over the past couple of years PCIe-based flash cards that insert directly into servers to accelerate I/O performance have become popular with several vendors offering different products.
Fusion-io makes specialized NAND flash cards designed to boost server throughput up to 278,000 I/Os per second while offering up to 1.28TB of data storage capacity. The cards cost between about $7,000 and $8,000. More recently, Fusion-io came out with a new card that allows VMs to share flash storage from their ioCache adapter card. That card has 600GB of capacity and sells for about $6,900.
In June, Micron also announced its own PCIe-based flash card, the realSSD P320h, which comes in two models that hold 350GB and 700GB, respectively. A 350GB model retails for $5,600.
Texas Memory Systems launched its second-generation PCIe-based SSD last month. Its RamSan-70 SSD has from 450GB to 900GB of capacity. TMS hasn't released pricing on the new card.
Prior to the SSD rollout earlier this year, Quarles & Brady ran its VDI environment off the all-SAS drive arrays. Using the SAS drives, it would take from six to seven minutes to provision a new desktop. When the firm moved to SSDs, that time was cut to less than a minute, Raether said.
"Boot times ... were about a minute and a half to two minutes. And we cut that down to about 25 seconds," he said.
The law firm also uses its SSD pools for its SQL databases. "We started playing around with configurations on SQL and after a while, we came up with an environment where we house our temp databases and log files strictly on SSDs. So we have separate pools. The second pool is the SAS array, and we house the databases on there," Raether said.
Over the past two and a half years that Raether has been using SSDs, people have asked him numerous times why he would putting temporary databases and logs on SSDs, instead of SAS drives? "You really want high read and low write [rates] there," he said. "Honestly, we saw a greater performance boost when we had the temp data bases and log files on them. We saw almost a 2X performance in speed just on performance I/O. Application responsiveness varies, but some of theses things improved drastically."
"If costs were cheaper, I'd put everything on SSD drives," Raether said.