Check out additional Server StorageIO Tips & Articles here along with commentary in the news here

Clarifying Clustered Storage Confusion

July 23, 2009 – 8:44 am

Clustered storage can be iSCSI, Fibre Channel block based or NAS (NFS or CIFS or proprietary file system) file system based. Clustered storage can also be found in virtual tape library (VTL) including dedupe solutions along with other storage solutions such as those for archiving, cloud, medical or other specialized grids among others.

Recently in the IT and data storage specific industry, there has been a flurry of merger and acquisition (M&A) (Here and here), new product enhancement or announcement activity around clustered storage. For example, HP buying clustered file system vendor IBRIX complimenting their previous acquisition of another clustered file system vendor (PolyServe) a few years ago, or, of iSCSI block clustered storage software vendor LeftHand earlier this year. Another recent acquisition is that of LSI buying clustered NAS vendor ONstor, not to mention Dell buying iSCSI block clustered storage vendor EqualLogic about a year and half ago, not to mention other vendor acquisitions or announcements involving storage and clustering.

Where the confusion enters into play is the term cluster which means many things to different people, and even more so when clustered storage is combined with NAS or file based storage. For example, clustered NAS may infer a clustered file system when in reality a solution may only be multiple NAS filers, NAS heads, controllers or storage processors configured for availability or failover.

What this means is that a NFS or CIFS file system may only be active on one node at a time, however in the event of a failover, the file system shifts from one NAS hardware device (e.g. NAS head or filer) to another. On the other hand, a clustered file system enables a NFS or CIFS or other file system to be active on multiple nodes (e.g. NAS heads, controllers, etc.) concurrently. The concurrent access may be for small random reads and writes for example supporting a popular website or file serving application, or, it may be for parallel reads or writes to a large sequential file.

Clustered storage is no longer exclusive to the confines of high-performance sequential and parallel scientific computing or ultra large environments. Small files and I/O (read or write), including meta-data information, are also being supported by a new generation of multipurpose, flexible, clustered storage solutions that can be tailored to support different applications workloads.

There are many different types of clustered and bulk storage systems. Clustered storage solutions may be block (iSCSI or Fibre Channel), NAS or file serving, virtual tape library (VTL), or archiving and object-or content-addressable storage. Clustered storage in general is similar to using clustered servers, providing scale beyond the limits of a single traditional system—scale for performance, scale for availability, and scale for capacity and to enable growth in a modular fashion, adding performance and intelligence capabilities along with capacity.

For smaller environments, clustered storage enables modular pay-as-you-grow capabilities to address specific performance or capacity needs. For larger environments, clustered storage enables growth beyond the limits of a single storage system to meet performance, capacity, or availability needs.

Applications that lend themselves to clustered and bulk storage solutions include:

  • Unstructured data files, including spreadsheets, PDFs, slide decks, and other documents
  • Email systems, including Microsoft Exchange Personal (.PST) files stored on file servers
  • Users’ home directories and online file storage for documents and multimedia
  • Web-based managed service providers for online data storage, backup, and restore
  • Rich media data delivery, hosting, and social networking Internet sites
  • Media and entertainment creation, including animation rendering and post processing
  • High-performance databases such as Oracle with NFS direct I/O
  • Financial services and telecommunications, transportation, logistics, and manufacturing
  • Project-oriented development, simulation, and energy exploration
  • Low-cost, high-performance caching for transient and look-up or reference data
  • Real-time performance including fraud detection and electronic surveillance
  • Life sciences, chemical research, and computer-aided design

Clustered storage solutions go beyond meeting the basic requirements of supporting large sequential parallel or concurrent file access. Clustered storage systems can also support random access of small files for highly concurrent online and other applications. Scalable and flexible clustered file servers that leverage commonly deployed servers, networking, and storage technologies are well suited for new and emerging applications, including bulk storage of online unstructured data, cloud services, and multimedia, where extreme scaling of performance (IOPS or bandwidth), low latency, storage capacity, and flexibility at a low cost are needed.

The bandwidth-intensive and parallel-access performance characteristics associated with clustered storage are generally known; what is not so commonly known is the breakthrough to support small and random IOPS associated with database, email, general-purpose file serving, home directories, and meta-data look-up (Figure 1). Note that a clustered storage system, and in particular, a clustered NAS may or may not include a clustered file system.

Clustered Storage Model: Source The Green and Virtual Data Center (CRC)
Figure 1 – Generic clustered storage model (Courtesy “The Green and Virtual Data Center  (CRC)”

More nodes, ports, memory, and disks do not guarantee more performance for applications. Performance depends on how those resources are deployed and how the storage management software enables those resources to avoid bottlenecks. For some clustered NAS and storage systems, more nodes are required to compensate for overhead or performance congestion when processing diverse application workloads. Other things to consider include support for industry-standard interfaces, protocols, and technologies.

Scalable and flexible clustered file server and storage systems provide the potential to leverage the inherent processing capabilities of constantly improving underlying hardware platforms. For example, software-based clustered storage systems that do not rely on proprietary hardware can be deployed on industry-standard high-density servers and blade centers and utilizes third-party internal or external storage.

Clustered storage is no longer exclusive to niche applications or scientific and high-performance computing environments. Organizations of all sizes can benefit from ultra scalable, flexible, clustered NAS storage that supports application performance needs from small random I/O to meta-data lookup and large-stream sequential I/O that scales with stability to grow with business and application needs.

Additional considerations for clustered NAS storage solutions include the following.

  • Can memory, processors, and I/O devices be varied to meet application needs?
  • Is there support for large file systems supporting many small files as well as large files?
  • What is the performance for small random IOPS and bandwidth for large sequential I/O?
  • How is performance enabled across different application in the same cluster instance?
  • Are I/O requests, including meta-data look-up, funneled through a single node?
  • How does a solution scale as the number of nodes and storage devices is increased?
  • How disruptive and time-consuming is adding new or replacing existing storage?
  • Is proprietary hardware needed, or can industry-standard servers and storage be used?
  • What data management features, including load balancing and data protection, exists?
  • What storage interface can be used: SAS, SATA, iSCSI, or Fibre Channel?
  • What types of storage devices are supported: SSD, SAS, Fibre Channel, or SATA disks?

As with most storage systems, it is not the total number of hard disk drives (HDDs), the quantity and speed of tiered-access I/O connectivity, the types and speeds of the processors, or even the amount of cache memory that determines performance. The performance differentiator is how a manufacturer combines the various components to create a solution that delivers a given level of performance with lower power consumption.

To avoid performance surprises, be leery of performance claims based solely on speed and quantity of HDDs or the speed and number of ports, processors and memory. How the resources are deployed and how the storage management software enables those resources to avoid bottlenecks are more important. For some clustered NAS and storage systems, more nodes are required to compensate for overhead or performance congestion.

Learn more about clustered storage (block, file, VTL/dedupe, archive), clustered NAS, clustered file system, grids and cloud storage among other topics in the following links:

"The Many faces of NAS – Which is appropriate for you?"

Article: Clarifying Storage Cluster Confusion
Presentation: Clustered Storage: “From SMB, to Scientific, to File Serving, to Commercial, Social Networking and Web 2.0”
Video Interview: How to Scale Data Storage Systems with Clustering
Guidelines for controlling clustering
The benefits of clustered storage

Along with other material on the StorageIO Tips and Tools or portfolio archive or events pages.

Ok, nuff said.

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2016 Server StorageIO and UnlimitedIO LLC All Rights Reserved

  1. 31 Responses to “Clarifying Clustered Storage Confusion”

  2. Great article Greg. Would be nice to have a follow up article though where you list a couple examples of the different types with real world architectures like with Lefthand, and how they say performance increases since they stack additional nodes with more controllers, etc. versus n+1 clustered controllers in Fujitsu ETERNUS and in 3PAR. Or what the HPC guys do like with Google stack and their custom software. Did they build their stack because nobody had something that they required?

    This is great stuff though.

    By Frank on Aug 26, 2009

  3. Thanks Frank,

    Glad you liked the post.

    Good note about a follow-up post, if you look at some of the links including presentations, there are examples of various block, file and other clustered storage solutions, perhaps not in the detail you would like, however that could be addressed in the future.

    Your comment about clustering and performance is interesting to me as I hear it regularly from vendors of server and storage clusters. The common line is that performance scales linearly with no degradation when adding nodes while maintaining data integrity (cough cough ;) ).

    Now and then I hear from vendors who will say the performance scales on a near linear basis, with little to no perceived overhead while scaling. Want to take a guess which one Im more inclined to believe of the cuff?

    There is the other common myth that adding more controllers, more ports, faster ports, faster disks, more cache results in better performance which may or may not be true as it’s how all of those pieces work together that makes the differences.

    In the cluster game, some want you to believe that simply adding more controllers will result in more performance which may be true, however what about the overhead of data integrity in the cluster? Some do better than others, some just don’t want to talk about it, and others want to talk around it.

    With the current SSD/flash craze, it’s interesting to watch vendors all of a sudden talk about IOPS however few are talking about latency, even with clusters, hmmm…

    Again thanks for the comments.


    By Greg Schulz on Aug 26, 2009

  4. Check out Alan Raddings new article: How to buy clustered NAS for a virtualized environment


    By Greg Schulz on Aug 27, 2009

  1. 28 Trackback(s)

  2. Sep 3, 2009: Featured IT Blogger of the Week: Greg Schulz of StorageIO - ITKE Community Blog
  3. Jul 11, 2011: Dell Storage Forum 2011 revisited | StorageIOblog
  4. Jul 19, 2011: Dell Storage Forum 2011 revisited - The StorageIO Blog - IBM Storage Community
  5. Jul 19, 2011: Dell Storage Forum 2011 revisited - StorageIO Blog - IBM Storage Community
  6. Sep 8, 2011: HDS buys NAS partner BlueArc after five year OEM courtship | StorageIOblog
  7. Sep 9, 2011: HDS buys BlueArc, any surprises here? - StorageIO Blog - IBM Storage Community
  8. Dec 12, 2011: Storage is Sexy Again « Red Hat Storage
  9. Dec 29, 2011: Top all time storageio cloud virtualization networking and data protection posts | StorageIOblog
  10. Jan 3, 2012: Top storageio cloud virtualization networking and data protection posts - StorageIO Blog - IBM Storage Community
  11. Jan 31, 2012: Post Holiday IT Shopping Bargains, Dell Buying Exanet? | StorageIOblog
  12. Jan 31, 2012: Back to school shopping: Dude, Dell Digests 3PAR Disk storage | StorageIOblog
  13. Feb 6, 2012: Congratulations to IBM for releasing XIV SPC storage benchmarks | StorageIOblog
  14. Sep 7, 2012: Do software vendors eliminate or move location of vendor lock-in? - StorageIO Blog - IBM Storage Community
  15. Sep 10, 2012: The Data Center Journal Does software eliminate or move location of vendor lock-in?
  16. Sep 11, 2012: Does software eliminate or move location of vendor lock-in? | StorageIOblog
  17. Sep 30, 2012: Cloud, virtualization, storage and networking in an election year « Data-Center.BlogNotions - Thoughts from Industry Experts
  18. Oct 9, 2012: Does Dell have a cloudy cloud strategy story (Part I)? - StorageIO Blog - IBM Storage Community
  19. Oct 25, 2012: Does Dell have a cloudy cloud strategy story (Part I)? | StorageIOblog
  20. Nov 26, 2012: Ceph Day Amsterdam 2012 (Object and cloud storage) | StorageIOblog
  21. Nov 27, 2012: Ceph Day Amsterdam 2012 (Object and cloud storage) - StorageIO Blog - IBM Storage Community
  22. Nov 30, 2012: The Data Center Journal Ceph Day Amsterdam 2012 (Object and cloud storage) | Up to the hour news
  23. Dec 10, 2012: The Data Center Journal Ceph Day Amsterdam 2012 (Object and cloud storage)
  24. Dec 10, 2012: Part III: PureSystems, something old, something new, something from big blue | StorageIOblog
  25. May 23, 2013: Ceph Day Amsterdam 2012 (Object and cloud storage) | StorageIOblog
  26. May 23, 2013: Does software eliminate or move location of vendor lock-in? | StorageIOblog
  27. Jun 28, 2013: Does IBM Power7 processor announcement signal storage upgrades? | StorageIOblog
  28. Aug 13, 2013: 2010 and 2011 Trends, Perspectives and Predictions: More of the same? | StorageIOblog
  29. Oct 5, 2013: Data footprint reduction (Part 2): Dell, IBM, Ocarina and Storwize | StorageIOblog

Post a Comment

Powered by Disqus