Tag: Azure

Posted on

Gaining Server Storage I/O Insight into Microsoft Windows Server 2016

Server Storage I/O Insight into Microsoft Windows Server 2016

server storage I/O trends
Updated 12/8/16

In case you had not heard, Microsoft announced the general availability (GA, also known as Release To Manufacturing (RTM) ) of the newest version of its Windows server operating system aka Windows Server 2016 along with System Center 2016. Note that as well as being released to traditional manufacturing distribution mediums as well as MSDN, the Windows Server 2016 bits are also available on Azure.

Microsoft Windows Server 2016
Windows Server 2016 Welcome Screen – Source Server StorageIOlab.com

For some this might be new news, or a refresh of what Microsoft announced a few weeks ago (e.g. the formal announcement). Likewise, some of you may not be aware that Microsoft is celebrating WIndows Server 20th Birthday (read more here).

Yet for others who have participated in the public beta aka public technical previews (TP) over the past year or two or simply after the information coming out of Microsoft and other venues, there should not be a lot of surprises.

Whats New With Windows Server 2016

Microsoft Windows Server 2016 Desktop
Windows Server 2016 Desktop and tools – Source Server StorageIOlab.com

Besides a new user interface including visual GUI and Powershell among others, there are many new feature functionalities summarized below:

  • Enhanced time-server with 1ms accuracy
  • Nano and Windows Containers (Linux via Hyper-V)
  • Hyper-V enhanced Linux services including shielded VMs
  • Simplified management (on-premisess and cloud)
  • Storage Spaces Direct (S2D) and Storage Replica (SR) – view more here and here


Storage Replica (SR) Scenarios including synchronous and asynchronous – Via Microsoft.com

  • Resilient File System aka ReFS (now default file system) storage tiering (cache)
  • Hot-swap virtual networking device support
  • Reliable Change Tracking (RCT) for faster Hyper-V backups
  • RCT improves resiliency vs. VSS change tracking
  • PowerShell and other management enhancements
  • Including subordinated / delegated management roles
  • Compliment Azure AD with on premise AD
  • Resilient/HA RDS using Azure SQL DB for connection broker
  • Encrypted VMs (at rest and during live migration)
  • AD Federation Services (FS) authenticate users in LDAP dir.
  • vTPM for securing and encrypting Hyper-V VMs
  • AD Certificate Services (CS) increase support for TPM
  • Enhanced TPM support for smart card access management
  • AD Domain Services (DS) security resiliency for hybrid and mobile devices

Here is a Microsoft TechNet post that goes into more detail of what is new in WIndows Server 2016.

Free ebook: Introducing Windows Server 2016 Technical Preview (Via Microsoft Press)

Check out the above free ebook, after looking through it, I recommend adding it to your bookshelf. There are lots of good intro and overview material for Windows Server 2016 to get you up to speed quickly, or as a refresh.

Storage Spaces Direct (S2D) CI and HCI

Storage Spaces Direct (S2D) builds on Storage Spaces that appeared in earlier Windows and Windows Server editions. Some of the major changes and enhancements include ability to leverage local direct attached storage (DAS) such as internal (or external) dedicated NVMe, SAS and SATA HDDs as well as flash SSDs that used for creating software defined storage for various scenarios.

Scenarios include converged infrastructure (CI) disaggregated as well as aggregated hyper-converged infrastructure (HCI) for Hyper-V among other workloads. Windows Server 2016 S2D nodes communicate (from a storage perspective) via a software storage bus. Data Protection and availability is enabled between S2D nodes via Storage Replica (SR) that can do software based synchronous and asynchronous replication.


Aggregated – Hyper-Converged Infrastructure (HCI) – Source Microsoft.com


Desegregated – Converged Infrastructure (CI) – Source Microsoft.com

The following is a Microsoft produced YouTube video providing a nice overview and insight into Windows Server 2016 and Microsoft Software Defined Storage aka S2D.




YouTube Video Storage Spaces Direct (S2D) via Microsoft.com

Server storage I/O performance

What About Performance?

A common question that comes up with servers, storage, I/O and software defined data infrastructure is what about performance?

Following are some various links to different workloads showing performance for Hyper-V, S2D and Windows Server. Note as with any benchmark, workload or simulation take them for what they are, something to compare that may or might not be applicable to your own workload and environments.

  • Large scale VM performance with Hyper-V and in-memory transaction processing (Via Technet)
  • Benchmarking Microsoft Hyper-V server, VMware ESXi and Xen Hypervisors (Via cisjournal PDF)
  • Server 2016 Impact on VDI User Experience (Via LoginVSI)
  • Storage IOPS update with Storage Spaces Direct (Via TechNet)
  • SQL Server workload (benchmark) Order Processing Benchmark using In-Memory OLTP (Via Github)
  • Setting up testing Windows Server 2016 and S2D using virtual machines (Via MSDN blogs)
  • Storage throughput with Storage Spaces Direct (S2D TP5 (Via TechNet)
  • Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I)

Where To Learn More

For those of you not as familiar with Microsoft Windows Server and related topics, or that simply need a refresh, here are several handy links as well as resources.

  • Introducing Windows Server 2016 (Free ebook from Microsoft Press)
  • What’s New in Windows Server 2016 (Via TechNet)
  • Microsoft S2D Software Storage Bus (Via TechNet)
  • Understanding Software Defined Storage with S2D in Windows Server 2016 (Via TechNet)
  • Microsoft Storage Replica (SR) (Via TechNet)
  • Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I)
  • Microsoft Windows S2D Software Defined Storage (Via TechNet)
  • Windows Server 2016 and Active Directory (Redmond Magazine Webinar)
  • Data Protection for Modern Microsoft Environments (Redmond Magazine Webinar)
  • Resilient File System aka ReFS (Via TechNet)
  • DISKSPD now on GitHub, and the mysterious VMFLEET released (Via TechNet)
  • Hyper-converged solution using Storage Spaces Direct in Windows Server 2016 (Via TechNet)
  • NVMe, SSD and HDD storage configurations in Storage Spaces Direct TP5 (Via TechNet)
  • General information about SSD at www.thessdplace.com and NVMe at www.thenvmeplace.com
  • How to run nested Hyper-V and Windows Server 2016 (Via Altaro and via MSDN)
  • How to run Nested Windows Server and Hyper-V on VMware vSphere ESXi (Via Nokitel)
  • Get the Windows Server 2016 evaluation bits here
  • Microsoft Azure Stack overview and related material via Microsoft
  • Introducing Windows Server 2016 (Via MicrosoftPress)
  • Various WIndows Server and S2D lab scripts (Via Github)
  • Storage Spaces Direct – Lab Environment Setup (Via Argon Systems)
  • Setting up S2D with a 4 node configuration (Via StarWind blog)
  • SQL Server workload (benchmark) Order Processing Benchmark using In-Memory OLTP (Via Github)
  • Setting up testing Windows Server 2016 and S2D here using virtual machines (Via MSDN blogs)
  • Hyper-V large-scale VM performance for in-memory transaction processing (Via Technet)
  • BrightTalk Webinar – Software-Defined Data Centers (SDDC) are in your Future (if not already here)
  • Microsoft TechNet: Understand the cache in Storage Spaces Direct
  • BrightTalk Weibniar – Software-Defined Data Infrastructures Enabling Software-Defined Data Centers
  • Happy 20th Birthday Windows Server, ready for Server 2016?
  • Server StorageIO resources including added links, tools, reports, events and more.

What This All Means

While Microsoft Windows Server recently celebrated its 20th birthday (or anniversary), a lot has changed as well as evolved. This includes Windows Servers 2016 supporting new deployment and consumption models (e.g. lightweight Nano, full data center with desktop interface, on-premises, bare metal, virtualized (Hyper-V, VMware, etc) as well as cloud). Besides how consumed and configured, which can also be for CI and HCI modes, Windows Server 2016 along with Hyper-V extend the virtualization and container capabilities into non-Microsoft environments specifically around Linux and Docker. Not only are the support for those environments and platforms enhanced, so to are the management capabilities and interfaces from Powershell to Bash Linux shell being part of WIndows 10 and Server 2016.

What this all means is that if you have not looked at Windows Server in some time, its time you do, even if you are not a WIndows or Microsoft fan, you will want to know what it is that has been updated (perhaps even update your fud if that is the case) to stay current. Get your hands on the bits and try Windows Server 2016 on a bare metal server, or as a VM guest, or via cloud including Azure, or simply leverage the above resources to learn more and stay informed.

Ok, nuff said, for now…

Cheers
Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, vSAN and VMware vExpert. Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier) and twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

Posted on

Happy 20th Birthday Microsoft Windows Server, get ready for Windows Server 2016

Happy 20th Birthday Windows Server, ready for Server 2016?

server storage I/O trends

In case you have not heard, Microsoft is celebrating the 20th birthday (or anniversary) of Windows Server.

Microsoft has a nice site with info graphics and timelines of where Windows Server has been and accomplished over the past 20 years (view here).

Some of you may remember from 20 years ago Windows Server with a different name aka Windows NT Server. Back in the day, if you recall (or read), server requirements were more in the 33 MHz vs. 3.3GHz range, 32MB of RAM Memory vs. 32GB to 320GB, 150MB HDD vs. 150GB SSD or 1.5TB HDD.

Keep in mind that 20 years ago Linux was a relative new thing with Red Hat not yet quite household or more specific enterprise name. The various Unix (e.g. IBM AIX, HP HP-UX, Sun Solaris, DEC Unix and Ultrix among many others) were still dominate, OS2 had peaked or close to, among others. Virtual Machines were Logical Partitions (LPAR) on Mainframes along with virtual PCs software and hardware assist boards.

IMHO there is no coincidence of Microsoft celebrating 20 years of WIndows Server going into the fall of 2016 and the upcoming release of Server 2016.

What’s New in Server 2016 (TP5)?

If you have not done so, check out the latest Tech Preview 5 (TP5) of Windows Server 2016 (get the bits e.g. software here to try) which includes Storage Spaces Direct (S2D) that leverages internal PCIe and drive format SSD (NVMe, SAS, SATA) along with HDDs (SAS, SATA) for creating local and scale-out converged (desegregated) and hyper-converged (aggregated) solutions. In addition to S2D there is Storage Replica (SR) which is replication of local storage part of S2D (not to be confused with DFS or other replication).

Other enhancements include ReFS as the default file system instead of NTFS (don’t worry, NTFS like FAT does not go away yet). There are enhancements to Hyper-V including VM shielding, hot-plug virtual network adapters, enhanced Linux support and fail over priorities among others. Other enhancements include updates for AD including improved integration with on-premises as well as Azure AD for hybrid environments, PowerShell updates, Docker management including Linux (via Hyper-V) and Windows via Nano) container engines.

Microsoft Windows Server 2016 TP5

Speaking of Nano, if you had not heard, this is a new very light weight Windows Kernel that removes 32 bit WOW and GUI support. The result is that Nano is a very small physical (under 1GB image instance size) using less disk, less memory and less CPU to do a given amount of work, oh, and boots super fast, even without SSD. By not having all the 32 bit and GUI overhead, the intent with Nano is there should be fewer updates and maintenance tasks to do, while enabling Windows containers for SQL Server and other applications.

In addition to PowerShell, AD and other management enhancements, Windows Server 2016 (TP5) also enables bridging two worlds e.g. traditional on-premises (or cloud) based Windows Server and Public Cloud (e.g. Azure) and Private or Hybrid including Azure Stack. Note that if you have not heard of Azure Stack and are looking at cloud stacks such as OpenStack, do your due diligence and at least familiarize yourself with Azure Stack.

View more about WIndows 2016 TP5 enhancements here.

Where To Learn More

What This All Means

Congratulations Microsoft and Windows Server on 20th birthday (anniversary) you have come a long way.

With the new features and functionality in Windows Server 2016, looks like there is still a good future for the software defined server.

Ok, nuff said, for now…

Cheers
Gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

Posted on

Big Files Lots of Little File Processing Benchmarking with Vdbench

Big Files Lots of Little File Processing Benchmarking with Vdbench


server storage data infrastructure i/o File Processing Benchmarking with Vdbench

Updated 2/10/2018

Need to test a server, storage I/O networking, hardware, software, services, cloud, virtual, physical or other environment that is either doing some form of file processing, or, that you simply want to have some extra workload running in the background for what ever reason? An option is File Processing Benchmarking with Vdbench.

I/O performance

Getting Started


Here’s a quick and relatively easy way to do it with Vdbench (Free from Oracle). Granted there are other tools, both for free and for fee that can similar things, however we will leave those for another day and post. Here’s the con to this approach, there is no Uui Gui like what you have available with some other tools Here’s the pro to this approach, its free, flexible and limited by your creative, amount of storage space, server memory and I/O capacity.

If you need a background on Vdbench and benchmarking, check out the series of related posts here (e.g. www.storageio.com/performance).

Get and Install the Vdbench Bits and Bytes


If you do not already have Vdbench installed, get a copy from the Oracle or Source Forge site (now points to Oracle here).

Vdbench is free, you simply sign-up and accept the free license, select the version down load (it is a single, common distribution for all OS) the bits as well as documentation.

Installation particular on Windows is really easy, basically follow the instructions in the documentation by copying the contents of the download folder to a specified directory, set up any environment variables, and make sure that you have Java installed.

Here is a hint and tip for Windows Servers, if you get an error message about counters, open a command prompt with Administrator rights, and type the command:

$ lodctr /r


The above command will reset your I/O counters. Note however that command will also overwrite counters if enabled so only use it if you have to.

Likewise *nix install is also easy, copy the files, make sure to copy the applicable *nix shell script (they are in the download folder), and verify Java is installed and working.

You can do a vdbench -t (windows) or ./vdbench -t (*nix) to verify that it is working.

Vdbench File Processing

There are many options with Vdbench as it has a very robust command and scripting language including ability to set up for loops among other things. We are only going to touch the surface here using its file processing capabilities. Likewise, Vdbench can run from a single server accessing multiple storage systems or file systems, as well as running from multiple servers to a single file system. For simplicity, we will stick with the basics in the following examples to exercise a local file system. The limits on the number of files and file size are limited by server memory and storage space.

You can specify number and depth of directories to put files into for processing. One of the parameters is the anchor point for the file processing, in the following examples =S:\SIOTEMP\FS1 is used as the anchor point. Other parameters include the I/O size, percent reads, number of threads, run time and sample interval as well as output folder name for the result files. Note that unlike some tools, Vdbench does not create a single file of results, rather a folder with several files including summary, totals, parameters, histograms, CSV among others.


Simple Vdbench File Processing Commands

For flexibility and ease of use I put the following three Vdbench commands into a simple text file that is then called with parameters on the command line.
fsd=fsd1,anchor=!fanchor,depth=!dirdep,width=!dirwid,files=!numfiles,size=!filesize

fwd=fwd1,fsd=fsd1,rdpct=!filrdpct,xfersize=!fxfersize,fileselect=random,fileio=random,threads=!thrds

rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=!etime,interval=!itime

Simple Vdbench script

# SIO_vdbench_filesystest.txt

#

# Example Vdbench script for file processing

#

# fanchor = file system place  where directories and files will be created

# dirwid = how wide should the directories be (e.g. how many directories wide)

# numfiles = how many files per directory

# filesize = size in in k, m, g e.g. 16k = 16KBytes

# fxfersize = file I/O transfer size in kbytes

# thrds = how many threads or workers

# etime = how long to run in minutes (m) or hours (h)

# itime = interval sample time e.g. 30 seconds

# dirdep = how deep the directory tree

# filrdpct = percent of reads e.g. 90 = 90 percent reads

# -p processnumber = optional specify a process number, only needed if running multiple vdbenchs at same time, number should be unique

# -o output file that describes what being done and some config info

#

# Sample command line shown for Windows, for *nix add ./

#

# The real Vdbench script with command line parameters indicated by !=

#


fsd=fsd1,anchor=!fanchor,depth=!dirdep,width=!dirwid,files=!numfiles,size=!filesize


fwd=fwd1,fsd=fsd1,rdpct=!filrdpct,xfersize=!fxfersize,fileselect=random,fileio=random,threads=!thrds


rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=!etime,interval=!itime

Big Files Processing Script


With the above script file defined, for Big Files I specify a command line such as the following.
$ vdbench -f SIO_vdbench_filesystest.txt fanchor=S:\SIOTemp\FS1 dirwid=1 numfiles=60 filesize=5G fxfersize=128k thrds=64 etime=10h itime=30 numdir=1 dirdep=1 filrdpct=90 -p 5576 -o SIOWS2012R220_NOFUZE_5Gx60_BigFiles_64TH_STX1200_020116

Big Files Processing Example Results


The following is one of the result files from the folder of results created via the above command for Big File processing showing totals.



Run totals


21:09:36.001 Starting RD=format_for_rd1


Feb 01, 2016 .Interval. .ReqstdOps.. ...cpu%...  read ....read.... ...write.... ..mb/sec... mb/sec .xfer.. ...mkdir... ...rmdir... ..create... ...open.... ...close... ..delete...

                          rate  resp total  sys   pct   rate  resp   rate  resp  read write  total    size  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp

21:23:34.101   avg_2-28 2848.2  2.70   8.8 8.32   0.0    0.0  0.00 2848.2  2.70  0.00 356.0 356.02  131071   0.0  0.00   0.0  0.00   0.1 109176   0.1  0.55   0.1  2006   0.0  0.00


21:23:35.009 Starting RD=rd1; elapsed=36000; fwdrate=max. For loops: None


07:23:35.000 avg_2-1200 4939.5  1.62  18.5 17.3  90.0 4445.8  1.79  493.7  0.07 555.7 61.72 617.44  131071   0.0  0.00   0.0  0.00   0.0  0.00   0.1  0.03   0.1  2.95   0.0  0.00

Lots of Little Files Processing Script


For lots of little files, the following is used.



$ vdbench -f SIO_vdbench_filesystest.txt fanchor=S:\SIOTEMP\FS1 dirwid=64 numfiles=25600 filesize=16k fxfersize=1k thrds=64 etime=10h itime=30 dirdep=1 filrdpct=90 -p 5576 -o SIOWS2012R220_NOFUZE_SmallFiles_64TH_STX1200_020116

Lots of Little Files Processing Example Results


The following is one of the result files from the folder of results created via the above command for Big File processing showing totals.
Run totals


09:17:38.001 Starting RD=format_for_rd1


Feb 02, 2016 .Interval. .ReqstdOps.. ...cpu%...  read ....read.... ...write.... ..mb/sec... mb/sec .xfer.. ...mkdir... ...rmdir... ..create... ...open.... ...close... ..delete...

                          rate  resp total  sys   pct   rate  resp   rate  resp  read write  total    size  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp

09:19:48.016    avg_2-5  10138  0.14  75.7 64.6   0.0    0.0  0.00  10138  0.14  0.00 158.4 158.42   16384   0.0  0.00   0.0  0.00 10138  0.65 10138  0.43 10138  0.05   0.0  0.00


09:19:49.000 Starting RD=rd1; elapsed=36000; fwdrate=max. For loops: None


19:19:49.001 avg_2-1200 113049  0.41  67.0 55.0  90.0 101747  0.19  11302  2.42 99.36 11.04 110.40    1023   0.0  0.00   0.0  0.00   0.0  0.00  7065  0.85  7065  1.60   0.0  0.00

Where To Learn More

View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

The above examples can easily be modified to do different things particular if you read the Vdbench documentation on how to setup multi-host, multi-storage system, multiple job streams to do different types of processing. This means you can benchmark a storage systems, server or converged and hyper-converged platform, or simply put a workload on it as part of other testing. There are even options for handling data footprint reduction such as compression and dedupe.

Ok, nuff said, for now.

Gs

Greg Schulz - Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

Some August 2015 Amazon Web Services (AWS) and Microsoft Azure Cloud Updates

Storage I/O trends

Some August 2015 Amazon Web Services (AWS) and Microsoft Azure Cloud Updates

Cloud Services Providers continue to extend their feature, function and capabilities and the following are two examples. Being a customer of both Amazon Web Services (AWS) as well as Microsoft Azure (among others), I receive monthly news updates about service improvements along with new features. Here are a couple of examples involving recent updates from AWS and Azure.

Azure enhancements

Microsoft Azure customer update

Azure Premium Storage generally available in Japan East

Solid State Device (SSD) based Azure Premium Storage is now available in Japan East region. Add up to 32 TB and more than 64,000 IOPs (read operations) per virtual machine with  Azure Premium Storage. Learn more about Azure storage and pricing here.

Azure Data Factory generally available

Data Factory is a cloud based data integration service for automated management as well as movement and transformation of data, learn more and view pricing options here.

AWS Partner Updates

Recent Amazon Web Services (AWS) customer update included the following pertaining to partner storage solutions.

AWS partner updates

AWS Partner Network APN

Learn more about AWS Partner Network (APN) here or click on the above image.

AWS APN competency programs include:

  • Storage
  • Healthcare
  • Life Sciences
  • SAP Solutions
  • Microsoft Solutions
  • Oracle Solutions
  • Marketing and Commerce
  • Big Data
  • Security
  • Digital Media

AWS Partner Network (APN) Solutions for Storage include:

Archiving to AWS Glacier

  • Commvault
  • NetApp (AltaVault)

    • Backup to AWS using S3

  • CloudBerry Lab
  • Commvault
  • Ctera
  • Druva
  • NetApp (AltaVault)

    • Primary Cloud File and NAS storage complementing on-premises (e.g. your local) storage

  • Avere
  • Ctera
  • NetApp (Cloud OnTap)
  • Panzura
  • SoftNAS
  • Zadara

    • Secure File Transfer

  • Aspera
  • Signiant

    • Note that the above are those listed on the AWS Storage Partner Page as of this being published and subject to change. Likewise other solutions that are not part of the AWS partner program may not be listed.

    Where to read, watch and learn more

    Storage I/O trends

    What this all means and wrap up

    Cloud Service Providers (CSP) continue to enhance their capabilities, as well as their footprints as part of growth. In addition to technology, tools and number of regions, sites and data centers, the CSPs are also expanding their partner networks both about how many partners, also in the scope of those partnerships. Some of these partnerships are in the scope of the cloud as a destination, others are for enabling hybrid where public clouds become an extension complementing traditional IT. Everything is not the same in most environments and one type of cloud approach does not have to suit or fit all needs, hence the value of hybrid cloud deployment and usage.

    Ok, nuff said, for now…

    Cheers
    Gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    Cloud conversations: If focused on cost you might miss other cloud storage benefits

    Storage I/O trends

    Cloud conversations: If focused on cost you might miss other cloud storage benefits

    Drew Robb (@robbdrew) has a good piece (e.g. article) over at InfoStor titled Eight Ways to Avoid Cloud Storage Pricing Surprises that you can read here.

    Drew start’s his piece out with this nice analogy or story:

    Let’s begin with a cautionary tale about pricing: a friend hired a moving company as they quoted a very attractive price for a complex move. They lured her in with a low-ball price then added more and more “extras” to the point where their price ended up higher than many of the other bids she passed up. And to make matters worse, they are already two weeks late with delivery of the furniture and are saying it might take another two weeks.

    Drew extends his example in his piece to compare how some cloud providers may start with pricing as low as some amount only for the customer to be surprised when they did not do their homework to learn about the various fees.

    Note that most reputable cloud providers do not hide their fees even though there are myths that all cloud vendors have hidden fees, instead they list what those costs are on their sites. However that means the smart shopper or person procuring cloud services needs to go look for those fee’s and what they mean to avoid surprises. On the other hand if you can not find what extra fee’s would be along with what is or is not included in a cloud service price, to quote Jenny’s line in the movie Forest Gump, "…Run, Forest! Run!…".

    In Drew’s piece he mentions five general areas to keep an eye on pertaining cloud storage costs including:

    • Be Duly Diligent
    • Trace Out Application Interaction
    • Avoid Fixed Usage Rates
    • Beware Lowballing
    • Demand Enterprise Visibility

    Beware Lowballing

    In Drew’s piece, he includes a comment from myself shown below.

    Just as in the moving business, lowballing is alive and well in cloud pricing. Greg Schulz, an analyst with StorageIO Group, warned users to pay attention to services that have very low-cost per GByte/TByte yet have extra fees and charges for use, activity or place service caps. Compare those with other services that have higher base fees and attempt to price it based on your real storage and usage patterns.

    “Watch out for usage and activity fees with lower cost services where you may get charged for looking at or visiting your data, not to mention for when you actually need to use it,” said Schulz. “Also be aware of limits or caps on performance that may apply to a particular class of service.”

    As a follow-up to Drew’s good article, I put together the following thoughts that appeared earlier this year over at InfoStor titled Cloud storage: Is It All About Cost? that you can read here. In that article I start out with the basic question of:

    So what is your take on cloud storage, and in what context?

    Is cloud storage all about removing cost, cost cutting, free storage?

    Or perhaps even getting something else in addition to free storage?

    I routinely talk with different people from various backgrounds, environments from around the world, and the one consistency I hear when it comes to cloud services including storage is that there is no consistency.

    What I mean by this is that there are the cloud crowd cheerleaders who view or cheer for anything cloud related, some of them actually use the cloud vs. simply cheering.

    What does this have to do with cloud costs

    Simple, how do you know if cloud is cheaper or more expensive if you do not know your own costs?

    How do you know if cloud storage is available, reliable, durable if you do not have a handle on your environment?

    Are you making apples to oranges comparisons or simple trading or leveraging hype and fud for or against?

    Similar to regular storage, how you choose to use and configure on-site traditional storage for high-availability, performance, security among other best practices should be applied to cloud solutions. After all, only you can prevent cloud (or on premise) data loss, granted it is a shared responsibility. Shared responsibility means your service provider or system vendor needs to deliver quality robust solution that you can then take responsibility for configure to use with resiliency.

    For some of you perhaps cloud might be about lowering, reducing or cutting storage costs, perhaps even getting some other service(s) in addition to free storage.

    On the other hand, some of you might be

    Yet another class of cloud storage (e.g. AWS EBS) are those intended or optimized to be accessed from within a cloud via cloud servers or compute instances (e.g. AWS EC2 among others) vs. those that are optimized for both inside the cloud as well as outside the cloud access (e.g. AWS S3 or Glacier with costs shown here). I am using AWS examples; however, you could use Microsoft Azure (pricing shown here), Google (including their new Nearline service with costs shown here), Rackspace, (calculator here or other cloud files pricing here), HP Cloud (costs shown here), IBM Softlayer (object storage costs here) and many others.

    Not all types of cloud storage are the same, which is similar to traditional storage you may be using or have used in your environment in the past. For example, there is high-capacity low-cost storage, including magnetic tape for data protection, archiving of in-active data along with near-line hard disk drives (HDD). There are different types of HDDs, as well as fast solid-state devices (SSD) along with hybrid or SSHD storage used for different purposes. This is where some would say the topic of cloud storage is highly complex.

    Where to learn more

    Data Protection Diaries
    Cloud Conversations: AWS overview and primer)
    Only you can prevent cloud data loss
    Is Computer Data Storage Complex? It Depends
    Eight Ways to Avoid Cloud Storage Pricing Surprises
    Cloud and Object Storage Center
    Cloud Storage: Is It All About Cost?
    Cloud conversations: Gaining cloud confidence from insights into AWS outages (Part II)
    Given outages, are you concerned with the security of the cloud?
    Is the cost of cloud storage really cheaper than traditional storage?
    Are more than five nines of availability really possible?
    What should I look for in an enterprise file sync-and-share app?
    How do primary storage clouds and cloud for backup differ?
    What should I consider when using SSD cloud?
    What’s most important to know about my cloud privacy policy?
    Data Archiving: Life Beyond Compliance
    My copies were corrupted: The 3-2-1 rule
    Take a 4-3-2-1 approach to backing up data

    What this means

    In my opinion there are cheap clouds (products, services, solutions) and there are low-cost options as well as there are value and premium offerings. Avoid confusing value with cheap or low-cost as something might have a higher cost, however including more capabilities or fees included that if useful can be more value. Look beyond the up-front cost aspects of clouds also considering ongoing recurring fees for actually using a server or solution.

    If you can find low-cost storage at or below a penny per GByte per month that could be a good value if it also includes many free access, retrieval GETS head and lists for management or reporting. On the other hand, if you find a service that is at or below a penny per GByte per month however charges for any access including retrieval, as well as network bandwidth fees along with reporting, that might not be as good of a value.

    Look beyond the basic price and watch out for statements like "…as low as…" to understand what is required to get that "..as low as.." price. Also understand what the extra fee’s are which most of the reputable providers list these on their sites, granted you have to look for them. If you are already using cloud services, pay attention to your monthly invoices and track what you are paying for to avoid surprises.

    From my InfoStor piece:

    For cloud storage, instead of simply focusing on lowest cost of storage per capacity, look for value, along with ability to configure or use with as much resiliency as you need. Value will mean different things depending on your needs and cloud storage servers, yet the solution should be cost-effective with availability including durability, secure and applicable performance.

    Shopping for cloud servers and storage is similar to acquiring regular servers and storage in that you need to understand what you are acquiring along with up-front and recurring fee’s to understand the total cost of ownership and cost of operations not to mention making apples to apples vs. apples to oranges comparisons.

    Btw, instead of simply using lower cost cloud services to cut cost, why not also use those capabilities to create or park another copy of your important data somewhere else just to be safe…

    What say you about cloud costs?

    Ok, nuff said, for now…

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    How to test your HDD SSD or all flash array (AFA) storage fundamentals

    How to test your HDD SSD AFA Hybrid or cloud storage

    server storage data infrastructure i/o hdd ssd all flash array afa fundamentals

    Updated 2/14/2018

    Over at BizTech Magazine I have a new article 4 Ways to Performance Test Your New HDD or SSD that provides a quick guide to verifying or learning what the speed characteristic of your new storage device are capable of.

    An out-take from the article used by BizTech as a "tease" is:

    These four steps will help you evaluate new storage drives. And … psst … we included the metrics that matter.

    Building off the basics, server storage I/O benchmark fundamentals

    The four basic steps in the article are:

    • Plan what and how you are going to test (what’s applicable for you)
    • Decide on a benchmarking tool (learn about various tools here)
    • Test the test (find bugs, errors before a long running test)
    • Focus on metrics that matter (what’s important for your environment)

    Server Storage I/O performance

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    To some the above (read the full article here) may seem like common sense tips and things everybody should know otoh there are many people who are new to servers storage I/O networking hardware software cloud virtual along with various applications, not to mention different tools.

    Thus the above is a refresher for some (e.g. Dejavu) while for others it might be new and revolutionary or simply helpful. Interested in HDD’s, SSD’s as well as other server storage I/O performance along with benchmarking tools, techniques and trends check out the collection of links here (Server and Storage I/O Benchmarking and Performance Resources).

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Server Storage I/O Benchmark Tools: Microsoft Diskspd (Part I)

    Server Storage I/O Benchmark Tools: Microsoft Diskspd (Part I)

    server storage I/O trends

    This is part-one of a two-part post pertaining Microsoft Diskspd.that is also part of a broader series focused on server storage I/O benchmarking, performance, capacity planning, tools and related technologies. You can view part-two of this post here, along with companion links here.

    Background

    Many people use Iometer for creating synthetic (artificial) workloads to support benchmarking for testing, validation and other activities. While Iometer with its GUI is relatively easy to use and available across many operating system (OS) environments, the tool also has its limits. One of the bigger limits for Iometer is that it has become dated with little to no new development for a long time, while other tools including some new ones continue to evolve in functionality, along with extensibility. Some of these tools have optional GUI for easy of use or configuration, while others simple have extensive scripting and command parameter capabilities. Many tools are supported across different OS including physical, virtual and cloud, while others such as Microsoft Diskspd are OS specific.

    Instead of focusing on Iometer and other tools as well as benchmarking techniques (we cover those elsewhere), lets focus on Microsoft Diskspd.


    server storage I/O performance

    What is Microsoft Diskspd?

    Microsoft Diskspd is a synthetic workload generation (e.g. benchmark) tool that runs on various Windows systems as an alternative to Iometer, vdbench, iozone, iorate, fio, sqlio among other tools. Diskspd is a command line tool which means it can easily be scripted to do reads and writes of various I/O size including random as well as sequential activity. Server and storage I/O can be buffered file system as well non-buffered across different types of storage and interfaces. Various performance and CPU usage information is provided to gauge the impact on a system when doing a given number of IOP’s, amount of bandwidth along with response time latency.

    What can Diskspd do?

    Microsoft Diskspd creates synthetic benchmark workload activity with ability to define various options to simulate different application characteristics. This includes specifying read and writes, random, sequential, IO size along with number of threads to simulate concurrent activity. Diskspd can be used for testing or validating server and storage I/O systems along with associated software, tools and components. In addition to being able to specify different workloads, Diskspd can also be told which processors to use (e.g. CPU affinity), buffering or non-buffered IO among other things.

    What type of storage does Diskspd work with?

    Physical and virtual storage including hard disk drive (HDD), solid state devices (SSD), solid state hybrid drives (SSHD) in various systems or solutions. Storage can be physical as well as partitions or file systems. As with any workload tool when doing writes, exercise caution to prevent accidental deletion or destruction of your data.


    What information does Diskspd produce?

    Diskspd provides output in text as well as XML formats. See an example of Diskspd output further down in this post.

    Where to get Diskspd?

    You can download your free copy of Diskspd from the Microsoft site here.

    The download and installation are quick and easy, just remember to select the proper version for your Windows system and type of processor.

    Another tip is to remember to set path environment variables point to where you put the Diskspd image.

    Also stating what should be obvious, don’t forget that if you are going to be doing any benchmark or workload generation activity on a system where the potential for a data to be over-written or deleted, make sure you have a good backup and tested restore before you begin, if something goes wrong.


    New to server storage I/O benchmarking or tools?

    If you are not familiar with server storage I/O performance benchmarking or using various workload generation tools (e.g. benchmark tools), Drew Robb (@robbdrew) has a Data Storage Benchmarking Guide article over at Enterprise Storage Forum that provides a good framework and summary quick guide to server storage I/O benchmarking.




    Via Drew:

    Data storage benchmarking can be quite esoteric in that vast complexity awaits anyone attempting to get to the heart of a particular benchmark.

    Case in point: The Storage Networking Industry Association (SNIA) has developed the Emerald benchmark to measure power consumption. This invaluable benchmark has a vast amount of supporting literature. That so much could be written about one benchmark test tells you just how technical a subject this is. And in SNIA’s defense, it is creating a Quick Reference Guide for Emerald (coming soon).


    But rather than getting into the nitty-gritty nuances of the tests, the purpose of this article is to provide a high-level overview of a few basic storage benchmarks, what value they might have and where you can find out more. 

    Read more here including some of my comments, tips and recommendations.


    In addition to Drew’s benchmarking quick reference guide, along with the server storage I/O benchmarking tools, technologies and techniques resource page (Server and Storage I/O Benchmarking 101 for Smarties.

    How do you use Diskspd?


    Tip: When you run Microsoft Diskspd it will create a file or data set on the device or volume being tested that it will do its I/O to, make sure that you have enough disk space for what will be tested (e.g. if you are going to test 1TB you need to have more than 1TB of disk space free for use). Another tip is to speed up the initializing (e.g. when Diskspd creates the file that I/Os will be done to) run as administrator.

    Tip: In case you forgot, a couple of other useful Microsoft tools (besides Perfmon) for working with and displaying server storage I/O devices including disks (HDD and SSDs) are the commands "wmic diskdrive list [brief]" and "diskpart". With diskpart exercise caution as it can get you in trouble just as fast as it can get you out of trouble.

    You can view the Diskspd commands after installing the tool and from a Windows command prompt type:

    C:\Users\Username> Diskspd


    The above command will display Diskspd help and information about the commands as follows.

    Usage: diskspd [options] target1 [ target2 [ target3 …] ]
    
version 2.0.12 (2014/09/17)
    

    Available targets:
    
       file_path
    
       #
       :
    

    Available options:











    -?display usage information
    -a#[,#[…]]advanced CPU affinity – affinitize threads to CPUs provided after -a in a round-robin manner within current KGroup (CPU count starts with 0); the same CPU can be listed more than once and the number of CPUs can be different than the number of files or threads (cannot be used with -n)

    -ag

    		group affinity – affinitize threads in a round-robin manner across KGroups
    -b[K|M|G]block size in bytes/KB/MB/GB [default=64K]

    -B[K|M|G|b]

    		base file offset in bytes/KB/MB/GB/blocks [default=0] (offset from the beginning of the file)
    -c[K|M|G|b]create files of the given size. Size can be stated in bytes/KB/MB/GB/blocks

    -Ccool down time – duration of the test after measurements finished [default=0s].

    -DPrint IOPS standard deviations. The deviations are calculated for samples of duration . is given in milliseconds and the default value is 1000.
    -dduration (in seconds) to run test [default=10s]
    -f[K|M|G|b]

    		file size – this parameter can be used to use only the part of the file/disk/partition for example to test only the first sectors of disk
    -fropen file with the FILE_FLAG_RANDOM_ACCESS hint
    -fsopen file with the FILE_FLAG_SEQUENTIAL_SCAN hint
    -Ftotal number of threads (cannot be used with -t)
    -gthroughput per thread is throttled to given bytes per millisecond note that this can not be specified when using completion routines
    -hdisable both software and hardware caching
    -inumber of IOs (burst size) before thinking. must be specified with -j
    -jtime to think in ms before issuing a burst of IOs (burst size). must be specified with -i
    -ISet IO priority to . Available values are: 1-very low, 2-low, 3-normal (default)
    -lUse large pages for IO buffers

    -Lmeasure latency statistics
    -ndisable affinity (cannot be used with -a)
    -onumber of overlapped I/O requests per file per thread (1=synchronous I/O, unless more than 1 thread is specified with -F) [default=2]
    -pstart async (overlapped) I/O operations with the same offset (makes sense only with -o2 or grater)
    -Penable printing a progress dot after each completed I/O operations (counted separately by each thread) [default count=65536]
    -r[K|M|G|b]random I/O aligned to bytes (doesn’t make sense with -s). can be stated in bytes/KB/MB/GB/blocks [default access=sequential, default alignment=block size]
    -R

    		output format. Default is text.
    -s[K|M|G|b]stride size (offset between starting positions of subsequent I/O operations)
    -Sdisable OS caching
    -tnumber of threads per file (cannot be used with -F)
    -T[K|M|G|b]stride between I/O operations performed on the same file by different threads [default=0] (starting offset = base file offset + (thread number * ) it makes sense only with -t or -F
    -vverbose mode
    -wpercentage of write requests (-w and -w0 are equivalent). absence of this switch indicates 100% reads IMPORTANT: Your data will be destroyed without a warning
    -W

    		warm up time – duration of the test before measurements start [default=5s].
    -xuse completion routines instead of I/O Completion Ports
    -Xuse an XML file for configuring the workload. Cannot be used with other parameters.
    -zset random seed [default=0 if parameter not provided, GetTickCount() if value not provided]




     
    		Write buffers command options. By default, the write buffers are filled with a repeating pattern (0, 1, 2, …, 255, 0, 1, …)
    -Z

    		zero buffers used for write tests
    -Z[K|M|G|b]use a global buffer filled with random data as a source for write operations.
    -Z[K|M|G|b],

    		use a global buffer filled with data from as a source for write operations. If is smaller than , its content will be repeated multiple times in the buffer. By default, the write buffers are filled with a repeating pattern (0, 1, 2, …, 255, 0, 1, …)







     Synchronization command options
    -ys
    		signals event
    before starting the actual run (no warmup) (creates a notification event if does not exist)
    -yfsignals event after the actual run finishes (no cooldown) (creates a notification event if does not exist)
    -yrwaits on event before starting the run (including warmup) (creates a notification event if does not exist)
    -ypallows to stop the run when event is set; it also binds CTRL+C to this event (creates a notification event if does not exist)
    -yesets event and quits









    Event Tracing command options

    -epuse paged memory for NT Kernel Logger (by default it uses non-paged memory)
    -equse perf timer
    -esuse system timer (default)
    -ecuse cycle count
    -ePROCESSprocess start & end
    -eTHREADthread start & end
    -eIMAGE_LOADimage load
    -eDISK_IOphysical disk IO
    -eMEMORY_PAGE_FAULTSall page faults
    -eMEMORY_HARD_FAULTShard faults only
    -eNETWORK

    		TCP/IP, UDP/IP send & receive
    -eREGISTRYregistry calls



    
Examples:
    

    Create 8192KB file and run read test on it for 1 second:
    

      diskspd -c8192K -d1 testfile.dat
    

    Set block size to 4KB, create 2 threads per file, 32 overlapped (outstanding)
    
I/O operations per thread, disable all caching mechanisms and run block-aligned random
    
access read test lasting 10 seconds:
    

      diskspd -b4K -t2 -r -o32 -d10 -h testfile.dat
    

    Create two 1GB files, set block size to 4KB, create 2 threads per file, affinitize threads
    
to CPUs 0 and 1 (each file will have threads affinitized to both CPUs) and run read test
    
lasting 10 seconds:
    

      diskspd -c1G -b4K -t2 -d10 -a0,1 testfile1.dat testfile2.dat

    Where to learn more


    The following are related links to read more about servver (cloud, virtual and physical) storage I/O benchmarking tools, technologies and techniques.
    resource page

    Server and Storage I/O Benchmarking 101 for Smarties.

    Microsoft Diskspd download and Microsoft Diskspd overview (via Technet)

    I/O, I/O how well do you know about good or bad server and storage I/Os?

    Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I and Part II)

    Wrap up and summary, for now…


    This wraps up part-one of this two-part post taking a look at Microsoft Diskspd benchmark and workload generation tool. In part-two (here) of this post series we take a closer look including a test drive using Microsoft Diskspd.

    Ok, nuff said (for now)

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)

    twitter @storageio


    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    I/O, I/O how well do you know good bad ugly server storage I/O iops?

    How well do you know good bad ugly I/O iops?

    server storage i/o iops activity data infrastructure trends

    Updated 2/10/2018

    There are many different types of server storage I/O iops associated with various environments, applications and workloads. Some I/Os activity are iops, others are transactions per second (TPS), files or messages per time (hour, minute, second), gets, puts or other operations. The best IO is one you do not have to do.

    What about all the cloud, virtual, software defined and legacy based application that still need to do I/O?

    If no IO operation is the best IO, then the second best IO is the one that can be done as close to the application and processor as possible with the best locality of reference.

    Also keep in mind that aggregation (e.g. consolidation) can cause aggravation (server storage I/O performance bottlenecks).

    aggregation causes aggravation
    Example of aggregation (consolidation) causing aggravation (server storage i/o blender bottlenecks)

    And the third best?

    It’s the one that can be done in less time or at least cost or effect to the requesting application, which means moving further down the memory and storage stack.

    solving server storage i/o blender and other bottlenecks
    Leveraging flash SSD and cache technologies to find and fix server storage I/O bottlenecks

    On the other hand, any IOP regardless of if for block, file or object storage that involves some context is better than those without, particular involving metrics that matter (here, here and here [webinar] )

    Server Storage I/O optimization and effectiveness

    The problem with IO’s is that they are a basic operations to get data into and out of a computer or processor, so there’s no way to avoid all of them, unless you have a very large budget. Even if you have a large budget that can afford an all flash SSD solution, you may still meet bottlenecks or other barriers.

    IO’s require CPU or processor time and memory to set up and then process the results as well as IO and networking resources to move data too their destination or retrieve them from where they are stored. While IO’s cannot be eliminated, their impact can be greatly improved or optimized by, among other techniques, doing fewer of them via caching and by grouping reads or writes (pre-fetch, write-behind).

    server storage I/O STI and SUT

    Think of it this way: Instead of going on multiple errands, sometimes you can group multiple destinations together making for a shorter, more efficient trip. However, that optimization may also mean your drive will take longer. So, sometimes it makes sense to go on a couple of quick, short, low-latency trips instead of one larger one that takes half a day even as it accomplishes many tasks. Of course, how far you have to go on those trips (i.e., their locality) makes a difference about how many you can do in a given amount of time.

    Locality of reference (or proximity)

    What is locality of reference?

    This refers to how close (i.e., its place) data exists to where it is needed (being referenced) for use. For example, the best locality of reference in a computer would be registers in the processor core, ready to be acted on immediately. This would be followed by levels 1, 2, and 3 (L1, L2, and L3) onboard caches, followed by main memory, or DRAM. After that comes solid-state memory typically NAND flash either on PCIe cards or accessible on a direct attached storage (DAS), SAN, or NAS device. 

    server storage I/O locality of reference

    Even though a PCIe NAND flash card is close to the processor, there still remains the overhead of traversing the PCIe bus and associated drivers. To help offset that impact, PCIe cards use DRAM as cache or buffers for data along with meta or control information to further optimize and improve locality of reference. In other words, this information is used to help with cache hits, cache use, and cache effectiveness vs. simply boosting cache use.

    SSD to the rescue?

    What can you do the cut the impact of IO’s?

    There are many steps one can take, starting with establishing baseline performance and availability metrics.

    The metrics that matter include IOP’s, latency, bandwidth, and availability. Then, leverage metrics to gain insight into your application’s performance.

    Understand that IO’s are a fact of applications doing work (storing, retrieving, managing data) no matter whether systems are virtual, physical, or running up in the cloud. But it’s important to understand just what a bad IO is, along with its impact on performance. Try to identify those that are bad, and then find and fix the problem, either with software, application, or database changes. Perhaps you need to throw more software caching tools, hypervisors, or hardware at the problem. Hardware may include faster processors with more DRAM and faster internal busses.

    Leveraging local PCIe flash SSD cards for caching or as targets is another option.

    You may want to use storage systems or appliances that rely on intelligent caching and storage optimization capabilities to help with performance, availability, and capacity.

    Where to gain insight into your server storage I/O environment

    There are many tools that you can be used to gain insight into your server storage I/O environment across cloud, virtual, software defined and legacy as well as from different layers (e.g. applications, database, file systems, operating systems, hypervisors, server, storage, I/O networking). Many applications along with databases have either built-in or optional tools from their provider, third-party, or via other sources that can give information about work activity being done. Likewise there are tools to dig down deeper into the various data information infrastructure to see what is happening at the various layers as shown in the following figures.

    application storage I/O performance
    Gaining application and operating system level performance insight via different tools

    windows and linux storage I/O performance
    Insight and awareness via operating system tools on Windows and Linux

    In the above example, Spotlight on Windows (SoW) which you can download for free from Dell here along with Ubuntu utilities are shown, You could also use other tools to look at server storage I/O performance including Windows Perfmon among others.

    vmware server storage I/O
    Hypervisor performance using VMware ESXi / vsphere built-in tools

    vmware server storage I/O performance
    Using Visual ESXtop to dig deeper into virtual server storage I/O performance

    vmware server storage i/o cache
    Gaining insight into virtual server storage I/O cache performance

    Wrap up and summary

    There are many approaches to address (e.g. find and fix) vs. simply move or mask data center and server storage I/O bottlenecks. Having insight and awareness into how your environment along with applications is important to know to focus resources. Also keep in mind that a bit of flash SSD or DRAM cache in the applicable place can go along way while a lot of cache will also cost you cash. Even if you cant eliminate I/Os, look for ways to decrease their impact on your applications and systems.

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    >Keep in mind: SSD including flash and DRAM among others are in your future, the question is where, when, with what, how much and whose technology or packaging.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Revisiting RAID data protection remains relevant resource links

    Revisiting RAID data protection remains relevant and resources

    Storage I/O trends

    Updated 2/10/2018

    RAID data protection remains relevant including erasure codes (EC), local reconstruction codes (LRC) among other technologies. If RAID were really not relevant anymore (e.g. actually dead), why do some people spend so much time trying to convince others that it is dead or to use a different RAID level or enhanced RAID or beyond raid with related advanced approaches?

    When you hear RAID, what comes to mind?

    A legacy monolithic storage system that supports narrow 4, 5 or 6 drive wide stripe sets or a modern system support dozens of drives in a RAID group with different options?

    RAID means many things, likewise there are different implementations (hardware, software, systems, adapters, operating systems) with various functionality, some better than others.

    For example, which of the items in the following figure come to mind, or perhaps are new to your RAID vocabulary?

    RAID questions

    There are Many Variations of RAID Storage some for the enterprise, some for SMB, SOHO or consumer. Some have better performance than others, some have poor performance for example causing extra writes that lead to the perception that all parity based RAID do extra writes (some actually do write gathering and optimization).

    Some hardware and software implementations using WBC (write back cache) mirrored or battery backed-BBU along with being able to group writes together in memory (cache) to do full stripe writes. The result can be fewer back-end writes compared to other systems. Hence, not all RAID implementations in either hardware or software are the same. Likewise, just because a RAID definition shows a particular theoretical implementation approach does not mean all vendors have implemented it in that way.

    RAID is not a replacement for backup rather part of an overall approach to providing data availability and accessibility.

    data protection and durability

    What’s the best RAID level? The one that meets YOUR needs

    There are different RAID levels and implementations (hardware, software, controller, storage system, operating system, adapter among others) for various environments (enterprise, SME, SMB, SOHO, consumer) supporting primary, secondary, tertiary (backup/data protection, archiving).

    RAID comparison
    General RAID comparisons

    Thus one size or approach does fit all solutions, likewise RAID rules of thumbs or guides need context. Context means that a RAID rule or guide for consumer or SOHO or SMB might be different for enterprise and vise versa, not to mention on the type of storage system, number of drives, drive type and capacity among other factors.

    RAID comparison
    General basic RAID comparisons

    Thus the best RAID level is the one that meets your specific needs in your environment. What is best for one environment and application may be different from what is applicable to your needs.

    Key points and RAID considerations include:

    · Not all RAID implementations are the same, some are very much alive and evolving while others are in need of a rest or rewrite. So it is not the technology or techniques that are often the problem, rather how it is implemented and then deployed.

    · It may not be RAID that is dead, rather the solution that uses it, hence if you think a particular storage system, appliance, product or software is old and dead along with its RAID implementation, then just say that product or vendors solution is dead.

    · RAID can be implemented in hardware controllers, adapters or storage systems and appliances as well as via software and those have different features, capabilities or constraints.

    · Long or slow drive rebuilds are a reality with larger disk drives and parity-based approaches; however, you have options on how to balance performance, availability, capacity, and economics.

    · RAID can be single, dual or multiple parity or mirroring-based.

    · Erasure and other coding schemes leverage parity schemes and guess what umbrella parity schemes fall under.

    · RAID may not be cool, sexy or a fun topic and technology to talk about, however many trendy tools, solutions and services actually use some form or variation of RAID as part of their basic building blocks. This is an example of using new and old things in new ways to help each other do more without increasing complexity.

    ·  Even if you are not a fan of RAID and think it is old and dead, at least take a few minutes to learn more about what it is that you do not like to update your dead FUD.

    Wait, Isn’t RAID dead?

    There is some dead marketing that paints a broad picture that RAID is dead to prop up something new, which in some cases may be a derivative variation of parity RAID.

    data dispersal
    Data dispersal and durability

    RAID rebuild improving
    RAID continues to evolve with rapid rebuilds for some systems

    Otoh, there are some specific products, technologies, implementations that may be end of life or actually dead. Likewise what might be dead, dying or simply not in vogue are specific RAID implementations or packaging. Certainly there is a lot of buzz around object storage, cloud storage, forward error correction (FEC) and erasure coding including messages of how they cut RAID. Catch is that some object storage solutions are overlayed on top of lower level file systems that do things such as RAID 6, granted they are out of sight, out of mind.

    RAID comparison
    General RAID parity and erasure code/FEC comparisons

    Then there are advanced parity protection schemes which include FEC and erasure codes that while they are not your traditional RAID levels, they have characteristic including chunking or sharding data, spreading it out over multiple devices with multiple parity (or derivatives of parity) protection.

    Bottom line is that for some environments, different RAID levels may be more applicable and alive than for others.

    Via BizTech – How to Turn Storage Networks into Better Performers

    • Maintain Situational Awareness
    • Design for Performance and Availability
    • Determine Networked Server and Storage Patterns
    • Make Use of Applicable Technologies and Techniques

    If RAID is alive, what to do with it?

    If you are new to RAID, learn more about the past, present and future keeping mind context. Keeping context in mind means that there are different RAID levels and implementations for various environments. Not all RAID 0, 1, 1/0, 10, 2, 3, 4, 5, 6 or other variations (past, present and emerging) are the same for consumer vs. SOHO vs. SMB vs. SME vs. Enterprise, nor are the usage cases. Some need performance for reads, others for writes, some for high-capacity with low performance using hardware or software. RAID Rules of thumb are ok and useful, however keep them in context to what you are doing as well as using.

    What to do next?

    Take some time to learn, ask questions including what to use when, where, why and how as well as if an approach or recommendation are applicable to your needs. Check out the following links to read some extra perspectives about RAID and keep in mind, what might apply to enterprise may not be relevant for consumer or SMB and vise versa.

    Some advise needed on SSD’s and Raid (Via Spiceworks)
    RAID 5 URE Rebuild Means The Sky Is Falling (Via BenchmarkReview)
    Double drive failures in a RAID-10 configuration (Via SearchStorage)
    Industry Trends and Perspectives: RAID Rebuild Rates (Via StorageIOblog)
    RAID, IOPS and IO observations (Via StorageIOBlog)
    RAID Relevance Revisited (Via StorageIOBlog)
    HDDs Are Still Spinning (Rust Never Sleeps) (Via InfoStor)
    When and Where to Use NAND Flash SSD for Virtual Servers (Via TheVirtualizationPractice)
    What’s the best way to learn about RAID storage? (Via Spiceworks)
    Design considerations for the host local FVP architecture (Via Frank Denneman)
    Some basic RAID fundamentals and definitions (Via SearchStorage)
    Can RAID extend nand flash SSD life? (Via StorageIOBlog)
    I/O Performance Issues and Impacts on Time-Sensitive Applications (Via CMG)
    The original RAID white paper (PDF) that while over 20 years old, it provides a basis, foundation and some history by Katz, Gibson, Patterson et al
    Storage Interview Series (Via Infortrend)
    Different RAID methods (Via RAID Recovery Guide)
    A good RAID tutorial (Via TheGeekStuff)
    Basics of RAID explained (Via ZDNet)
    RAID and IOPs (Via VMware Communities)

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    What is my favorite or preferred RAID level?

    That depends, for some things its RAID 1, for others RAID 10 yet for others RAID 4, 5, 6 or DP and yet other situations could be a fit for RAID 0 or erasure codes and FEC. Instead of being focused on just one or two RAID levels as the solution for different problems, I prefer to look at the environment (consumer, SOHO, small or large SMB, SME, enterprise), type of usage (primary or secondary or data protection), performance characteristics, reads, writes, type and number of drives among other factors. What might be a fit for one environment would not be a fit for others, thus my preferred RAID level along with where implemented is the one that meets the given situation. However also keep in mind is tying RAID into part of an overall data protection strategy, remember, RAID is not a replacement for backup.

    What this all means

    Like other technologies that have been declared dead for years or decades, aka the Zombie technologies (e.g. dead yet still alive) RAID continues to be used while the technologies evolves. There are specific products, implementations or even RAID levels that have faded away, or are declining in some environments, yet alive in others. RAID and its variations are still alive, however how it is used or deployed in conjunction with other technologies also is evolving.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Book review: Rethinking Enterprise Storage by Microsoft/Storsimple Marc Farley

    Storage I/O trends

    Book review: Rethinking Enterprise Storage – A Hybrid Cloud Model by Marc Farley

    The O’Reilly @oreillymedia media folks (oops, excuse me, Microsoft Press) sent me out (that’s a disclosure btw) an real soft cover print copy of Rethinking Enterprise Storage – A Hybrid Cloud Model by Marc Farley aka @MicroFarley of Microsoft/Storsimple that features a forward by Martin Glassborow aka @Storagebod.

    Rethinking Enterprise Storage - A Hybrid Cloud Model

    Topics and themes covered in the book

    • Understanding scale storage architectures (hmm, great way of saying hybrid ;)
    • Rethinking data protection including disaster recovery (DR) best practices
    • Enhancing data protection using cloud snapshots beyond traditional backups
    • Deterministic thin recovery capabilities while dynamically expanding capacity to the cloud
    • Implement data footprint reduction (DFR) including archiving digital documents to the cloud
    • Insight and awareness into keep performance indicators along with various use cases

    Rethinking Enterprise Storage book Details

    Publisher: Microsoft Press
    Author: Marc Farley
    Paper back
    Features: Many diagrams, figures, index, glossary
    Pages: 101
    ISBN: 978-0-7356-7990-3
    Published: 2013
    MSRP: $9.99 USD

    Sample pages of rethinking enterprise storage
    One of the many books many figures on the right, on the left i needed to hold a page down ;)!

    What’s inside the book

    Make no mistake that this is a Microsoft and Storsimple themed book, however IMHO Marc (aka Farley) does a great job of making it more relevant than just another vendor product book (JAVPB). While it is a Microsoft focused book around enabling hybrid cloud storage for various applications, the premises presented could be adapted for other environments or implementations. The book at 101 pages including table of contents (TOC), index, appendix, glossary and other front matter is a very easy and fast read while providing more information or coverage than what might be found in a "Dummies" type themed book.

    Looking inside Rethinking Enterprise Storage by Marc Farley
    Start thinking outside the box (or cloud), imagine what you can do with a Hybrid cloud!

    Summary

    Overall I found the book to be good and not just because I know Marc or that the O’Reilly folks sent me a free copy (I had actually previously received the electronic ebook version), rather that it is timely and does a nice job of conveying the topic theme and setting up the conversation, time to rethink storage for enterprise and other environments. IMHO the question is not if hybrid cloud storage is in your future, rather when, where, why, for what, how, with whom and related conversations. While you can buy a copy of the book at various venues, it shouldn’t take a lot of effort to get your own printed soft cover copy, or an ebook version.

    Btw, here’s a pod cast discussion with Marc Farley from spring 2013 at SNW, as well as a link to a hybrid cloud and object storage post he did over at Microsoft Technet.

    To summarize and quote Marc Farley "Hey now…."

    Ok, nuff said

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2013 StorageIO and UnlimitedIO All Rights Reserved

    Posted on

    Cloud conversations: Has Nirvanix shutdown caused cloud confidence concerns?

    Storage I/O trends

    Cloud conversations: Has Nirvanix shutdown caused cloud confidence concerns?

    Recently seven plus year old cloud storage startup Nirvanix announced that they were finally shutting down and that customers should move their data.

    nirvanix customer message

    Nirvanix has also posted an announcement that they have established an agreement with IBM Softlayer (read about that acquisition here) to help customers migrate to those services as well as to those of Amazon Web Services (AWS), (read more about AWS in this primer here), Google and Microsoft Azure.

    Cloud customer concerns?

    With Nirvanix shutting down there has been plenty of articles, blog posts, twitter tweets and other conversations asking if Clouds are safe.

    Btw, here is a link to my ongoing poll where you can cast your vote on what you think about clouds.

    IMHO clouds can be safe if used in safe ways which includes knowing and addressing your concerns, not to mention following best practices, some of which pre-date the cloud era, sometimes by a few decades.

    Nirvanix Storm Clouds

    More on this in a moment, however lets touch base on Nirvanix and why I said they were finally shutting down.

    The reason I say finally shutting down is that there were plenty of early warning signs and storm clouds circling Nirvanix for a few years now.

    What I mean by this is that in their seven plus years of being in business, there have been more than a few CEO changes, something that is not unheard of.

    Likewise there have been some changes to their business model ranging from selling their software as a service to a solution to hosting among others, again, smart startups and establishes organizations will adapt over time.

    Nirvanix also invested heavily in marketing, public relations (PR) and analyst relations (AR) to generate buzz along with gaining endorsements as do most startups to get recognition, followings and investors if not real customers on board.

    In the case of Nirvanix, the indicator signs mentioned above also included what seemed like a semi-annual if not annual changing of CEOs, marketing and others tying into business model adjustments.

    cloud storage

    It was only a year or so ago that if you gauged a company health by the PR and AR news or activity and endorsements you would have believed Nirvanix was about to crush Amazon, Rackspace or many others, perhaps some actually did believe that, followed shortly there after by the abrupt departure of their then CEO and marketing team. Thus just as fast as Nirvanix seemed to be the phoenix rising in stardom their aura started to dim again, which could or should have been a warning sign.

    This is not to solo out Nirvanix, however given their penchant for marketing and now what appears to some as a sudden collapse or shutdown, they have also become a lightning rod of sort for clouds in general. Given all the hype and fud around clouds when something does happen the distract ors will be quick to jump or pile on to say things like "See, I told you, clouds are bad".

    Meanwhile the cloud cheerleaders may go into denial saying there are no problems or issues with clouds, or they may go back into a committee meeting to create a new stack, standard, API set marketing consortium alliance. ;) On the other hand, there are valid concerns with any technology including clouds that in general there are good implementations that can be used the wrong way, or questionable implementations and selections used in what seem like good ways that can go bad.

    This is not to say that clouds in general whether as a service, solution or product on a public, private or hybrid bases are any riskier than traditional hardware, software and services. Instead what this should be is a wake up call for people and organizations to review clouds citing their concerns along with revisiting what to do or can be done about them.

    Clouds: Being prepared

    Ben Woo of Neuralytix posted this question comment to one of the Linked In groups Collateral Considerations If You Were/Are A Nirvanix Customer which I posted some tips and recommendations including:

    1) If you have another copy of your data somewhere else (which you should btw), how will your data at Nirvanix be securely erased, and the storage it resides on be safely (and secure) decommissioned?

    2) if you do have another copy of your data elsewhere, how current is it, can you bring it up to date from various sources (including update from Nirvanix while they stay online)?

    3) Where will you move your data to short or near term, as well as long-term.

    4) What changes will you make to your procurement process for cloud services in the future to protect against situations like this happening to you?

    5) As part of your plan for putting data into the cloud, refine your strategy for getting it out, moving it to another service or place as well as having an alternate copy somewhere.

    Fwiw any data I put into a cloud service there is also another copy somewhere else which even though there is a cost, there is a benefit, The benefit is that ability to decide which to use if needed, as well as having a backup/spare copy.

    Storage I/O trends

    Cloud Concerns and Confidence

    As part of cloud procurement as services or products, the same proper due diligence should occur as if you were buying traditional hardware, software, networking or services. That includes checking out not only the technology, also the companies financial, business records, customer references (both good and not so good or bad ones) to gain confidence. Part of gaining that confidence also involves addressing ahead of time how you will get your data out of or back from that services if needed.

    Keep in mind that if your data is very important, are you going to keep it in just one place? For example I have data backed-up as well as archived to cloud providers, however I also have local copies either on-site or off.

    Likewise there is data I have local kept at alternate locations including cloud. Sure that is costly, however by not treating all of my data and applications the same, I’m able to balance those costs out, plus use cost advantages of different services as well as on-site to be effective. I may be spending no less on data protection, in fact I’m actually spending a bit more, however I also have more copies and versions of important data and in multiple locations. Data that is not changing often does not get protected as often, however there are multiple copies to meet different needs or threat risks.

    Storage I/O trends

    Don’t be scared of clouds, be prepared

    While some of the other smaller cloud storage vendors will see some new customers, I suspect that near to mid-term, it will be the larger, more established and well funded providers that gain the most from this current situation. Granted some customers are looking for alternatives to the mega cloud providers such as Amazon, Google, HP, IBM, Microsoft and Rackspace among others, however there are a long list of others some of which who are not so well-known that should be such as Centurylink/Savvis, Verizon/Terremark, Sungurd, Dimension Data, Peak, Bluehost, Carbonite, Mozy (owned by EMC), Xerox ACS, Evault (owned by Seagate) not to mention a long list of many others.

    Something to be aware of as part of doing your due diligence is determining who or what actually powers a particular cloud service. The larger providers such as Rackspace, Amazon, Microsoft, HP among others have their own infrastructure while some of the smaller service providers may in fact use one of the larger (or even smaller) providers as their real back-end. Hence understanding who is behind a particular cloud service is important to help decide the viability and stability of who it is you are subscribed to or working with.

    Something that I have said for the past couple of years and a theme of my book Cloud and Virtual Data Storage Networking (CRC Taylor & Francis) is do not be scared of clouds, however be ready, do your homework.

    This also means having cloud concerns is a good thing, again don’t be scared, however find what those concerns are along with if they are major or minor. From that list you can start to decide how or if they can be worked around, as well as be prepared ahead of time should you either need all of your cloud data back quickly, or should that service become un-available.

    Also when it comes to clouds, look beyond lowest cost or for free, likewise if something sounds too good to be true, perhaps it is. Instead look for value or how much do you get per what you spend including confidence in the service, service level agreements (SLA), security, and other items.

    Keep in mind, only you can prevent data loss either on-site or in the cloud, granted it is a shared responsibility (With a poll).

    Additional related cloud conversation items:
    Cloud conversations: AWS EBS Optimized Instances
    Poll: What Do You Think of IT Clouds?
    Cloud conversations: Gaining cloud confidence from insights into AWS outages
    Cloud conversations: confidence, certainty and confidentiality
    Cloud conversation, Thanks Gartner for saying what has been said
    Cloud conversations: AWS EBS, Glacier and S3 overview (Part III)
    Cloud conversations: Gaining cloud confidence from insights into AWS outages (Part II)
    Don’t Let Clouds Scare You – Be Prepared
    Everything Is Not Equal in the Datacenter, Part 3
    Amazon cloud storage options enhanced with Glacier
    What do VARs and Clouds as well as MSPs have in common?
    How many degrees separate you and your information?

    Ok, nuff said.

    Cheers
    Gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    Can we get a side of context with them IOPS server storage metrics?

    Can we get a side of context with them server storage metrics?

    Whats the best server storage I/O network metric or benchmark? It depends as there needs to be some context with them IOPS and other server storage I/O metrics that matter.

    There is an old saying that the best I/O (Input/Output) is the one that you do not have to do.

    In the meantime, let’s get a side of some context with them IOPS from vendors, marketers and their pundits who are tossing them around for server, storage and IO metrics that matter.

    Expanding the conversation, the need for more context

    The good news is that people are beginning to discuss storage beyond space capacity and cost per GByte, TByte or PByte for both DRAM or nand flash Solid State Devices (SSD), Hard Disk Drives (HDD) along with Hybrid HDD (HHDD) and Solid State Hybrid Drive (SSHD) based solutions. This applies to traditional enterprise or SMB IT data center with physical, virtual or cloud based infrastructures.

    hdd and ssd iops

    This is good because it expands the conversation beyond just cost for space capacity into other aspects including performance (IOPS, latency, bandwidth) for various workload scenarios along with availability, energy effective and management.

    Adding a side of context

    The catch is that IOPS while part of the equation are just one aspect of performance and by themselves without context, may have little meaning if not misleading in some situations.

    Granted it can be entertaining, fun to talk about or simply make good press copy for a million IOPS. IOPS vary in size depending on the type of work being done, not to mention reads or writes, random and sequential which also have a bearing on data throughout or bandwidth (Mbytes per second) along with response time. Not to mention block, file, object or blob as well as table.

    However, are those million IOP’s applicable to your environment or needs?

    Likewise, what do those million or more IOPS represent about type of work being done? For example, are they small 64 byte or large 64 Kbyte sized, random or sequential, cached reads or lazy writes (deferred or buffered) on a SSD or HDD?

    How about the response time or latency for achieving them IOPS?

    In other words, what is the context of those metrics and why do they matter?

    storage i/o iops
    Click on image to view more metrics that matter including IOP’s for HDD and SSD’s

    Metrics that matter give context for example IO sizes closer to what your real needs are, reads and writes, mixed workloads, random or sequential, sustained or bursty, in other words, real world reflective.

    As with any benchmark take them with a grain (or more) of salt, they key is use them as an indicator then align to your needs. The tool or technology should work for you, not the other way around.

    Here are some examples of context that can be added to help make IOP’s and other metrics matter:

    • What is the IOP size, are they 512 byte (or smaller) vs. 4K bytes (or larger)?
    • Are they reads, writes, random, sequential or mixed and what percentage?
    • How was the storage configured including RAID, replication, erasure or dispersal codes?
    • Then there is the latency or response time and IO queue depths for the given number of IOPS.
    • Let us not forget if the storage systems (and servers) were busy with other work or not.
    • If there is a cost per IOP, is that list price or discount (hint, if discount start negotiations from there)
    • What was the number of threads or workers, along with how many servers?
    • What tool was used, its configuration, as well as raw or cooked (aka file system) IO?
    • Was the IOP’s number with one worker or multiple workers on a single or multiple servers?
    • Did the IOP’s number come from a single storage system or total of multiple systems?
    • Fast storage needs fast serves and networks, what was their configuration?
    • Was the performance a short burst, or long sustained period?
    • What was the size of the test data used; did it all fit into cache?
    • Were short stroking for IOPS or long stroking for bandwidth techniques used?
    • Data footprint reduction (DFR) techniques (thin provisioned, compression or dedupe) used?
    • Were write data committed synchronously to storage, or deferred (aka lazy writes used)?

    The above are just a sampling and not all may be relevant to your particular needs, however they help to put IOP’s into more contexts. Another consideration around IOPS are the configuration of the environment, from an actual running application using some measurement tool, or are they generated from a workload tool such as IOmeter, IOrate, VDbench among others.

    Sure, there are more contexts and information that would be interesting as well, however learning to walk before running will help prevent falling down.

    Storage I/O trends

    Does size or age of vendors make a difference when it comes to context?

    Some vendors are doing a good job of going for out of this world record-setting marketing hero numbers.

    Meanwhile other vendors are doing a good job of adding context to their IOP or response time or bandwidth among other metrics that matter. There is a mix of startup and established that give context with their IOP’s or other metrics, likewise size or age does not seem to matter for those who lack context.

    Some vendors may not offer metrics or information publicly, so fine, go under NDA to learn more and see if the results are applicable to your environments.

    Likewise, if they do not want to provide the context, then ask some tough yet fair questions to decide if their solution is applicable for your needs.

    Storage I/O trends

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    What this means is let us start putting and asking for metrics that matter such as IOP’s with context.

    If you have a great IOP metric, if you want it to matter than include some context such as what size (e.g. 4K, 8K, 16K, 32K, etc.), percentage of reads vs. writes, latency or response time, random or sequential.

    IMHO the most interesting or applicable metrics that matter are those relevant to your environment and application. For example if your main application that needs SSD does about 75% reads (random) and 25% writes (sequential) with an average size of 32K, while fun to hear about, how relevant is a million 64 byte read IOPS? Likewise when looking at IOPS, pay attention to the latency, particular if SSD or performance is your main concern.

    Get in the habit of asking or telling vendors or their surrogates to provide some context with them metrics if you want them to matter.

    So how about some context around them IOP’s (or latency and bandwidth or availability for that matter)?

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Part II: How many IOPS can a HDD HHDD SSD do with VMware?

    How many IOPS can a HDD HHDD SSD do with VMware?

    server storage data infrastructure i/o iop hdd ssd trends

    Updated 2/10/2018

    This is the second post of a two-part series looking at storage performance, specifically in the context of drive or device (e.g. mediums) characteristics of How many IOPS can a HDD HHDD SSD do with VMware. In the first post the focus was around putting some context around drive or device performance with the second part looking at some workload characteristics (e.g. benchmarks).

    A common question is how many IOPS (IO Operations Per Second) can a storage device or system do?

    The answer is or should be it depends.

    Here are some examples to give you some more insight.

    For example, the following shows how IOPS vary by changing the percent of reads, writes, random and sequential for a 4K (4,096 bytes or 4 KBytes) IO size with each test step (4 minutes each).

    IO Size for test
    Workload Pattern of test
    Avg. Resp (R+W) ms
    Avg. IOP Sec (R+W)
    Bandwidth KB Sec (R+W)
    4KB
    100% Seq 100% Read
    0.0
    29,736
    118,944
    4KB
    60% Seq 100% Read
    4.2
    236
    947
    4KB
    30% Seq 100% Read
    7.1
    140
    563
    4KB
    0% Seq 100% Read
    10.0
    100
    400
    4KB
    100% Seq 60% Read
    3.4
    293
    1,174
    4KB
    60% Seq 60% Read
    7.2
    138
    554
    4KB
    30% Seq 60% Read
    9.1
    109
    439
    4KB
    0% Seq 60% Read
    10.9
    91
    366
    4KB
    100% Seq 30% Read
    5.9
    168
    675
    4KB
    60% Seq 30% Read
    9.1
    109
    439
    4KB
    30% Seq 30% Read
    10.7
    93
    373
    4KB
    0% Seq 30% Read
    11.5
    86
    346
    4KB
    100% Seq 0% Read
    8.4
    118
    474
    4KB
    60% Seq 0% Read
    13.0
    76
    307
    4KB
    30% Seq 0% Read
    11.6
    86
    344
    4KB
    0% Seq 0% Read
    12.1
    82
    330

    Dell/Western Digital (WD) 1TB 7200 RPM SATA HDD (Raw IO) thread count 1 4K IO size

    In the above example the drive is a 1TB 7200 RPM 3.5 inch Dell (Western Digital) 3Gb SATA device doing raw (non file system) IO. Note the high IOP rate with 100 percent sequential reads and a small IO size which might be a result of locality of reference due to drive level cache or buffering.

    Some drives have larger buffers than others from a couple to 16MB (or more) of DRAM that can be used for read ahead caching. Note that this level of cache is independent of a storage system, RAID adapter or controller or other forms and levels of buffering.

    Does this mean you can expect or plan on getting those levels of performance?

    I would not make that assumption, and thus this serves as an example of using metrics like these in the proper context.

    Building off of the previous example, the following is using the same drive however with a 16K IO size.

    IO Size for test
    Workload Pattern of test
    Avg. Resp (R+W) ms
    Avg. IOP Sec (R+W)
    Bandwidth KB Sec (R+W)
    16KB
    100% Seq 100% Read
    0.1
    7,658
    122,537
    16KB
    60% Seq 100% Read
    4.7
    210
    3,370
    16KB
    30% Seq 100% Read
    7.7
    130
    2,080
    16KB
    0% Seq 100% Read
    10.1
    98
    1,580
    16KB
    100% Seq 60% Read
    3.5
    282
    4,522
    16KB
    60% Seq 60% Read
    7.7
    130
    2,090
    16KB
    30% Seq 60% Read
    9.3
    107
    1,715
    16KB
    0% Seq 60% Read
    11.1
    90
    1,443
    16KB
    100% Seq 30% Read
    6.0
    165
    2,644
    16KB
    60% Seq 30% Read
    9.2
    109
    1,745
    16KB
    30% Seq 30% Read
    11.0
    90
    1,450
    16KB
    0% Seq 30% Read
    11.7
    85
    1,364
    16KB
    100% Seq 0% Read
    8.5
    117
    1,874
    16KB
    60% Seq 0% Read
    10.9
    92
    1,472
    16KB
    30% Seq 0% Read
    11.8
    84
    1,353
    16KB
    0% Seq 0% Read
    12.2
    81
    1,310

    Dell/Western Digital (WD) 1TB 7200 RPM SATA HDD (Raw IO) thread count 1 16K IO size

    The previous two examples are excerpts of a series of workload simulation tests (ok, you can call them benchmarks) that I have done to collect information, as well as try some different things out.

    The following is an example of the summary for each test output that includes the IO size, workload pattern (reads, writes, random, sequential), duration for each workload step, totals for reads and writes, along with averages including IOP’s, bandwidth and latency or response time.

    disk iops

    Want to see more numbers, speeds and feeds, check out the following table which will be updated with extra results as they become available.

    Device
    Vendor
    Make

    Model

    Form Factor
    		Capacity
    Interface
    RPM Speed
    Raw
    Test Result
    HDD
    HGST
    Desktop
    HK250-160
    2.5
    160GB
    SATA
    5.4K
    HDD
    Seagate
    Mobile
    ST2000LM003
    2.5
    2TB
    SATA
    5.4K
    HDD
    Fujitsu
    Desktop
    MHWZ160BH
    2.5
    160GB
    SATA
    7.2K
    HDD
    Seagate
    Momentus
    ST9160823AS
    2.5
    160GB
    SATA
    7.2K
    HDD
    Seagate
    MomentusXT
    ST95005620AS
    2.5
    500GB
    SATA
    7.2K(1)
    HDD
    Seagate
    Barracuda
    ST3500320AS
    3.5
    500GB
    SATA
    7.2K
    HDD
    WD/Dell
    Enterprise
    WD1003FBYX
    3.5
    1TB
    SATA
    7.2K
    HDD
    Seagate
    Barracuda
    ST3000DM01
    3.5
    3TB
    SATA
    7.2K
    HDD
    Seagate
    Desktop
    ST4000DM000
    3.5
    4TB
    SATA
    HDD
    HDD
    Seagate
    Capacity
    ST6000NM00
    3.5
    6TB
    SATA
    HDD
    HDD
    Seagate
    Capacity
    ST6000NM00
    3.5
    6TB
    12GSAS
    HDD
    HDD
    Seagate
    Savio 10K.3
    ST9300603SS
    2.5
    300GB
    SAS
    10K
    HDD
    Seagate
    Cheetah
    ST3146855SS
    3.5
    146GB
    SAS
    15K
    HDD
    Seagate
    Savio 15K.2
    ST9146852SS
    2.5
    146GB
    SAS
    15K
    HDD
    Seagate
    Ent. 15K
    ST600MP0003
    2.5
    600GB
    SAS
    15K
    SSHD
    Seagate
    Ent. Turbo
    ST600MX0004
    2.5
    600GB
    SAS
    SSHD
    SSD
    Samsung
    840 PRo
    MZ-7PD256
    2.5
    256GB
    SATA
    SSD
    HDD
    Seagate
    600 SSD
    ST480HM000
    2.5
    480GB
    SATA
    SSD
    SSD
    Seagate
    1200 SSD
    ST400FM0073
    2.5
    400GB
    12GSAS
    SSD

    Performance characteristics 1 worker (thread count) for RAW IO (non-file system)

    Note: (1) Seagate Momentus XT is a Hybrid Hard Disk Drive (HHDD) based on a 7.2K 2.5 HDD with SLC nand flash integrated for read buffer in addition to normal DRAM buffer. This model is a XT I (4GB SLC nand flash), may add an XT II (8GB SLC nand flash) at some future time.

    As a starting point, these results are raw IO with file system based information to be added soon along with more devices. These results are for tests with one worker or thread count, other results will be added with such as 16 workers or thread counts to show how those differ.

    The above results include all reads, all writes, mix of reads and writes, along with all random, sequential and mixed for each IO size. IO sizes include 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1024K and 2048K. As with any workload simulation, benchmark or comparison test, take these results with a grain of salt as your mileage can and will vary. For example you will see some what I consider very high IO rates with sequential reads even without file system buffering. These results might be due to locality of reference of IO’s being resolved out of the drives DRAM cache (read ahead) which vary in size for different devices. Use the vendor model numbers in the table above to check the manufactures specs on drive DRAM and other attributes.

    If you are used to seeing 4K or 8K and wonder why anybody would be interested in some of the larger sizes take a look at big fast data or cloud and object storage. For some of those applications 2048K may not seem all that big. Likewise if you are used to the larger sizes, there are still applications doing smaller sizes. Sorry for those who like 512 byte or smaller IO’s as they are not included. Note that for all of these unless indicated a 512 byte standard sector or drive format is used as opposed to emerging Advanced Format (AF) 4KB sector or block size. Watch for some more drive and device types to be added to the above, along with results for more workers or thread counts, along with file system and other scenarios.

    Using VMware as part of a Server, Storage and IO (aka StorageIO) test platform

    vmware vexpert

    The above performance results were generated on Ubuntu 12.04 (since upgraded to 14.04 which was hosted on a VMware vSphere 5.1 (upgraded to 5.5U2) purchased version (you can get the ESXi free version here) with vCenter enabled system. I also have VMware workstation installed on some of my Windows-based laptops for doing preliminary testing of scripts and other activity prior to running them on the larger server-based VMware environment. Other VMware tools include vCenter Converter, vSphere Client and CLI. Note that other guest virtual machines (VMs) were idle during the tests (e.g. other guest VMs were quiet). You may experience different results if you ran Ubuntu native on a physical machine or with different adapters, processors and device configurations among many other variables (that was a disclaimer btw ;) ).

    Storage I/O trends

    All of the devices (HDD, HHDD, SSD’s including those not shown or published yet) were Raw Device Mapped (RDM) to the Ubuntu VM bypassing VMware file system.

    Example of creating an RDM for local SAS or SATA direct attached device.

    vmkfstools -z /vmfs/devices/disks/naa.600605b0005f125018e923064cc17e7c /vmfs/volumes/dat1/RDM_ST1500Z110S6M5.vmdk

    The above uses the drives address (find by doing a ls -l /dev/disks via VMware shell command line) to then create a vmdk container stored in a dat. Note that the RDM being created does not actually store data in the .vmdk, it’s there for VMware management operations.

    If you are not familiar with how to create a RDM of a local SAS or SATA device, check out this post to learn how.This is important to note in that while VMware was used as a platform to support the guest operating systems (e.g. Ubuntu or Windows), the real devices are not being mapped through or via VMware virtual drives.

    vmware iops

    The above shows examples of RDM SAS and SATA devices along with other VMware devices and dats. In the next figure is an example of a workload being run in the test environment.

    vmware iops

    One of the advantages of using VMware (or other hypervisor) with RDM’s is that I can quickly define via software commands where a device gets attached to different operating systems (e.g. the other aspect of software defined storage). This means that after a test run, I can quickly simply shutdown Ubuntu, remove the RDM device from that guests settings, move the device just tested to a Windows guest if needed and restart those VMs. All of that from where ever I happen to be working from without physically changing things or dealing with multi-boot or cabling issues.

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    So how many IOPs can a device do?

    That depends, however have a look at the above information and results.

    Check back from time to time here to see what is new or has been added including more drives, devices and other related themes.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    How many I/O iops can flash SSD or HDD do?

    How many i/o iops can flash ssd or hdd do with vmware?

    sddc data infrastructure Storage I/O ssd trends

    Updated 2/10/2018

    A common question I run across is how many I/O iopsS can flash SSD or HDD storage device or system do or give.

    The answer is or should be it depends.

    This is the first of a two-part series looking at storage performance, and in context specifically around drive or device (e.g. mediums) characteristics across HDD, HHDD and SSD that can be found in cloud, virtual, and legacy environments. In this first part the focus is around putting some context around drive or device performance with the second part looking at some workload characteristics (e.g. benchmarks).

    What about cloud, tape summit resources, storage systems or appliance?

    Lets leave those for a different discussion at another time.

    Getting started

    Part of my interest in tools, metrics that matter, measurements, analyst, forecasting ties back to having been a server, storage and IO performance and capacity planning analyst when I worked in IT. Another aspect ties back to also having been a sys admin as well as business applications developer when on the IT customer side of things. This was followed by switching over to the vendor world involved with among other things competitive positioning, customer design configuration, validation, simulation and benchmarking HDD and SSD based solutions (e.g. life before becoming an analyst and advisory consultant).

    Btw, if you happen to be interested in learn more about server, storage and IO performance and capacity planning, check out my first book Resilient Storage Networks (Elsevier) that has a bit of information on it. There is also coverage of metrics and planning in my two other books The Green and Virtual Data Center (CRC Press) and Cloud and Virtual Data Storage Networking (CRC Press). I have some copies of Resilient Storage Networks available at a special reader or viewer rate (essentially shipping and handling). If interested drop me a note and can fill you in on the details.

    There are many rules of thumb (RUT) when it comes to metrics that matter such as IOPS, some that are older while others may be guess or measured in different ways. However the answer is that it depends on many things ranging from if a standalone hard disk drive (HDD), Hybrid HDD (HHDD), Solid State Device (SSD) or if attached to a storage system, appliance, or RAID adapter card among others.

    Taking a step back, the big picture

    hdd image
    Various HDD, HHDD and SSD’s

    Server, storage and I/O performance and benchmark fundamentals

    Even if just looking at a HDD, there are many variables ranging from the rotational speed or Revolutions Per Minute (RPM), interface including 1.5Gb, 3.0Gb, 6Gb or 12Gb SAS or SATA or 4Gb Fibre Channel. If simply using a RUT or number based on RPM can cause issues particular with 2.5 vs. 3.5 or enterprise and desktop. For example, some current generation 10K 2.5 HDD can deliver the same or better performance than an older generation 3.5 15K. Other drive factors (see this link for HDD fundamentals) including physical size such as 3.5 inch or 2.5 inch small form factor (SFF), enterprise or desktop or consumer, amount of drive level cache (DRAM). Space capacity of a drive can also have an impact such as if all or just a portion of a large or small capacity devices is used. Not to mention what the drive is attached to ranging from in internal SAS or SATA drive bay, USB port, or a HBA or RAID adapter card or in a storage system.

    disk iops
    HDD fundamentals

    How about benchmark and performance for marketing or comparison tricks including delayed, deferred or asynchronous writes vs. synchronous or actually committed data to devices? Lets not forget about short stroking (only using a portion of a drive for better IOP’s) or even long stroking (to get better bandwidth leveraging spiral transfers) among others.

    Almost forgot, there are also thick, standard, thin and ultra thin drives in 2.5 and 3.5 inch form factors. What’s the difference? The number of platters and read write heads. Look at the following image showing various thickness 2.5 inch drives that have various numbers of platters to increase space capacity in a given density. Want to take a wild guess as to which one has the most space capacity in a given footprint? Also want to guess which type I use for removable disk based archives along with for onsite disk based backup targets (compliments my offsite cloud backups)?

    types of disks
    Thick, thin and ultra thin devices

    Beyond physical and configuration items, then there are logical configuration including the type of workload, large or small IOPS, random, sequential, reads, writes or mixed (various random, sequential, read, write, large and small IO). Other considerations include file system or raw device, number of workers or concurrent IO threads, size of the target storage space area to decide impact of any locality of reference or buffering. Some other items include how long the test or workload simulation ran for, was the device new or worn in before use among other items.

    Tools and the performance toolbox

    Then there are the various tools for generating IO’s or workloads along with recording metrics such as reads, writes, response time and other information. Some examples (mix of free or for fee) include Bonnie, Iometer, Iorate, IOzone, Vdbench, TPC, SPC, Microsoft ESRP, SPEC and netmist, Swifttest, Vmark, DVDstore and PCmark 7 among many others. Some are focused just on the storage system and IO path while others are application specific thus exercising servers, storage and IO paths.

    performance tools
    Server, storage and IO performance toolbox

    Having used Iometer since the late 90s, it has its place and is popular given its ease of use. Iometer is also long in the tooth and has its limits including not much if any new development, never the less, I have it in the toolbox. I also have Futremark PCmark 7 (full version) which turns out has some interesting abilities to do more than exercise an entire Windows PC. For example PCmark can use a secondary drive for doing IO to.

    PCmark can be handy for spinning up with VMware (or other tools) lots of virtual Windows systems pointing to a NAS or other shared storage device doing real world type activity. Something that could be handy for testing or stressing virtual desktop infrastructures (VDI) along with other storage systems, servers and solutions. I also have Vdbench among others tools in the toolbox including Iorate which was used to drive the workloads shown below.

    What I look for in a tool are how extensible are the scripting capabilities to define various workloads along with capabilities of the test engine. A nice GUI is handy which makes Iometer popular and yes there are script capabilities with Iometer. That is also where Iometer is long in the tooth compared to some of the newer generation of tools that have more emphasis on extensibility vs. ease of use interfaces. This also assumes knowing what workloads to generate vs. simply kicking off some IOPs using default settings to see what happens.

    Another handy tool is for recording what’s going on with a running system including IO’s, reads, writes, bandwidth or transfers, random and sequential among other things. This is where when needed I turn to something like HiMon from HyperIO, if you have not tried it, get in touch with Tom West over at HyperIO and tell him StorageIO sent you to get a demo or trial. HiMon is what I used for doing start, stop and boot among other testing being able to see IO’s at the Windows file system level (or below) including very early in the boot or shutdown phase.

    Here is a link to some other things I did awhile back with HiMon to profile some Windows and VDI activity test profiling.

    What’s the best tool or benchmark or workload generator?

    The one that meets your needs, usually your applications or something as close as possible to it.

    disk iops
    Various 2.5 and 3.5 inch HDD, HHDD, SSD with different performance

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    That depends, however continue reading part II of this series to see some results for various types of drives and workloads.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.