How many I/O iops can flash SSD or HDD do?

How many i/o iops can flash ssd or hdd do with vmware?

sddc data infrastructure Storage I/O ssd trends

Updated 2/10/2018

A common question I run across is how many I/O iopsS can flash SSD or HDD storage device or system do or give.

The answer is or should be it depends.

This is the first of a two-part series looking at storage performance, and in context specifically around drive or device (e.g. mediums) characteristics across HDD, HHDD and SSD that can be found in cloud, virtual, and legacy environments. In this first part the focus is around putting some context around drive or device performance with the second part looking at some workload characteristics (e.g. benchmarks).

What about cloud, tape summit resources, storage systems or appliance?

Lets leave those for a different discussion at another time.

Getting started

Part of my interest in tools, metrics that matter, measurements, analyst, forecasting ties back to having been a server, storage and IO performance and capacity planning analyst when I worked in IT. Another aspect ties back to also having been a sys admin as well as business applications developer when on the IT customer side of things. This was followed by switching over to the vendor world involved with among other things competitive positioning, customer design configuration, validation, simulation and benchmarking HDD and SSD based solutions (e.g. life before becoming an analyst and advisory consultant).

Btw, if you happen to be interested in learn more about server, storage and IO performance and capacity planning, check out my first book Resilient Storage Networks (Elsevier) that has a bit of information on it. There is also coverage of metrics and planning in my two other books The Green and Virtual Data Center (CRC Press) and Cloud and Virtual Data Storage Networking (CRC Press). I have some copies of Resilient Storage Networks available at a special reader or viewer rate (essentially shipping and handling). If interested drop me a note and can fill you in on the details.

There are many rules of thumb (RUT) when it comes to metrics that matter such as IOPS, some that are older while others may be guess or measured in different ways. However the answer is that it depends on many things ranging from if a standalone hard disk drive (HDD), Hybrid HDD (HHDD), Solid State Device (SSD) or if attached to a storage system, appliance, or RAID adapter card among others.

Taking a step back, the big picture

hdd image
Various HDD, HHDD and SSD’s

Server, storage and I/O performance and benchmark fundamentals

Even if just looking at a HDD, there are many variables ranging from the rotational speed or Revolutions Per Minute (RPM), interface including 1.5Gb, 3.0Gb, 6Gb or 12Gb SAS or SATA or 4Gb Fibre Channel. If simply using a RUT or number based on RPM can cause issues particular with 2.5 vs. 3.5 or enterprise and desktop. For example, some current generation 10K 2.5 HDD can deliver the same or better performance than an older generation 3.5 15K. Other drive factors (see this link for HDD fundamentals) including physical size such as 3.5 inch or 2.5 inch small form factor (SFF), enterprise or desktop or consumer, amount of drive level cache (DRAM). Space capacity of a drive can also have an impact such as if all or just a portion of a large or small capacity devices is used. Not to mention what the drive is attached to ranging from in internal SAS or SATA drive bay, USB port, or a HBA or RAID adapter card or in a storage system.

disk iops
HDD fundamentals

How about benchmark and performance for marketing or comparison tricks including delayed, deferred or asynchronous writes vs. synchronous or actually committed data to devices? Lets not forget about short stroking (only using a portion of a drive for better IOP’s) or even long stroking (to get better bandwidth leveraging spiral transfers) among others.

Almost forgot, there are also thick, standard, thin and ultra thin drives in 2.5 and 3.5 inch form factors. What’s the difference? The number of platters and read write heads. Look at the following image showing various thickness 2.5 inch drives that have various numbers of platters to increase space capacity in a given density. Want to take a wild guess as to which one has the most space capacity in a given footprint? Also want to guess which type I use for removable disk based archives along with for onsite disk based backup targets (compliments my offsite cloud backups)?

types of disks
Thick, thin and ultra thin devices

Beyond physical and configuration items, then there are logical configuration including the type of workload, large or small IOPS, random, sequential, reads, writes or mixed (various random, sequential, read, write, large and small IO). Other considerations include file system or raw device, number of workers or concurrent IO threads, size of the target storage space area to decide impact of any locality of reference or buffering. Some other items include how long the test or workload simulation ran for, was the device new or worn in before use among other items.

Tools and the performance toolbox

Then there are the various tools for generating IO’s or workloads along with recording metrics such as reads, writes, response time and other information. Some examples (mix of free or for fee) include Bonnie, Iometer, Iorate, IOzone, Vdbench, TPC, SPC, Microsoft ESRP, SPEC and netmist, Swifttest, Vmark, DVDstore and PCmark 7 among many others. Some are focused just on the storage system and IO path while others are application specific thus exercising servers, storage and IO paths.

performance tools
Server, storage and IO performance toolbox

Having used Iometer since the late 90s, it has its place and is popular given its ease of use. Iometer is also long in the tooth and has its limits including not much if any new development, never the less, I have it in the toolbox. I also have Futremark PCmark 7 (full version) which turns out has some interesting abilities to do more than exercise an entire Windows PC. For example PCmark can use a secondary drive for doing IO to.

PCmark can be handy for spinning up with VMware (or other tools) lots of virtual Windows systems pointing to a NAS or other shared storage device doing real world type activity. Something that could be handy for testing or stressing virtual desktop infrastructures (VDI) along with other storage systems, servers and solutions. I also have Vdbench among others tools in the toolbox including Iorate which was used to drive the workloads shown below.

What I look for in a tool are how extensible are the scripting capabilities to define various workloads along with capabilities of the test engine. A nice GUI is handy which makes Iometer popular and yes there are script capabilities with Iometer. That is also where Iometer is long in the tooth compared to some of the newer generation of tools that have more emphasis on extensibility vs. ease of use interfaces. This also assumes knowing what workloads to generate vs. simply kicking off some IOPs using default settings to see what happens.

Another handy tool is for recording what’s going on with a running system including IO’s, reads, writes, bandwidth or transfers, random and sequential among other things. This is where when needed I turn to something like HiMon from HyperIO, if you have not tried it, get in touch with Tom West over at HyperIO and tell him StorageIO sent you to get a demo or trial. HiMon is what I used for doing start, stop and boot among other testing being able to see IO’s at the Windows file system level (or below) including very early in the boot or shutdown phase.

Here is a link to some other things I did awhile back with HiMon to profile some Windows and VDI activity test profiling.

What’s the best tool or benchmark or workload generator?

The one that meets your needs, usually your applications or something as close as possible to it.

disk iops
Various 2.5 and 3.5 inch HDD, HHDD, SSD with different performance

Where To Learn More

View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

That depends, however continue reading part II of this series to see some results for various types of drives and workloads.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Welcome to the Cloud Bulk Object Storage Resources Center

Updated 8/31/19

Cloud Bulk Big Data Software Defined Object Storage Resources

server storage I/O trends Object Storage resources

Welcome to the Cloud, Big Data, Software Defined, Bulk and Object Storage Resources Center Page objectstoragecenter.com.

This object storage resources, along with software defined, cloud, bulk, and scale-out storage page is part of the server StorageIOblog microsite collection of resources. Software-defined, Bulk, Cloud and Object Storage exist to support expanding and diverse application data demands.

Other related resources include:

  • Software Defined, Cloud, Bulk and Object Storage Fundamentals
  • Software Defined Data Infrastructure Essentials book (CRC Press)
  • Cloud, Software Defined, Scale-Out, Object Storage News Trends
  •  Object storage SDDC SDDI
    Via Software Defined Data Infrastructure Essentials (CRC Press 2017)

    Bulk, Cloud, Object Storage Solutions and Services

    There are various types of cloud, bulk, and object storage including public services such as Amazon Web Services (AWS) Simple Storage Service (S3), Backblaze, Google, Microsoft Azure, IBM Softlayer, Rackspace among many others. There are also solutions for hybrid and private deployment from Cisco, Cloudian, CTERA, Cray, DDN, Dell EMC, Elastifile, Fujitsu, Vantera/HDS, HPE, Hedvig, Huawei, IBM, NetApp, Noobaa, OpenIO, OpenStack, Quantum, Rackspace, Rozo, Scality, Spectra, Storpool, StorageCraft, Suse, Swift, Virtuozzo, WekaIO, WD, among many others.

    Bulk Cloud Object storage SDDC SDDI
    Via Software Defined Data Infrastructure Essentials (CRC Press 2017)

    Cloud products and services among others, along with associated data infrastructures including object storage, file systems, repositories and access methods are at the center of bulk, big data, big bandwidth and little data initiatives on a public, private, hybrid and community basis. After all, not everything is the same in cloud, virtual and traditional data centers or information factories from active data to in-active deep digital archiving.

    Object Context Matters

    Before discussing Object Storage lets take a step back and look at some context that can clarify some confusion around the term object. The word object has many different meanings and context, both inside of the IT world as well as outside. Context matters with the term object such as a verb being a thing that can be seen or touched as well as a person or thing of action or feeling directed towards.

    Besides a person, place or physical thing, an object can be a software-defined data structure that describes something. For example, a database record describing somebody’s contact or banking information, or a file descriptor with name, index ID, date and time stamps, permissions and access control lists along with other attributes or metadata. Another example is an object or blob stored in a cloud or object storage system repository, as well as an item in a hypervisor, operating system, container image or other application.

    Besides being a verb, an object can also be a noun such as disapproval or disagreement with something or someone. From an IT context perspective, an object can also refer to a programming method (e.g. object-oriented programming [oop], or Java [among other environments] objects and classes) and systems development in addition to describing entities with data structures.

    In other words, a data structure describes an object that can be a simple variable, constant, complex descriptor of something being processed by a program, as well as a function or unit of work. There are also objects unique or with context to specific environments besides Java or databases, operating systems, hypervisors, file systems, cloud and other things.

    The Need For Bulk, Cloud and Object Storage

    There is no such thing as an information recession with more data being generated, moved, processed, stored, preserved and served, granted there are economic realities. Likewise as a society our dependence on information being available for work or entertainment, from medical healthcare to social media and all points in between continues to increase (check out the Human Face of Big Data).

    In addition, people and data are living longer, as well as getting larger (hence little data, big data and very big data). Cloud products and services along with associated object storage, file systems, repositories and access methods are at the center of big data, big bandwidth and little data initiatives on a public, private, hybrid and community basis. After all, not everything is the same in cloud, virtual and traditional data centers or information factories from active data to in-active deep digital archiving.

    Click here to view (and hear) more content including cloud and object storage fundamentals

    Click here to view software defined, bulk, cloud and object storage trend news

    cloud object storage

    Where to learn more

    The following resources provide additional information about big data, bulk, software defined, cloud and object storage.



    Via InfoStor: Object Storage Is In Your Future
    Via FujiFilm IT Summit: Software Defined Data Infrastructures (SDDI) and Hybrid Clouds
    Via MultiChannel: After ditching cloud business, Verizon inks Virtual Network Services deal with Amazon
    Via MultiChannel: Verizon Digital Media Services now offers integrated Microsoft Azure Storage
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via InfoStor: Cloud Storage Concerns, Considerations and Trends
    Via InfoStor: Object Storage Is In Your Future
    Via Server StorageIO: April 2015 Newsletter Focus on Cloud and Object storage
    Via StorageIOblog: AWS S3 Cross Region Replication storage enhancements
    Cloud conversations: AWS EBS, Glacier and S3 overview
    AWS (Amazon) storage gateway, first, second and third impressions
    Cloud and Virtual Data Storage Networking (CRC Book)

    View more news, trends and related cloud object storage activity here.

    Videos and podcasts at storageio.tv also available via Applie iTunes.

    Human Face of Big Data
    Human Face of Big Data (Book review)

    Seven Databases in Seven weeks Seven Databases in Seven Weeks (Book review)

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Object and cloud storage are in your future, the questions are when, where, with what and how among others.

    Watch for more content and links to be added here soon to this object storage center page including posts, presentations, pod casts, polls, perspectives along with services and product solutions profiles.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Trick or treat and vendor fun games

    In the spirit of Halloween and zombies season, a couple of thoughts come to mind about vendor tricks and treats. This is an industry trends and perspectives post, part of an ongoing series looking at various technology and fun topics.

    The first trick or treat game pertains to the blame game; you know either when something breaks, or at the other extreme, before you have even made a decision to buy something. The trick or treat game for decision-making goes something like this.

    StorageIO industry trends cloud, virtualization and big data

    Vendor “A” says products succeed with their solution while failure results with a solution from “B” when doing “X”. Otoh, vendor “B” claims that “X” will fail when using a solution from vendor “A”. In fact, you can pick what you want to substitute for “X”, perhaps VDI, PCIe, Big Data, Little Data, Backup, Archive, Analytics, Private Cloud, Public Cloud, Hybrid Cloud, eDiscovery you name it.

    This is not complicated math or big data problem requiring a high-performance computing (HPC) platform. A HPC Zetta-Flop processing ability using 512 bit addressing of 9.9 (e.g. 1 nine) PettaBytes of battery-backed DRAM and an IO capability of 9.99999 (e.g. 5 9’s) trillion 8 bit IOPS to do table pivots or runge kutta numerical analysis, map reduce, SAS or another modeling with optional iProduct or Android interface are not needed.

    image of StorageIO big data HPC cloud storageimage of StorageIO big data HPC cloud storage
    StorageIO images of touring Texas Advanced Computing (e.g. HPC) Center

    Can you solve this equation? Hint it does not need a PhD or any other advanced degree. Another hint, if you have ever been at any side of the technology product and services decision-making table, regardless of the costume you wore, you should know the answer.

    Of course the question of would “X” fail regardless of who or what “A” or “B” let alone a “C”, “D” or “F”? In other words, it is not the solution, technology, vendor or provider, rather the problem or perhaps even lack thereof that is the issue. Or is it a case where there is a solution from “A”, “B” or any others that is looking for a problem, and if it is the wrong problem, there can be a wrong solution thus failure?

    StorageIO industry trends cloud, virtualization and big data

    Another trick or treat game is vendors public relations (PR) or analyst relations (AR) people to ask for one thing and delivery or ask another. For example, some vendor, service provider, their marketing AR and PR people or surrogates make contact wanting to tell of various success and failure story. Of course, this is usually their success and somebody else’s failure, or their victory over something or someone who sometimes can be interesting. Of course, there are also the treats to get you to listen to the above, such as tempt you with a project if you meet with their subject, which may be a trick of a disappearing treat (e.g. magic, poof it is gone after the discussion).

    There are another AR and PR trick and treat where they offer on behalf of their representative organization or client to a perspective or exclusive insight on their competitor. Of course, the treat from their perspective is that they will generously expose all that is wrong with what a competitor is saying about their own (e.g. the competitors) product.

    StorageIO industry trends cloud, virtualization and big data

    Let me get this straight, I am not supposed to believe what somebody says about his or her own product, however, supposed to believe what a competitor says is wrong with the competition’s product, and what is right with his or her own product.

    Hmm, ok, so let me get this straight, a competitor say “A” wants to tell me what somebody say from “B” has told me is wrong and I should schedule a visit with a truth squad member from “A” to get the record set straight about “B”?

    Does that mean then that I go to “B” for a rebuttal, as well as an update about “A” from “B”, assuming that what “A” has told me is also false about themselves, and perhaps about “B” or any other?

    Too be fair, depending on your level of trust and confidence in either a vendor, their personal or surrogates, you might tend to believe more from them vs. others, or at least until you been tricked after given treats. There may be some that have been tricked, or they tried applying to many treats to present a story that behind the costume might be a bit scary.

    StorageIO industry trends cloud, virtualization and big data

    Having been through enough of these, and I candidly believe that sometimes “A” or “B” or any other party actually do believe that they have more or better info about their competitor and that they can convince somebody about what their competitor is doing better than the competitor can. I also believe that there are people out there who will go to “A” or “B” and believe what they are told by based on their preference, bias or interests.

    When I hear from vendors, VARs, solution or service providers and others, it’s interesting hearing point, counterpoint and so forth, however if time is limited, I’am more interested in hearing from such as “A” about them, what they are doing, where success, where challenges, where going and if applicable, under NDA go into more detail.

    StorageIO industry trends cloud, virtualization and big data

    Customer success stories are good, however again, if interested in what works, what kind of works, or what does not work, chances are when looking for G2 vs. GQ, a non-scripted customer conversation or perspective of the good, the bad and the ugly is preferred, even if under NDA. Again, if time is limited which it usually is, focus on what is being done with your solution, where it is going and if compelled send follow-up material that can of course include MUD and FUD about others if that is your preference.

    Then there is when during a 30 minute briefing, the vendor or solution provider is still talking about trends, customer pain points, what competitors are doing at 21 minutes into the call with no sign of an announcement, update or news in site

    Lets not forget about the trick where the vendor marketing or PR person reaches out and says that the CEO, CMO, CTO or some other CxO or Chief Jailable Officer (CJO) wants to talk with you. Part of the trick is when the CxO actually makes it to the briefing and is not ready, does not know why the call is occurring, or, thinks that a request for an audience has been made with them for an interview or something else.

    StorageIO industry trends cloud, virtualization and big data

    A treat is when 3 to 4 minutes into a briefing, the vendor or solution provider has already framed up what and why they are doing something. This means getting to what they are announcing or planning on doing and getting into a conversation to discuss what they are doing and making good follow-up content and resources available.

    StorageIO industry trends cloud, virtualization and big data

    Sometimes a treat is when a briefer goes on autopilot nailing their script for 29 of a 30 minute session then use the last-minute to ask if there are any questions. The reason autopilot briefings can be a treat is when they are going over what is in the slide deck, webex, or press release thus affording an opportunity to get caught up on other things while talk at you. Hmm, perhaps need to consider playing some tricks in reward for those kind of treats? ;)

    StorageIO industry trends cloud, virtualization and big data

    Do not be scared, not everybody is out to trick you with treats, and not all treats have tricks attached to them. Be prepared, figure out who is playing tricks with treats, and who has treats without tricks.

    Oh, and as a former IT customer, vendor and analyst, one of my favorites is contact information of my dogs to vendors who require registration on their websites for basic things such as data sheets. Another is supplying contact information of competing vendors sales reps to vendors who also require registration for basic data sheets or what should otherwise be generally available information as opposed to more premium treats. Of course there are many more fun tricks, however lets leave those alone for now.

    Note: Zombie voting rules apply which means vote early, vote often, and of course vote for those who cannot include those that are dead (real or virtual).

    Where To Learn More

    View additiona related material via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Watch out for tricks and treats, have a safe and fun Zombie (aka Halloween) season. See you while out and about this fall and don’t forget to take part in the ongoing zombie technology poll. Oh, and be safe with trick or treat and vendor fun games

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2018. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    What is the best kind of IO? The one you do not have to do

    What is the best kind of IO? The one you do not have to do

    data infrastructure server storage I/O trends

    Updated 2/10/2018

    What is the best kind of IO? If no IO (input/output) operation is the best IO, than the second best IO is the one that can be done as close to the application and processor with best locality of reference. Then the third best IO is the one that can be done in less time, or at least cost or impact to the requesting application which means moving further down the memory and storage stack (figure 1).

    Storage and IO or I/O locality of reference and storage hirearchy
    Figure 1 memory and storage hierarchy

    The problem with IO is that they are basic operation to get data into and out of a computer or processor so they are required; however, they also have an impact on performance, response or wait time (latency). IO require CPU or processor time and memory to set up and then process the results as well as IO and networking resources to move data to their destination or retrieve from where stored. While IOs cannot be eliminated, their impact can be greatly improved or optimized by doing fewer of them via caching, grouped reads or writes (pre-fetch, write behind) among other techniques and technologies.

    Think of it this way, instead of going on multiple errands, sometimes you can group multiple destinations together making for a shorter, more efficient trip; however, that optimization may also take longer. Hence sometimes it makes sense to go on a couple of quick, short low latency trips vs. one single larger one that takes half a day however accomplishes many things. Of course, how far you have to go on those trips (e.g. locality) makes a difference of how many you can do in a given amount of time.

    What is locality of reference?

    Locality of reference refers to how close (e.g location) data exists for where it is needed (being referenced) for use. For example, the best locality of reference in a computer would be registers in the processor core, then level 1 (L1), level 2 (L2) or level 3 (L3) onboard cache, followed by dynamic random access memory (DRAM). Then would come memory also known as storage on PCIe cards such as nand flash solid state device (SSD) or accessible via an adapter on a direct attached storage (DAS), SAN or NAS device. In the case of a PCIe nand flash SSD card, even though physically the nand flash SSD is closer to the processor, there is still the overhead of traversing the PCIe bus and associated drivers. To help offset that impact, PCIe cards use DRAM as cache or buffers for data along with Meta or control information to further optimize and improve locality of reference. In other words, help with cache hits, cache use and cache effectiveness vs. simply boosting cache utilization.

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    What can you do the cut the impact of IO

    • Establish baseline performance and availability metrics for comparison
    • Realize that IOs are a fact of IT virtual, physical and cloud life
    • Understand what is a bad IO along with its impact
    • Identify why an IO is bad, expensive or causing an impact
    • Find and fix the problem, either with software, application or database changes
    • Throw more software caching tools, hyper visors or hardware at the problem
    • Hardware includes faster processors with more DRAM and fast internal busses
    • Leveraging local PCIe flash SSD cards for caching or as targets
    • Utilize storage systems or appliances that have intelligent caching and storage optimization capabilities (performance, availability, capacity).
    • Compare changes and improvements to baseline, quantify improvement

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    As the Hard Disk Drive HDD continues to spin

    As the Hard Disk Drive HDD continues to spin

    server storage data infrastructure i/o iop hdd ssd trends

    Updated 2/10/2018

    Despite having been repeatedly declared dead at the hands of some new emerging technology over the past several decades, the Hard Disk Drive (HDD) continues to spin and evolve as it moves towards its 60th birthday.

    More recently HDDs have been declared dead due to flash SSD that according to some predictions, should have caused the HDD to be extinct by now.

    Meanwhile, having not yet died in addition to having qualified for its AARP membership a few years ago, the HDD continues to evolve in capacity, smaller form factor, performance, reliability, density along with cost improvements.

    Back in 2006 I did an article titled Happy 50th, hard drive, but will you make it to 60?

    IMHO it is safe to say that the HDD will be around for at least a few more years if not another decade (or more).

    This is not to say that the HDD has outlived its usefulness or that there are not other tiered storage mediums to do specific jobs or tasks better (there are).

    Instead, the HDD continues to evolve and is complimented by flash SSD in a way that HDDs are complimenting magnetic tape (another declared dead technology) each finding new roles to support more data being stored for longer periods of time.

    After all, there is no such thing as a data or information recession!

    What the importance of this is about technology tiering and resource alignment, matching the applicable technology to the task at hand.

    Technology tiering (Servers, storage, networking, snow removal) is about aligning the applicable resource that is best suited to a particular need in a cost as well as productive manner. The HDD remains a viable tiered storage medium that continues to evolve while taking on new roles coexisting with SSD and tape along with cloud resources. These and other technologies have their place which ideally is finding or expanding into new markets instead of simply trying to cannibalize each other for market share.

    Here is a link to a good story by Lucas Mearian on the history or evolution of the hard disk drive (HDD) including how a 1TB device that costs about $60 today would have cost about a trillion dollars back in the 1950s. FWIW, IMHO the 1 trillion dollars is low and should be more around 2 to 5 trillion for the one TByte if you apply common costs for management, people, care and feeding, power, cooling, backup, BC, DR and other functions.

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    IMHO, it is safe to say that the HDD is here to stay for at least a few more years (if not decades) or at least until someone decides to try a new creative marketing approach by declaring it dead (again).

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    What is DFR or Data Footprint Reduction?

    What is DFR or Data Footprint Reduction?

    What is DFR or Data Footprint Reduction?

    Updated 10/9/2018

    What is DFR or Data Footprint Reduction?

    Data Footprint Reduction (DFR) is a collection of techniques, technologies, tools and best practices that are used to address data growth management challenges. Dedupe is currently the industry darling for DFR particularly in the scope or context of backup or other repetitive data.

    However DFR expands the scope of expanding data footprints and their impact to cover primary, secondary along with offline data that ranges from high performance to inactive high capacity.

    Consequently the focus of DFR is not just on reduction ratios, its also about meeting time or performance rates and data protection windows.

    This means DFR is about using the right tool for the task at hand to effectively meet business needs, and cost objectives while meeting service requirements across all applications.

    Examples of DFR technologies include Archiving, Compression, Dedupe, Data Management and Thin Provisioning among others.

    Read more about DFR in Part I and Part II of a two part series found here and here.

    Where to learn more

    Learn more about data footprint reducton (DFR), data footprint overhead and related topics via the following links:

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What this all means

    That is all for now, hope you find these ongoing series of current or emerging Industry Trends and Perspectives posts of interest.

    Ok, nuff said, for now.

    Cheers Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2018. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Two companies on parallel tracks moving like trains offset by time: EMC and NetApp

    View from VIA Rail Canada taken using Gregs iFlip

    I see some similarities and parallels between two competing companies. Those companies happen to be in the same sector (e.g. IT data storage) however offset by time (about a decade or) subject to continued execution by both.

    Those two companies are EMC and NetApp.

    Some people might assert that these two companies are complete opposites. Perhaps claiming that one is on the up swing while the other on the down path (have heard claims and counter claims of both being on the other path). I will leave the discussion or debate of which is on the up and which is on the down path to the twittervile and blogsphere ultimate tag team mud wrestling arena or You Tube video rooms.

    I see EMC and NetApp a bit differently which you can take it for what that is, simply an opinion or perspective having been the competitor and partner of both when I was on the vendor side of the table and later covering the two as an industry analyst.

    Without going too far down the memory lane route, in a nut shell, I recall when EMC was still a fledgling startup who wanted to sell me (I was on the customer side then) rebrand Fujitsu disk drives to attach to my VAX/VMS systems and memory for our mainframes. Come to think about it, Emulex was also selling disk drives back then before reinventing themselves later as an HBA and hub vendor.

    Later as a vendor, around late 94 or early 95, it was the up and coming small little bay area NAS filer appliance vendor (e.g. the toaster era) that we partnered with including a very brief OEM deal involving repackaging their product which was NetApp or Network Appliance as they were formerly known then. Once that ended after a year or so NetApp become a competitor as was EMC who at the time had as the main act the Symmetrix and about to do the EPOCH backup and McData acquisitions as well as landing the HP OEM deal for open systems.

    Ironically NetApp was out to knock off Auspex which happened fairly quickly while EMC was struggling to get its NAS act together with the early DART behemoth while successfully knocking out IBM and other entrenched high-end solutions. In a twist of fate, the company I was working for ended up selling off all of their RAID (initially a few, then later all of them) patents to EMC for some cash and later transitioned out of the hardware business becoming simply a VAR of EMC (that was MTI).

    While at INRANGE which later merged into CNT before acquired by McData (I left before that) and then Brocade, both EMC and NetApp were partners across different product lines.

    What they have in common

    Ok, enough of the memory lane stuff; lets get back to where the similarities exist.

    Back in the mid 90s, EMC was essentially a one trick pony with a very software feature function rich large storage system that sold for a premium generating lots of cash from its use of cache. Likewise, NetApp is a vendor that while it has many product offerings and has some acquisitions, still relies very much on their flagship NAS storage systems that are also feature function (e.g. software) rich that leverage cache to generate cash.

    Both companies are growing in terms of revenues, installed base, partners/OEMs and product diversity. Likewise each company needs to continue expansion into those as well as other adjacent areas.

    Can NetApp catch EMC? Maybe, maybe not, however IMHO the question should be are there other areas that NetApp can extend its reach into causing EMC to react to those, like how EMC took advantage of opportunities causing IBM and others to react.

    Here are some other similarities I see of and for EMC and NetApp:

    • Both have great outreach programs where information is provided without having to ask or dig in a proactive way, yet when something is needed, they give it without fanfare
    • Both are engaging at multiple levels, from customer, to financial and investors, to var, to partner, trade groups, to trade and other media, to analysts to social networking and beyond
    • Both are passionate about their companies, cultures, products, solutions and customers
    • Both can walk the talk, however both also like to talk and see the other balk
    • Both lead by example and not afraid to tell you what they think about something
    • Both embrace social media in connection with traditional mediums for communication with people as opposed to a giant megaphone for talking at or spamming people (when will other vendors figure that out?)
    • Both also are willing to hear what you have to say even if they do not agree with it
    • Neither is scared of the other (or at least not in public)
    • Both cause the other to play and execute a stronger game
    • Both are not above throwing a mud ball or fire cracker at the other
    • Both are not above burying the hatchet and getting along when or where needed
    • Both compete vigorously on some fronts, yet partner (publicly or privately) on other fronts
    • Both have been direct focused with some vars and some OEMs
    • Both started somewhere else and now going and moving to different places and in some ways returning to their roots or at least making sure they are not forgotten
    • Both are synonymous with their core focus products and background
    • One comes from an open systems focus working to prove itself in the enterprise
    • One comes from the enterprise establishing itself in SOHO, SMB and other spaces
    • Both have many solutions, some would say long in the tooth, others would say revolutionary
    • Both are growing via organic growth as well as acquisition and partnering
    • Both have celebrity leaders and team role players to support and back then up
    • Both also have deep benches and technical folks in the trenches to get things done
    • Both have developed leadership along with rank and file employees internal
    • Both have gone outside and brought in leadership and skilled players to expand their employee ranks
    • Both are very much involved with server virtualization (Microsoft and VMware)
    • Both are very much involved in storage virtualization and associated management
    • Both are involved with cloud solutions for enabling public or private storage
    • Both are independent storage vendors not part of a larger server organization
    • Both have interoperability programs with other vendors servers and software and networks
    • Both also get beat up about their pricing models for extensive software feature function portfolios associated with respective storage solutions
    • Both get criticized by customers or the industry as is often the case of market leaders

    What I see EMC needing to do

    • Articulate where their multiple products and services fit and play into their different target market opportunities while worrying less about the color hue of logos or video backgrounds
    • Avoiding competing with itself or becoming its own major or main competitor
    • Clarify cloud (public and private) cloud confusion transitioning into cloud cash and opportunity
    • Minimize or cut channel contention and confusion internally and across partners
    • Remember where they came from and core competences however avoid a death grip on them
    • Look to the future, leverage lessons learned that helped EMC succeed where others failed
    • EMC needs NetApp as a strong NAS competitor as each plays stronger when against the other. This is like watching world-class athletes, artists or musicians that step up their games or works when paired with another

    What I see NTAP needing to do

    • Doing an acquisition in an adjacent space, perhaps even a reverse merger of sorts to move up and out into a broader space that compliments their core offerings. For example, something outside of the normal comfort zone which arguably Datadomain would have been close to their comfort zone. Likewise acquiring a software player such as Commvault would be similar to EMC having acquired Legato, Documentum and so forth. That is NetApp would have to do a series of those. So why not something really big like a reverse merger or partial acquisition of say Symantecs data protection and management group (aka the old Veritas suite including backup, management tools, clustered file server software, volume managers etc).
    • In addition to adjacent acquisition, opportunities plays such as the recent Bycast move makes sense however then those need to be integrated and rolled out similar to what EMC has done with so many of their purchases.
    • Minimize or cut channel contention and confusion both internal across products and with partners.
    • NetApp started at the lower end SMB, grew into the SME and now enterprise place, however they tried with the StorVault and backed out of that market leaving it to EMC Iomega, Cisco, HP, Dell and others. Maybe they do not need a low-end play, however I rather liked the low-end StorVault story as well as where it was going. Oh well, needless to say I ended up buying an EMC Iomega IX4 as the StorVault left the market. Hmm, does that mean NetApp should acquire SNAP or Drobo or some other low-end SOHO play? Only if the price is right and there is an existing customer base and channel in place otherwise it would be a distraction from the core business. BTW, did I mention EMC Legato, oh excuse me, Networker came from the desktop and SMB environment however grew to the enterprise (yes I know, that is debatable) however now is difficult to put into SOHO environments.
    • Does NetApp need a stronger block storage play, perhaps a 3PAR acquisition? Maybe, perhaps not depending on if they are competing for today’s market or tomorrows.
    • Does NetApp need to be acquired? I think they can stay independent; however they need to expand their presence and footprint from a product, partner and customer perspective.
    • NetApp needs a strong NAS competitor in the likes of an EMC as the competition IMHO makes each stronger as well as providing competition which should play well for customers. Not to mention the back and forth mud ball and fire cracker tossing can be entertaining for some.

    What is your take?

    Are EMC and NetApp two companies on parallel tracks offset by time and perhaps execution?

    Cast your vote and see what others have indicated in the following poll.

    View from VIA Rail Canada taken using Gregs iFlip

    Ok, nuff said.

    Cheers gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, vSAN and VMware vExpert. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

    What do NAS NASA NASCAR have in common?

    What do NAS NASA NASCAR have in common?

    server storage I/O data infrastructure trends

    Updated 2/10/2018

    The other day it dawned on me what do NAS, NASA NASCAR have in common?

    Several things in addition to all starting with the letters NAS it turns out.

    For example, they all deal with round objects, NAS or Network Attached storage involved with circular spinning disk drives, NASA or National Aeronautical Space Administration besides involved with aircraft that have tires that go round and round, or airplanes circling waiting for landing.

    In the case of NASA they are also involved with sending craft or devices to circle other planets or moons and land or crash into them. Sometimes NAS along with other storage systems have disk drives that crash, similar to how NASCAR events see accidents.
    NAS

    Ceder Lake 3M NASCAR at dirt track - Photo (C) 2008 Karen Schulz all rights reserved

    Ceder Lake dirt track 3M NASCAR night (Photo (C) 2008 Karen Schulz)

    NASCAR is also involved with vehicles that dont or at least should not fly, however they do go round and round on a track, often paved however sometimes mud or dirt tracks plus high tech exists with computers and various data models, not to mention the NASCAR air force.

    In addition to being involved with round objects and activities, all three are also involved in computing, generating, processing, storing and retrieving for analysis of data, not to mention high performance requirements.

    NAS based storage can also be relied upon for serving the needs of NASA and NASCAR data and informational needs.

    And FWIW, just for fun, look at what you get when you spell NAS, NASA or NASCAR backwards:

    RACSAN
    ASAN
    SAN

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Not much actually other than to stimulate some thought, discussion as well as perhaps have some fun with technology during the holiday season.

    Im sure if I put some more thought to it, more similarities would or will come to mind.

    However, for now, thats it for a quick thought, what similarities do you see or know about with NAS, NASA and NASCAR?

    Ok, nuf fun for now, time to work on some other posts, content and projects.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Server Storage I/O Network Virtualization Whats Next?

    Server Storage I/O Network Virtualization Whats Next?
    Server Storage I/O Network Virtualization Whats Next?
    Updated 9/28/18

    There are many faces and thus functionalities of virtualization beyond the one most commonly discussed which is consolidation or aggregation. Other common forms of virtualization include emulation (which is part of enabling consolidation) which can be in the form of a virtual tape library for storage to bridge new disk technology to old software technology, processes, procedures and skill sets. Other forms of virtualization functionality for life beyond consolidation include abstraction for transparent movement of applications or operating systems on servers, or data on storage to support planned and un-planned maintenance, upgrades, BC/DR and other activities.

    So the gist is that there are many forms of virtualization technologies and techniques for servers, storage and even I/O networks to address different issues including life beyond consolidation. However the next wave of consolidation could and should be that of reducing the number of logical images, or, the impact of the multiple operating systems and application images, along with their associated management costs.

    This may be easier said than done, however, for those looking to cut costs even further than from what can be realized by reducing physical footprints (e.g. going from 10 to 1 or from 250 to 25 physical servers), there could be upside however it will come at a cost. The cost is like that of reducing data and storage footprint impacts with such as data management and archiving.

    Savings can be realized by archiving and deleting data via data management however that is easier said than done given the cost in terms of people time and ability to decide what to archive, even for non-compliance data along with associated business rules and policies to be defined (for automation) along with hardware, software and services (managed services, consulting and/or cloud and SaaS).

    Where To Learn More

    View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2018. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.