archive Archives

August 24, 2023November 26, 2023

PACE your Server Storage I/O decision making, its about application requirements

PACE your Server Storage I/O decision-making, it’s about application requirements. Regardless of if you are looking for physical, software-defined virtual, cloud or container storage, block, file or object, primary, secondary or protection copies, standalone, converged, hyper-converged, cluster in a box or other forms of storage and packaging, when it comes to server storage I/O decision-making, it’s about the applications.

I often see people deciding on the best storage before the questions of requirements, needs and wants are even mentioned. Sure the technology is important, so too are the techniques and trends including using new things in new ways, as well as old things in new ways. There are lots of buzzwords on the storage scene these days. But don’t even think about buying it until you truly understand your business’ storage needs.

However when it comes down to it unless you have a unique need, most environments server, and storage I/O resources exist to protect preserve and serve applications and their information or data. Recently I did a couple of articles over at Network Computing; these are tied to server and storage I/O decision-making balancing technology buzzwords with business and application requirements.

PACE and common applications characteristics

A theme I mention in the above two articles as well as elsewhere on server, storage I/O and applications is PACE. That is, application Performance Availability Capacity Economics (PACE). Different applications will have various attributes, in general, as well as how they are used. For example database transaction activity vs. reporting or analytics, logs and journals vs. redo logs, indices, tables, indices, import/export, scratch and temp space. PACE (figure 2.7) describes the applications and data characters and needs.

Common Application Pace Attributes

All applications have PACE attributes

Those PACE attributes vary by application and usage
Some applications and their data are more active vs. others
PACE characteristics will vary within different parts of an application

Think of an application along with associated data PACE as its personality or how it behaves, what it does, how it does it and when along with value, benefit or cost along with Quality of Service (QoS) attributes. Understanding the applications in different environments, data value and associated PACE attributes is essential for making informed server, storage I/O decisions from configuration to acquisitions or upgrades, when, where, why and how to protect, or performance optimization along with capacity planning, reporting, and troubleshooting, not to mention addressing budget concerns.

Data and Application PACE

Primary PACE attributes for active and inactive applications and data:
P – Performance and activity (how things get used)
A – Availability and durability (resiliency and protection)
C – Capacity and space (what things use or occupy)
E – Energy and Economics (people, budgets and other barriers)

Some applications need more performance (server computer, or storage and network I/O) while others need space capacity (storage, memory, network or I/O connectivity). Likewise, some applications have different availability needs (data protection, durability, security, resiliency, backup, BC, DR) that determine various tools, technologies and techniques to use. Budgets are also a concern which for some applications meaning enabling more performance per cost while others are focused on maximizing space capacity and protection level per cost. PACE attributes also define or influence policies for QoS (performance, availability, capacity), as well as thresholds, limits, quotas, retention and disposition among others.

Where to learn more

Learn more about data infrastructures and tradecraft related trends, tools, technologies and topics via the following links:

What does server storage I/O scaling mean to you?
Selecting Storage: Start With Requirements (Via NetworkComputing)
Selecting Storage: It’s All About The Applications (Via NetworkComputing)
Is Computer Data Storage Complex? It Depends
Software Defined Data Infrastructure Essentials Book by Greg Schulz
Part 2 – 4 3 2 1 Data Protection Application Data Availability
Part 3 – Application Data Characteristics Types Everything Is Not The Same
Part 4 – Application Data Volume Velocity Variety Everything Is Not The Same
Dont Stop Learning Expand Your Skills Experiences Everyday
Part 5 – Application Data Access Life cycle Patterns Everything Not The Same
Data Infrastructure server storage I/O network Recommended Reading
World Backup Day 2018 Data Protection Readiness Reminder
Data Infrastructure Server Storage I/O related Tradecraft Overview
Data Infrastructure Server Storage I/O Tradecraft Trends
Data Infrastructure Overview, Its What’s Inside of Data Centers
4 3 2 1 and 3 2 1 data protection best practices
Garbage data in, garbage information out, big data or big garbage?
GDPR (General Data Protection Regulation) Resources Are You Ready?
If NVMe is the answer, what are the questions?
NVMe Primer (or refresh), The NVMe Place, The SSD Place, and the Object Storage Center

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means

The best storage will be the one that meets or exceeds your application requirements instead of the solution that meets somebody else’s needs or wants. Keep in mind, PACE your Server Storage I/O decision making, it is about application requirements

Ok, nuff said, for now.

Cheers Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2018. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

April 2, 2018November 26, 2023

Data Protection Recovery Life Post World Backup Day Pre GDPR

Data Protection Recovery Life Post World Backup Day Pre GDPR trends

It’s time for Data Protection Recovery Life Post World Backup Day Pre GDPR Start Date.

The annual March 31 world backup day focus has come and gone once again.

However, that does not mean data protection including backup as well as recovery along with security gets a 364-day vacation until March 31, 2019 (or the days leading up to it).

Granted, for some environments, public relations, editors, influencers and other industry folks backup day will take some time off while others jump on the ramp up to GDPR which goes into effect May 25, 2018.

Expanding Focus Data Protection and GDPR

As I mentioned in this post here, world backup day should be expanded to include increased focus not just on backup, also recovery as well as other forms of data protection. Likewise, May 25 2018 is not the deadline or finish line or the destination for GDPR (e.g. Global Data Protection Regulations), rather, it is the starting point for an evolving journey, one that has global impact as well as applicability. Recently I participated in a fireside chat discussion with Danny Allan of Veeam who shared his GDPR expertise as well as experiences, lessons learned, tips of Veeam as they started their journey, check it out here.

Expanding Focus Data Protection Recovery and other Things that start with R

As part of expanding the focus on Data Protection Recovery Life Post World Backup Day Pre GDPR, that also means looking at, discussing things that start with R (like Recovery). Some examples besides recovery include restoration, reassess, review, rethink protection, recovery point, RPO, RTO, reconstruction, resiliency, ransomware, RAID, repair, remediation, restart, resume, rollback, and regulations among others.

Data Protection Tips, Reminders and Recommendations

There are no blue participation ribbons for failed recovery. However, there can be pink slips.
Only you can prevent on-premises or cloud data loss. However, it is also a shared responsibility with vendors and service providers
You can’t go forward in the future when there is a disaster or loss of data if you can’t go back in time for recovery
GDPR appliances to organizations around the world of all size and across all sectors including nonprofit
Keep new school 4 3 2 1 data protection in mind while evolving from old school 3 2 1 backup rules

4 3 2 1 backup data protection rule

A Fundamental premise of data infrastructures is to enable applications and their data, protect, preserve, secure and serve
Remember to protect your applications, as well as data including metadata, settings configurations
Test your restores including can you use the data along with security settings
Don’t cause a disaster in the course of testing your data protection, backups or recovery
Expand (or refresh) your data protection and data infrastructure education tradecraft skills experiences

Where to learn more

Learn more about data protection, world backup day, recovery, restoration, GDPR along with related data infrastructure topics for cloud, legacy and other software defined environments via the following links:

AWS Cloud Application Data Protection Webinar
March 2018 Server StorageIO Data Infrastructure Update Newsletter
Application Data Value Characteristics Everything Is Not The Same (five-part mini-series)
Application Data Availability 4 3 2 1 Data Protection (part of the mini-series)
World Backup Day: Best Practices for a Hybrid Approach
Data Protection Diaries Fundamental Topics Tools Techniques Technologies Tips
World Backup Day 2018 Data Protection Readiness Reminder
Data Protection Fundamental Topics Tools Techniques Technologies Tips
Data Protection Diaries (Archive, Backup/Restore, BC, BR, DR, HA,Replication, Security)
Veeam GDPR preparedness experiences Webinar walking the talk
Data Infrastructure Server Storage I/O related Tradecraft Overview
Data Infrastructure Overview, Its What’s Inside of Data Centers
4 3 2 1 and 3 2 1 data protection best practices
Garbage data in, garbage information out, big data or big garbage?
GDPR (General Data Protection Regulation) Resources Are You Ready?
Have you heard about the new CLOUD Act data regulation?
Data Infrastructure server storage I/O network Recommended Reading
Object Storage Center resources (www.objectstoragecenter.com)
The SSD Place (SSD, NVM, PM, SCM, Flash, NVMe, 3D XPoint, MRAM and related topics)
The NVMe Place (NVMe related topics, trends, tools, technologies, tip resources)

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means and wrap-up

Data protection including business continuance (BC), business resiliency (BR), disaster recovery (DR), availability, accessibility, backup, snapshots, encryption, security, privacy among others is a 7 x 24 x 365 day a year focus. The focus of data protection also needs to evolve from an after the fact cost overhead to proactive, business enabler Meanwhile, welcome to Data Protection Recovery Post World Backup Day Pre GDPR Start Date.

Ok, nuff said, for now.

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

March 13, 2018November 26, 2023

Application Data Value Characteristics Everything Is Not The Same (Part I)

Application Data Value Characteristics Everything Is Not The Same

This is part one of a five-part mini-series looking at Application Data Value Characteristics Everything Is Not The Same as a companion excerpt from chapter 2 of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017). available at Amazon.com and other global venues. In this post, we start things off by looking at general application server storage I/O characteristics that have an impact on data value as well as access.

Everything is not the same across different organizations including Information Technology (IT) data centers, data infrastructures along with the applications as well as data they support. For example, there is so-called big data that can be many small files, objects, blobs or data and bit streams representing telemetry, click stream analytics, logs among other information.

Keep in mind that applications impact how data is accessed, used, processed, moved and stored. What this means is that a focus on data value, access patterns, along with other related topics need to also consider application performance, availability, capacity, economic (PACE) attributes.

If everything is not the same, why is so much data along with many applications treated the same from a PACE perspective?

Data Infrastructure resources including servers, storage, networks might be cheap or inexpensive, however, there is a cost to managing them along with data.

Managing includes data protection (backup, restore, BC, DR, HA, security) along with other activities. Likewise, there is a cost to the software along with cloud services among others. By understanding how applications use and interact with data, smarter, more informed data management decisions can be made.

IT Applications and Data Infrastructure Layers

Keep in mind that everything is not the same across various organizations, data centers, data infrastructures, data and the applications that use them. Also keep in mind that programs (e.g. applications) = algorithms (code) + data structures (how data defined and organized, structured or unstructured).

There are traditional applications, along with those tied to Internet of Things (IoT), Artificial Intelligence (AI) and Machine Learning (ML), Big Data and other analytics including real-time click stream, media and entertainment, security and surveillance, log and telemetry processing among many others.

What this means is that there are many different application with various character attributes along with resource (server compute, I/O network and memory, storage requirements) along with service requirements.

Common Applications Characteristics

Different applications will have various attributes, in general, as well as how they are used, for example, database transaction activity vs. reporting or analytics, logs and journals vs. redo logs, indices, tables, indices, import/export, scratch and temp space. Performance, availability, capacity, and economics (PACE) describes the applications and data characters and needs shown in the following figure.

Application PACE attributes (via Software Defined Data Infrastructure Essentials)

All applications have PACE attributes, however:

PACE attributes vary by application and usage
Some applications and their data are more active than others
PACE characteristics may vary within different parts of an application

Think of applications along with associated data PACE as its personality or how it behaves, what it does, how it does it, and when, along with value, benefit, or cost as well as quality-of-service (QoS) attributes.

Understanding applications in different environments, including data values and associated PACE attributes, is essential for making informed server, storage, I/O decisions and data infrastructure decisions. Data infrastructures decisions range from configuration to acquisitions or upgrades, when, where, why, and how to protect, and how to optimize performance including capacity planning, reporting, and troubleshooting, not to mention addressing budget concerns.

Primary PACE attributes for active and inactive applications and data are:

P – Performance and activity (how things get used)
A – Availability and durability (resiliency and data protection)
C – Capacity and space (what things use or occupy)
E – Economics and Energy (people, budgets, and other barriers)

Some applications need more performance (server computer, or storage and network I/O), while others need space capacity (storage, memory, network, or I/O connectivity). Likewise, some applications have different availability needs (data protection, durability, security, resiliency, backup, business continuity, disaster recovery) that determine the tools, technologies, and techniques to use.

Budgets are also nearly always a concern, which for some applications means enabling more performance per cost while others are focused on maximizing space capacity and protection level per cost. PACE attributes also define or influence policies for QoS (performance, availability, capacity), as well as thresholds, limits, quotas, retention, and disposition, among others.

Performance and Activity (How Resources Get Used)

Some applications or components that comprise a larger solution will have more performance demands than others. Likewise, the performance characteristics of applications along with their associated data will also vary. Performance applies to the server, storage, and I/O networking hardware along with associated software and applications.

For servers, performance is focused on how much CPU or processor time is used, along with memory and I/O operations. I/O operations to create, read, update, or delete (CRUD) data include activity rate (frequency or data velocity) of I/O operations (IOPS). Other considerations include the volume or amount of data being moved (bandwidth, throughput, transfer), response time or latency, along with queue depths.

Activity is the amount of work to do or being done in a given amount of time (seconds, minutes, hours, days, weeks), which can be transactions, rates, IOPs. Additional performance considerations include latency, bandwidth, throughput, response time, queues, reads or writes, gets or puts, updates, lists, directories, searches, pages views, files opened, videos viewed, or downloads.

Server, storage, and I/O network performance include:

Processor CPU usage time and queues (user and system overhead)
Memory usage effectiveness including page and swap
I/O activity including between servers and storage
Errors, retransmission, retries, and rebuilds

the following figure shows a generic performance example of data being accessed (mixed reads, writes, random, sequential, big, small, low and high-latency) on a local and a remote basis. The example shows how for a given time interval (see lower right), applications are accessing and working with data via different data streams in the larger image left center. Also shown are queues and I/O handling along with end-to-end (E2E) response time.

Server I/O performance fundamentals (via Software Defined Data Infrastructure Essentials)

Click here to view a larger version of the above figure.

Also shown on the left in the above figure is an example of E2E response time from the application through the various data infrastructure layers, as well as, lower center, the response time from the server to the memory or storage devices.

Various queues are shown in the middle of the above figure which are indicators of how much work is occurring, if the processing is keeping up with the work or causing backlogs. Context is needed for queues, as they exist in the server, I/O networking devices, and software drivers, as well as in storage among other locations.

Some basic server, storage, I/O metrics that matter include:

Queue depth of I/Os waiting to be processed and concurrency
CPU and memory usage to process I/Os
I/O size, or how much data can be moved in a given operation
I/O activity rate or IOPs = amount of data moved/I/O size per unit of time
Bandwidth = data moved per unit of time = I/O size × I/O rate
Latency usually increases with larger I/O sizes, decreases with smaller requests
I/O rates usually increase with smaller I/O sizes and vice versa
Bandwidth increases with larger I/O sizes and vice versa
Sequential stream access data may have better performance than some random access data
Not all data is conducive to being sequential stream, or random
Lower response time is better, higher activity rates and bandwidth are better

Queues with high latency and small I/O size or small I/O rates could indicate a performance bottleneck. Queues with low latency and high I/O rates with good bandwidth or data being moved could be a good thing. An important note is to look at several metrics, not just IOPs or activity, or bandwidth, queues, or response time. Also, keep in mind that metrics that matter for your environment may be different from those for somebody else.

Something to keep in perspective is that there can be a large amount of data with low performance, or a small amount of data with high-performance, not to mention many other variations. The important concept is that as space capacity scales, that does not mean performance also improves or vice versa, after all, everything is not the same.

Where to learn more

Learn more about Application Data Value, application characteristics, PACE along with data protection, software defined data center (SDDC), software defined data infrastructures (SDDI) and related topics via the following links:

Part 1 – Application Data Value Characteristics Everything Is Not The Same
Part 2 – 4 3 2 1 Data Protection Application Data Availability
Part 3 – Application Data Characteristics Types Everything Is Not The Same
Part 4 – Application Data Volume Velocity Variety Everything Is Not The Same
Part 5 – Application Data Access Life cycle Patterns Everything Not The Same
Data Infrastructure server storage I/O network Recommended Reading
World Backup Day 2018 Data Protection Readiness Reminder
Data Infrastructure Server Storage I/O related Tradecraft Overview
Data Infrastructure Overview, Its What’s Inside of Data Centers
4 3 2 1 and 3 2 1 data protection best practices
Garbage data in, garbage information out, big data or big garbage?
GDPR (General Data Protection Regulation) Resources Are You Ready?
Which Enterprise HDD to use for a Content Server Platform
The SSD Place (SSD, NVM, PM, SCM, Flash, NVMe, 3D XPoint, MRAM and related topics)
The NVMe Place (NVMe related topics, trends, tools, technologies, tip resources)
Data Protection Diaries (Archive, Backup/Restore, BC, BR, DR, HA,Replication, Security)

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means and wrap-up

Ok, nuff said, for now.

March 13, 2018November 26, 2023

Application Data Availability 4 3 2 1 Data Protection

4 3 2 1 data protection Application Data Availability Everything Is Not The Same

Application Data Availability 4 3 2 1 Data Protection

This is part two of a five-part mini-series looking at Application Data Value Characteristics everything is not the same as a companion excerpt from chapter 2 of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017). available at Amazon.com and other global venues. In this post, we continue looking at application performance, availability, capacity, economic (PACE) attributes that have an impact on data value as well as availability.

Availability (Accessibility, Durability, Consistency)

Just as there are many different aspects and focus areas for performance, there are also several facets to availability. Note that applications performance requires availability and availability relies on some level of performance.

Availability is a broad and encompassing area that includes data protection to protect, preserve, and serve (backup/restore, archive, BC, BR, DR, HA) data and applications. There are logical and physical aspects of availability including data protection as well as security including key management (manage your keys or authentication and certificates) and permissions, among other things.

Availability = accessibility (can you get to your application and data) + durability (is the data intact and consistent). This includes basic Reliability, Availability, Serviceability (RAS), as well as high availability, accessibility, and durability. “Durable” has multiple meanings, so context is important. Durable means how data infrastructure resources hold up to, survive, and tolerate wear and tear from use (i.e., endurance), for example, Flash SSD or mechanical devices such as Hard Disk Drives (HDDs). Another context for durable refers to data, meaning how many copies in various places.

Server, storage, and I/O network availability topics include:

Resiliency and self-healing to tolerate failure or disruption
Hardware, software, and services configured for resiliency
Accessibility to reach or be reached for handling work
Durability and consistency of data to be available for access
Protection of data, applications, and assets including security

Additional server I/O and data infrastructure along with storage topics include:

Backup/restore, replication, snapshots, sync, and copies
Basic Reliability, Availability, Serviceability, HA, fail over, BC, BR, and DR
Alternative paths, redundant components, and associated software
Applications that are fault-tolerant, resilient, and self-healing
Non disruptive upgrades, code (application or software) loads, and activation
Immediate data consistency and integrity vs. eventual consistency
Virus, malware, and other data corruption or loss prevention

From a data protection standpoint, the fundamental rule or guideline is 4 3 2 1, which means having at least four copies consisting of at least three versions (different points in time), at least two of which are on different systems or storage devices and at least one of those is off-site (on-line, off-line, cloud, or other). There are many variations of the 4 3 2 1 rule shown in the following figure along with approaches on how to manage technology to use. We will go into deeper this subject in later chapters. For now, remember the following.

4 3 2 1 data protection (via Software Defined Data Infrastructure Essentials)

4    At least four copies of data (or more), Enables durability in case a copy goes bad, deleted, corrupted, failed device, or site.
3    The number (or more) versions of the data to retain, Enables various recovery points in time to restore, resume, restart from.
2    Data located on two or more systems (devices or media/mediums), Enables protection against device, system, server, file system, or other fault/failure.

1 With at least one of those copies being off-premise and not live (isolated from active primary copy), Enables resiliency across sites, as well as space, time, distance gap for protection.

Capacity and Space (What Gets Consumed and Occupied)

In addition to being available and accessible in a timely manner (performance), data (and applications) occupy space. That space is memory in servers, as well as using available consumable processor CPU time along with I/O (performance) including over networks.

Data and applications also consume storage space where they are stored. In addition to basic data space, there is also space consumed for metadata as well as protection copies (and overhead), application settings, logs, and other items. Another aspect of capacity includes network IP ports and addresses, software licenses, server, storage, and network bandwidth or service time.

Server, storage, and I/O network capacity topics include:

Consumable time-expiring resources (processor time, I/O, network bandwidth)
Network IP and other addresses
Physical resources of servers, storage, and I/O networking devices
Software licenses based on consumption or number of users
Primary and protection copies of data and applications
Active and standby data infrastructure resources and sites
Data footprint reduction (DFR) tools and techniques for space optimization
Policies, quotas, thresholds, limits, and capacity QoS
Application and database optimization

DFR includes various techniques, technologies, and tools to reduce the impact or overhead of protecting, preserving, and serving more data for longer periods of time. There are many different approaches to implementing a DFR strategy, since there are various applications and data.

Common DFR techniques and technologies include archiving, backup modernization, copy data management (CDM), clean up, compress, and consolidate, data management, deletion and dedupe, storage tiering, RAID (including parity-based, erasure codes , local reconstruction codes [LRC] , and Reed-Solomon , Ceph Shingled Erasure Code (SHEC ), among others), along with protection configurations along with thin-provisioning, among others.

DFR can be implemented in various complementary locations from row-level compression in database or email to normalized databases, to file systems, operating systems, appliances, and storage systems using various techniques.

Also, keep in mind that not all data is the same; some is sparse, some is dense, some can be compressed or deduped while others cannot. Likewise, some data may not be compressible or dedupable. However, identical copies can be identified with links created to a common copy.

Economics (People, Budgets, Energy and other Constraints)

If one thing in life and technology that is constant is change, then the other constant is concern about economics or costs. There is a cost to enable and maintain a data infrastructure on premise or in the cloud, which exists to protect, preserve, and serve data and information applications.

However, there should also be a benefit to having the data infrastructure to house data and support applications that provide information to users of the services. A common economic focus is what something costs, either as up-front capital expenditure (CapEx) or as an operating expenditure (OpEx) expense, along with recurring fees.

In general, economic considerations include:

Budgets (CapEx and OpEx), both up front and in recurring fees
Whether you buy, lease, rent, subscribe, or use free and open sources
People time needed to integrate and support even free open-source software
Costs including hardware, software, services, power, cooling, facilities, tools
People time includes base salary, benefits, training and education

Where to learn more

Part 1 – Application Data Value Characteristics Everything Is Not The Same
Part 2 – 4 3 2 1 Data Protection Application Data Availability
Part 3 – Application Data Characteristics Types Everything Is Not The Same
Part 4 – Application Data Volume Velocity Variety Everything Is Not The Same
Part 5 – Application Data Access life cycle Patterns Everything Not The Same
Data Infrastructure server storage I/O network Recommended Reading
World Backup Day 2018 Data Protection Readiness Reminder
Data Infrastructure Server Storage I/O related Tradecraft Overview
Data Infrastructure Overview, Its What’s Inside of Data Centers
4 3 2 1 and 3 2 1 data protection best practices
Garbage data in, garbage information out, big data or big garbage?
GDPR (General Data Protection Regulation) Resources Are You Ready?
Which Enterprise HDD to use for a Content Server Platform
The SSD Place (SSD, NVM, PM, SCM, Flash, NVMe, 3D XPoint, MRAM and related topics)
The NVMe Place (NVMe related topics, trends, tools, technologies, tip resources)
Data Protection Diaries (Archive, Backup/Restore, BC, BR, DR, HA,Replication, Security)

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means and wrap-up

Keep in mind that with Application Data Value Characteristics Everything Is Not The Same across various organizations, data centers, data infrastructures spanning legacy, cloud and other software defined data center (SDDC) environments. All applications have some element of performance, availability, capacity, economic (PACE) needs as well as resource demands. There is often a focus around data storage about storage efficiency and utilization which is where data footprint reduction (DFR) techniques, tools, trends and as well as technologies address capacity requirements. However with data storage there is also an expanding focus around storage effectiveness also known as productivity tied to performance, along with availability including 4 3 2 1 data protection. Continue reading the next post (Part III Application Data Characteristics Types Everything Is Not The Same) in this series here.

Ok, nuff said, for now.

March 13, 2018November 26, 2023

Application Data Characteristics Types Everything Is Not The Same

This is part three of a five-part mini-series looking at Application Data Value Characteristics everything is not the same as a companion excerpt from chapter 2 of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017). available at Amazon.com and other global venues. In this post, we continue looking at application and data characteristics with a focus on different types of data. There is more to data than simply being big data, fast data, big fast or unstructured, structured or semistructured, some of which has been touched on in this series, with more to follow. Note that there is also data in terms of the programs, applications, code, rules, policies as well as configuration settings, metadata along with other items stored.

Various Types of Data

Data types along with characteristics include big data, little data, fast data, and old as well as new data with a different value, life-cycle, volume and velocity. There are data in files and objects that are big representing images, figures, text, binary, structured or unstructured that are software defined by the applications that create, modify and use them.

There are many different types of data and applications to meet various business, organization, or functional needs. Keep in mind that applications are based on programs which consist of algorithms and data structures that define the data, how to use it, as well as how and when to store it. Those data structures define data that will get transformed into information by programs while also being stored in memory and on data stored in various formats.

Just as various applications have different algorithms, they also have different types of data. Even though everything is not the same in all environments, or even how the same applications get used across various organizations, there are some similarities. Even though there are different types of applications and data, there are also some similarities and general characteristics. Keep in mind that information is the result of programs (applications and their algorithms) that process data into something useful or of value.

Data typically has a basic life cycle of:

Creation and some activity, including being protected
Dormant, followed by either continued activity or going inactive
Disposition (delete or remove)

In general, data can be

Temporary, ephemeral or transient
Dynamic or changing (“hot data”)
Active static on-line, near-line, or off-line (“warm-data”)
In-active static on-line or off-line (“cold data”)

Data is organized

Structured
Semi-structured
Unstructured

General data characteristics include:

Value = From no value to unknown to some or high value
Volume = Amount of data, files, objects of a given size
Variety = Various types of data (small, big, fast, structured, unstructured)
Velocity = Data streams, flows, rates, load, process, access, active or static

The following figure shows how different data has various values over time. Data that has no value today or in the future can be deleted, while data with unknown value can be retained.

Different data with various values over time

Application Data Value across sddc
Data Value Known, Unknown and No Value

General characteristics include the value of the data which in turn determines its performance, availability, capacity, and economic considerations. Also, data can be ephemeral (temporary) or kept for longer periods of time on persistent, non-volatile storage (you do not lose the data when power is turned off). Examples of temporary scratch include work and scratch areas such as where data gets imported into, or exported out of, an application or database.

Data can also be little, big, or big and fast, terms which describe in part the size as well as volume along with the speed or velocity of being created, accessed, and processed. The importance of understanding characteristics of data and how their associated applications use them is to enable effective decision-making about performance, availability, capacity, and economics of data infrastructure resources.

Data Value

There is more to data storage than how much space capacity per cost.

All data has one of three basic values:

No value = ephemeral/temp/scratch = Why keep it?
Some value = current or emerging future value, which can be low or high = Keep
Unknown value = protect until value is unlocked, or no remaining value

In addition to the above basic three, data with some value can also be further subdivided into little value, some value, or high value. Of course, you can keep subdividing into as many more or different categories as needed, after all, everything is not always the same across environments.

Besides data having some value, that value can also change by increasing or decreasing in value over time or even going from unknown to a known value, known to unknown, or to no value. Data with no value can be discarded, if in doubt, make and keep a copy of that data somewhere safe until its value (or lack of value) is fully known and understood.

The importance of understanding the value of data is to enable effective decision-making on where and how to protect, preserve, and cost-effectively store the data. Note that cost-effective does not necessarily mean the cheapest or lowest-cost approach, rather it means the way that aligns with the value and importance of the data at a given point in time.

Where to learn more

Learn more about Application Data Value, application characteristics, PACE along with data protection, software-defined data center (SDDC), software-defined data infrastructures (SDDI) and related topics via the following links:

Part 1 – Application Data Value Characteristics Everything Is Not The Same
Part 2 – 4 3 2 1 Data Protection Application Data Availability
Part 3 – Application Data Characteristics Types Everything Is Not The Same
Part 4 – Application Data Volume Velocity Variety Everything Is Not The Same
Part 5 – Application Data Access life cycle Patterns Everything Not The Same
Data Infrastructure server storage I/O network Recommended Reading
World Backup Day 2018 Data Protection Readiness Reminder
Data Infrastructure Server Storage I/O related Tradecraft Overview
Data Infrastructure Overview, Its What’s Inside of Data Centers
4 3 2 1 and 3 2 1 data protection best practices
Garbage data in, garbage information out, big data or big garbage?
GDPR (General Data Protection Regulation) Resources Are You Ready?
Which Enterprise HDD to use for a Content Server Platform
The SSD Place (SSD, NVM, PM, SCM, Flash, NVMe, 3D XPoint, MRAM and related topics)
The NVMe Place (NVMe related topics, trends, tools, technologies, tip resources)
Data Protection Diaries (Archive, Backup/Restore, BC, BR, DR, HA,Replication, Security)

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means and wrap-up

Data has different value at various times, and that value is also evolving. Everything Is Not The Same across various organizations, data centers, data infrastructures spanning legacy, cloud and other software defined data center (SDDC) environments. Continue reading the next post (Part IV Application Data Volume Velocity Variety Everything Not The Same) in this series here.

Ok, nuff said, for now.

March 13, 2018October 18, 2024

Application Data Volume Velocity Variety Everything Is Not The Same

Application Data Volume Velocity Variety Everything Not The Same

This is part four of a five-part mini-series looking at Application Data Value Characteristics everything is not the same as a companion excerpt from chapter 2 of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017). available at Amazon.com and other global venues. In this post, we continue looking at application and data characteristics with a focus on data volume velocity and variety, after all, everything is not the same, not to mention many different aspects of big data as well as little data.

Volume of Data

More data is growing at a faster rate every day, and that data is being retained for longer periods. Some data being retained has known value, while a growing amount of data has an unknown value. Data is generated or created from many sources, including mobile devices, social networks, web-connected systems or machines, and sensors including IoT and IoD. Besides where data is created from, there are also many consumers of data (applications) that range from legacy to mobile, cloud, IoT among others.

Unknown-value data may eventually have value in the future when somebody realizes that he can do something with it, or a technology tool or application becomes available to transform the data with unknown value into valuable information.

Some data gets retained in its native or raw form, while other data get processed by application program algorithms into summary data, or is curated and aggregated with other data to be transformed into new useful data. The figure below shows, from left to right and front to back, more data being created, and that data also getting larger over time. For example, on the left are two data items, objects, files, or blocks representing some information.

In the center of the following figure are more columns and rows of data, with each of those data items also becoming larger. Moving farther to the right, there are yet more data items stacked up higher, as well as across and farther back, with those items also being larger. The following figure can represent blocks of storage, files in a file system, rows, and columns in a database or key-value repository, or objects in a cloud or object storage system.

Application Data Value sddc
Increasing data velocity and volume, more data and data getting larger

In addition to more data being created, some of that data is relatively small in terms of the records or data structure entities being stored. However, there can be a large quantity of those smaller data items. In addition to the amount of data, as well as the size of the data, protection or overhead copies of data are also kept.

Another dimension is that data is also getting larger where the data structures describing a piece of data for an application have increased in size. For example, a still photograph was taken with a digital camera, cell phone, or another mobile handheld device, drone, or other IoT device, increases in size with each new generation of cameras as there are more megapixels.

Variety of Data

In addition to having value and volume, there are also different varieties of data, including ephemeral (temporary), persistent, primary, metadata, structured, semi-structured, unstructured, little, and big data. Keep in mind that programs, applications, tools, and utilities get stored as data, while they also use, create, access, and manage data.

There is also primary data and metadata, or data about data, as well as system data that is also sometimes referred to as metadata. Here is where context comes into play as part of tradecraft, as there can be metadata describing data being used by programs, as well as metadata about systems, applications, file systems, databases, and storage systems, among other things, including little and big data.

Context also matters regarding big data, as there are applications such as statistical analysis software and Hadoop, among others, for processing (analyzing) large amounts of data. The data being processed may not be big regarding the records or data entity items, but there may be a large volume. In addition to big data analytics, data, and applications, there is also data that is very big (as well as large volumes or collections of data sets).

For example, video and audio, among others, may also be referred to as big fast data, or large data. A challenge with larger data items is the complexity of moving over the distance promptly, as well as processing requiring new approaches, algorithms, data structures, and storage management techniques.

Likewise, the challenges with large volumes of smaller data are similar in that data needs to be moved, protected, preserved, and served cost-effectively for long periods of time. Both large and small data are stored (in memory or storage) in various types of data repositories.

In general, data in repositories is accessed locally, remotely, or via a cloud using:

Object and blobs stream, queue, and Application Programming Interface (API)
File-based using local or networked file systems
Block-based access of disk partitions, LUNs (logical unit numbers), or volumes

The following figure shows varieties of application data value including (left) photos or images, audio, videos, and various log, event, and telemetry data, as well as (right) sparse and dense data.

Application Data Value bits bytes blocks blobs bitstreams sddc
Varieties of data (bits, bytes, blocks, blobs, and bitstreams)

Velocity of Data

Data, in addition to having value (known, unknown, or none), volume (size and quantity), and variety (structured, unstructured, semi structured, primary, metadata, small, big), also has velocity. Velocity refers to how fast (or slowly) data is accessed, including being stored, retrieved, updated, scanned, or if it is active (updated, or fixed static) or dormant and inactive. In addition to data access and life cycle, velocity also refers to how data is used, such as random or sequential or some combination. Think of data velocity as how data, or streams of data, flow in various ways.

Velocity also describes how data is used and accessed, including:

Active (hot), static (warm and WORM), or dormant (cold)
Random or sequential, read or write-accessed
Real-time (online, synchronous) or time-delayed

Why this matters is that by understanding and knowing how applications use data, or how data is accessed via applications, you can make informed decisions. Also, having insight enables how to design, configure, and manage servers, storage, and I/O resources (hardware, software, services) to meet various needs. Understanding Application Data Value including the velocity of the data both for when it is created as well as when used is important for aligning the applicable performance techniques and technologies.

Where to learn more

Learn more about Application Data Value, application characteristics, performance, availability, capacity, economic (PACE) along with data protection, software-defined data center (SDDC), software-defined data infrastructures (SDDI) and related topics via the following links:

- Part 1 – Application Data Value Characteristics Everything Is Not The Same
- Part 2 – 4 3 2 1 Data Protection Application Data Availability
- Part 3 – Application Data Characteristics Types Everything Is Not The Same
- Part 4 – Application Data Volume Velocity Variety Everything Is Not The Same
- Part 5 – Application Data Access life cycle Patterns Everything Not The Same
- Software Defined, Cloud, Object and Blob Storage
- Data Infrastructure server storage I/O network Recommended Reading
- World Backup Day 2018 Data Protection Readiness Reminder
- Data Infrastructure Server Storage I/O related Tradecraft Overview
- Data Infrastructure Overview, Its What’s Inside of Data Centers
- 4 3 2 1 and 3 2 1 data protection best practices
- Garbage data in, garbage information out, big data or big garbage?
- GDPR (General Data Protection Regulation) Resources Are You Ready?
- Which Enterprise HDD to use for a Content Server Platform
- The SSD Place (SSD, NVM, PM, SCM, Flash, NVMe, 3D XPoint, MRAM and related topics)
- The NVMe Place (NVMe related topics, trends, tools, technologies, tip resources)
- Data Protection Diaries (Archive, Backup/Restore, BC, BR, DR, HA,Replication, Security)

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means and wrap-up

Data has different value, size, as well as velocity as part of its characteristic including how used by various applications. Keep in mind that with Application Data Value Characteristics Everything Is Not The Same across various organizations, data centers, data infrastructures spanning legacy, cloud and other software defined data center (SDDC) environments. Continue reading the next post (Part V Application Data Access life cycle Patterns Everything Is Not The Same) in this series here.

Ok, nuff said, for now.

March 13, 2018November 26, 2023

Application Data Access Lifecycle Patterns Everything Is Not The Same

Application Data Access Life cycle Patterns Everything Is Not The Same(Part V)

Application Data Access Life cycle Patterns Everything Is Not The Same

This is part five of a five-part mini-series looking at Application Data Value Characteristics everything is not the same as a companion excerpt from chapter 2 of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017). available at Amazon.com and other global venues. In this post, we look at various application and data lifecycle patterns as well as wrap up this series.

Active (Hot), Static (Warm and WORM), or Dormant (Cold) Data and Lifecycles

When it comes to Application Data Value, a common question I hear is why not keep all data?

If the data has value, and you have a large enough budget, why not? On the other hand, most organizations have a budget and other constraints that determine how much and what data to retain.

Another common question I get asked (or told) it isn’t the objective to keep less data to cut costs?

If the data has no value, then get rid of it. On the other hand, if data has value or unknown value, then find ways to remove the cost of keeping more data for longer periods of time so its value can be realized.

In general, the data life cycle (called by some cradle to grave, birth or creation to disposition) is created, save and store, perhaps update and read with changing access patterns over time, along with value. During that time, the data (which includes applications and their settings) will be protected with copies or some other technique, and eventually disposed of.

Between the time when data is created and when it is disposed of, there are many variations of what gets done and needs to be done. Considering static data for a moment, some applications and their data, or data and their applications, create data which is for a short period, then goes dormant, then is active again briefly before going cold (see the left side of the following figure). This is a classic application, data, and information life-cycle model (ILM), and tiering or data movement and migration that still applies for some scenarios.

Application Data Value
Changing data access patterns for different applications

However, a newer scenario over the past several years that continues to increase is shown on the right side of the above figure. In this scenario, data is initially active for updates, then goes cold or WORM (Write Once/Read Many); however, it warms back up as a static reference, on the web, as big data, and for other uses where it is used to create new data and information.

Data, in addition to its other attributes already mentioned, can be active (hot), residing in a memory cache, buffers inside a server, or on a fast storage appliance or caching appliance. Hot data means that it is actively being used for reads or writes (this is what the term Heat map pertains to in the context of the server, storage data, and applications. The heat map shows where the hot or active data is along with its other characteristics.

Context is important here, as there are also IT facilities heat maps, which refer to physical facilities including what servers are consuming power and generating heat. Note that some current and emerging data center infrastructure management (DCIM) tools can correlate the physical facilities power, cooling, and heat to actual work being done from an applications perspective. This correlated or converged management view enables more granular analysis and effective decision-making on how to best utilize data infrastructure resources.

In addition to being hot or active, data can be warm (not as heavily accessed) or cold (rarely if ever accessed), as well as online, near-line, or off-line. As their names imply, warm data may occasionally be used, either updated and written, or static and just being read. Some data also gets protected as WORM data using hardware or software technologies. WORM (immutable) data, not to be confused with warm data, is fixed or immutable (cannot be changed).

When looking at data (or storage), it is important to see when the data was created as well as when it was modified. However, you should avoid the mistake of looking only at when it was created or modified: Instead, also look to see when it was the last read, as well as how often it is read. You might find that some data has not been updated for several years, but it is still accessed several times an hour or minute. Also, keep in mind that the metadata about the actual data may be being updated, even while the data itself is static.

Also, look at your applications characteristics as well as how data gets used, to see if it is conducive to caching or automated tiering based on activity, events, or time. For example, there is a large amount of data for an energy or oil exploration project that normally sits on slower lower-cost storage, but that now and then some analysis needs to run on.

Using data and storage management tools, given notice or based on activity, which large or big data could be promoted to faster storage, or applications migrated to be closer to the data to speed up processing. Another example is weekly, monthly, quarterly, or year-end processing of financial, accounting, payroll, inventory, or enterprise resource planning (ERP) schedules. Knowing how and when the applications use the data, which is also understanding the data, automated tools, and policies, can be used to tier or cache data to speed up processing and thereby boost productivity.

All applications have performance, availability, capacity, economic (PACE) attributes, however:

PACE attributes vary by Application Data Value and usage
Some applications and their data are more active than others
PACE characteristics may vary within different parts of an application
PACE application and data characteristics along with value change over time

Read more about Application Data Value, PACE and application characteristics in Software Defined Data Infrastructure Essentials (CRC Press 2017).

Where to learn more

Part 1 – Application Data Value Characteristics Everything Is Not The Same
Part 2 – 4 3 2 1 Data Protection Application Data Availability
Part 3 – Application Data Characteristics Types Everything Is Not The Same
Part 4 – Application Data Volume Velocity Variety Everything Is Not The Same
Part 5 – Application Data Access Lifecycle Patterns Everything Not The Same
Data Infrastructure server storage I/O network Recommended Reading
World Backup Day 2018 Data Protection Readiness Reminder
Data Infrastructure Server Storage I/O related Tradecraft Overview
Data Infrastructure Overview, Its What’s Inside of Data Centers
4 3 2 1 and 3 2 1 data protection best practices
Garbage data in, garbage information out, big data or big garbage?
GDPR (General Data Protection Regulation) Resources Are You Ready?
Which Enterprise HDD to use for a Content Server Platform
The SSD Place (SSD, NVM, PM, SCM, Flash, NVMe, 3D XPoint, MRAM and related topics)
The NVMe Place (NVMe related topics, trends, tools, technologies, tip resources)
Data Protection Diaries (Archive, Backup/Restore, BC, BR, DR, HA,Replication, Security)

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What this all means and wrap-up

Keep in mind that Application Data Value everything is not the same across various organizations, data centers, data infrastructures, data and the applications that use them.

Also keep in mind that there is more data being created, the size of those data items, files, objects, entities, records are also increasing, as well as the speed at which they get created and accessed. The challenge is not just that there is more data, or data is bigger, or accessed faster, it’s all of those along with changing value as well as diverse applications to keep in perspective. With new Global Data Protection Regulations (GDPR) going into effect May 25, 2018, now is a good time to assess and gain insight into what data you have, its value, retention as well as disposition policies.

Remember, there are different data types, value, life-cycle, volume and velocity that change over time, and with Application Data Value Everything Is Not The Same, so why treat and manage everything the same?

Ok, nuff said, for now.

January 25, 2018November 26, 2023

Data Infrastructure Resource Links cloud data protection tradecraft trends

Data Infrastructure Resource Links Server Storage I/O Network

By Greg Schulz – www.storageioblog.com April 28, 2018

Various data infrastructure resource links.

The following are a collection of server storageioblog data infrastructure resource links.

Where to learn more

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What This All Means

Check out the above links to data infrastructure resource links.

Ok, nuff said, for now.

December 7, 2017November 3, 2024

November 2017 Server StorageIO Data Infrastructure Update Newsletter

Volume 17, Issue 11 (November 2017)

Hello and welcome to the November 2017 issue of the Server StorageIO update newsletter.

2017 has a few more weeks left which look to be busy with end of year, holidays and other activities. Like the rest of 2017 November saw a lot of activity in and around the industry, setting up 2018 as yet another sequel to the busiest and most exciting year ever.

This is also the time of year when predictions for the following year (e.g. 2018) start to roll out, some of which are variations from those of the past or perennial favorites (e.g. the year of flash, the year of cloud, the year of software defined, the year of <insert_your_favorite_item_here>. Look for predictions and perspectives in future posts and newsletters.

Having been a busy month, let’s get to the content…

Recommended Reading
Various Events and Webinars
Industry Resources and Links

Enjoy this edition of the Server StorageIO data infrastructure update newsletter.

Cheers GS

Data Infrastructure and IT Industry Activity Trends

Some recent Industry Activities, Trends, News and Announcements include:

On the heals of completing its acquisition of Brocade (note previously Avago (who bought LSI) also bought Broadcom and then changed its name to the more well-known entity. Broadcom also announced relocating it headquarters from Singapore to the US, along an over $100 Billion USD acquisition offer of Qualcomm (here is interesting perspective Apple might play). Broadcom has been focused more on server, storage, I/O and general networking technology, while Qualcomm on mobile including phones and related items. Note that Qualcomm has previously made a $38.5 Billion USD offer for NXP semiconductors waiting regularity approval. View recent Broadcom financial results here.

Also in November server storage I/O controller chip maker Marvell (not to be confused with entertainment provider Marvel) announced a merger with Cavium who had previously acquired Qlogic among others. The resulting combined entity to be called Marvell will have an estimated $16 Billion USD revenue stream focused on server, storage, I/O and networking technologies among others.

In other merger and acquisition activity, VMware announced acquisition of VeloCloud for software defined wide area networking (SD-WAN).

With Super Compute 2017 (SC17) in November there were several announcements including from ATTO, DDN, Enmotus and Micron, Everspin, along with many others. By the way, in case you missed it at end of October Microsoft and Cray announced a partnership to bring Super Compute capabilities to Azure clouds. Speaking of Microsoft, there was also an announcement of adding VMware running on top of Azure (granted without VMware support), similar in concept to VMware on AWS (read hare).

Also at the end of November was AWS Reinvent with many announcements (more on those in a follow-up newsletter and posts). Prior to Reinvent AWS announced several server, storage and other data infrastructure security enhancements including for S3. Highlights from AWS reinvent include Fargate (serverless aka containers at scale without managing infrastructure), Elastic Container Services for Kubernetes (EKS), Greengrass (machine learning [ML] data infrastructure), along with many others.

Fargate is for those who want to leverage serveless microservices containers without having to devote DevOps and related activity to the care and feeding of its data infrastructure. In other words, Fargate is for those who want to focus maximum effort on the business applications, vs. the business of setting up and maintaining the data infrastructure for serverless On the other hand, AWS also announced EKS for those who want or need to customize their serverless data infrastructure including around Kubernetes among others.

In other industry activity, Taiwanese based Foxconn who manufactures technology for the who’s who of the industry announced progress towards their future Wisconsin based factory complex.

Over at HPE, the big news announcement is that CEO Meg Whitman is stepping down. HPE also announced new AMD powered Gen 10 Proliant services, as well as multi-cloud management solutions. HPE also announced new partnerships with DDN for HPC and SC, with Rackspace for selling private cloud services, along with Cloudian EMEA partnership among others.

OwnBackup announced a new version of their data protection software, while low-cost budget bulk storage service backblaze (B2) announced their more recent quarterly drive failure (or success) reliability reports. Meanwhile over at Quantum they released former Ceo Jon Gacek and rotated in new management.

Red Hat announced Ceph Storage 3 including CephFS (POSIX compatible file system), iSCSI gateway including support for VMware and Windows that lack native Ceph drivers, daemon deployment in Linux containers for smaller hardware footprint. Also included are enhanced monitoring, troubleshooting and diagnostics to streamline deployment and ongoing management. Red Hat also announced OpenShift version 3.7 for containers.

SANblaze announced NVMf and dual port NVMe capabilities for NVMe fabrics, while Linbit won an European grant to build out a software defined storage cloud scale out solution.

I often get asked who are the hot, new, trendy or other vendors and services to keep an eye on some of which I have mentioned in previous newsletters, as well as posts such as here and here. Moving in to 2018 some to keep an eye on (not all are new or trendy, yet they can enable you to be productive, or differentiate) include the following.

AWS, Bluemedora, Chelsio, Cloudian, CloudPassage, Compuverde, Databricks, Datadog, Datos, Enmotus, Everspin, Excelero, Fluree (Blockchain database), Google, Mellonox, Microsemi, Microsoft, Marvel and Cavium, MyWorkDrive, Red Hat, Rook, Rozo, Rubrik, Strongbox, Storone, Turbonomic, Ubuntu, Veeam, Velostrata, Virtuozo, VMware, WekaIO and others.

What the above means, is that it has been a busy month as well as year, and, the year is not over yet. There are still plenty of shopping days left both for christmas and the holidays, as well as for IT year-end spending, vendors looking to do acquisitions, or other last-minute projects. Speaking of which, drop me a note if you have any end of year, or new year projects Server StorageIO can assist you with.

Check out other industry news, comments, trends perspectives here.

Server StorageIO Commentary in the news, tips and articles

Recent Server StorageIO industry trends perspectives commentary in the news.

Via HPE Insights: Comments on Public cloud versus on-prem storage
Via DataCenterKnowledge: Data Center Standards: Where’s the Value?
Via arsTechnica: Comments on cloud backup disaster recovery

View more Server, Storage and I/O trends and perspectives comments here

Server StorageIOblog Data Infrastructure Posts

Recent and popular Server StorageIOblog posts include:

IT transformation Serverless Life Beyond DevOps Podcast
In this Server StorageIO podcast episode New York Times CTO / CIO Nick Rockwell (@nicksrockwell) joins me for a conversation discussing Digital, Business and IT transformation, Serverless Life Beyond DevOps and related topics.Read more here.

Data Protection Diaries Fundamental Topics Tools Techniques Technologies Tips
This is a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).
Posts in this series include:
- Part 1 – Data Infrastructure Data Protection Fundamentals
- Part 2 – Reliability, Availability, Serviceability ( RAS) Data Protection Fundamentals
- Part 3 – Access Availability RAID Erasure Codes ( EC) including LRC
- Part 4 – Data Protection Recovery Points (Archive, Backup, Snapshots, Versions)
- Part 5 – Point In Time Data Protection Granularity Points of Interest
- Part 6 – Data Protection Security Logical Physical Software Defined
- Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends
- Part 8 – Data Protection Diaries Walking Data Protection Talk
- Part 9 – who’s Doing What ( Toolbox Technology Tools)
- Part 10 – Data Protection Resources Where to Learn More
Read more here.

HPE Announces AMD Powered Gen 10 ProLiant DL385 For Software Defined Workloads
HPE Announced a new AMD EPYC 7000 Powered Gen 10 ProLiant DL385 for Software Defined Workloads including server virtualization, software-defined data center (SDDC), software-defined data infrastructure (SDDI), software-defined storage among others. These new servers are part of a broader Gen10 HPE portfolioof ProLiant DL systems. Read more here.

AWS Announces New S3 Cloud Storage Security Encryption Features
Amazon Web Services (AWS) recently announced new Simple Storage Service (S3) encryption and security enhancements including Default Encryption, Permission Checks, Cross-Region Replication ACL Overwrite, Cross-Region Replication with KMS and Detailed Inventory Report. Another recent announcement by AWS is for PrivateLinks endpoints within a Virtual Private Cloud (VPC). Read more here.

Server StorageIO Recommended Reading (Watching and Listening) List

In addition to my own books including Software Defined Data Infrastructure Essentials (CRC Press 2017), the following are Server StorageIO data infrastructure recommended reading, watching and listening list items. The list includes various IT, Data Infrastructure and related topics. Speaking of my books, Didier Van Hoye (@WorkingHardInIt) has a good review over on his site you can view here, also check out the rest of his great content while there.

Intel Recommended Reading List (IRRL) for developers is a good resource to check out.

For those who are into Linux, container and hypervisor performance along with internals including cloud based, check out Brendan Gregg site. He has a lot of great material including some recent interesting posts ranging from dealing with workplace jerks, to whats inside AWS EC2 new KVM (switch from Xen based) hypervisors among others.

Here is a post by New York Times CIO/CTO Nick Rockwell The (Futile) Resistance to Serverless, also check out my podcast discussion with Nick here.

Over at Next Platform they have some interesting perspectives on Intel’s next Exascale architecture worth spending a few minutes to read.

Watch for more items to be added to the recommended reading list book shelf soon.

Events and Activities

Recent and upcoming event activities.

Nov. 9, 2017 – Webinar – All You Need To Know about ROBO Data Protection Backup
Nov. 2, 2017 – Webinar – Modern Data Protection for Hyper-Convergence

See more webinars and activities on the Server StorageIO Events page here.

Server StorageIO Industry Resources and Links

Useful links and pages:
Data Infrastructure Recommend Reading and watching list
Microsoft TechNet – Various Microsoft related from Azure to Docker to Windows
storageio.com/links – Various industry links (over 1,000 with more to be added soon)
objectstoragecenter.com – Cloud and object storage topics, tips and news items
OpenStack.org – Various OpenStack related items
storageio.com/downloads – Various presentations and other download material
storageio.com/protect – Various data protection items and topics
thenvmeplace.com – Focus on NVMe trends and technologies
thessdplace.com – NVM and Solid State Disk topics, tips and techniques
storageio.com/converge – Various CI, HCI and related SDS topics
storageio.com/performance – Various server, storage and I/O benchmark and tools
VMware Technical Network – Various VMware related items

Connect and Converse With Us

Ok, nuff said, for now.

November 26, 2017November 26, 2023

Data Protection Diaries Reliability, Availability, Serviceability RAS Fundamentals

Reliability, Availability, Serviceability RAS Fundamentals

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulz – www.storageioblog.com November 26, 2017

This is Part 2 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Click here to view the previous post Part 1 Data Infrastructure Data Protection Fundamentals, and click here to view the next post Part 3 Data Protection Access Availability RAID Erasure Codes (EC) including LRC.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection availability from Chapter 9 which includes access, durability, RAS, RAID and Erasure Codes (including LRC), mirroring and replication along with related topics.

Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

Reliability, Availability, Serviceability (RAS) Data Protection Fundamentals

Reliability, Availability Serviceability (RAS) and other access availability along with Data Protection topics are covered in chapter 9. A resilient data infrastructure (software-defined, SDDC and legacy) protects, preserves, secures and serves information involving various layers of technology. These technologies enable various layers ( altitudes) of functionality, from devices up to and through the various applications themselves.

SDDI SDDC Data Protection Big Picture
Figure 9.2 Various threat issues and challenges that drive the need for data protection

Some applications need a faster rebuild, while others need sustained performance (bandwidth, latency, IOPs, or transactions) with the slower rebuild; some need lower cost at the expense of performance; others are ok with more space if other objectives are meet. The result is that since everything is different yet there are similarities, there is also the need to tune how data Infrastructure protects, preserves, secures, and serves applications and data.

General reliability, availability, serviceability, and data protection functionality includes:

Manually or automatically via policies, start, stop, pause, resume protection
Adjust priorities of protection tasks, including speed, for faster or slower protection
Fast-reacting to changes, disruptions or failures, or slower cautious approaches
Workload and application load balancing (performance, availability, and capacity)

RAS can be optimized for:

Reduced redundancy for lower overall costs vs. resiliency
Basic or standard availability (leverage component plus)
High availability (use better components, multiple systems, multiple sites)
Fault-tolerant with no single points of failure (SPOF)
Faster restart, restore, rebuild, or repair with higher overhead costs
Lower overhead costs (space and performance) with lower resiliency
Lower impact to applications during rebuild vs. faster repair
Maintenance and planned outages or for continues operations

Common availability Data Protection related terms, technologies, techniques, trends and topics pertaining to data protection from availability and access to durability and consistency to point in time protection and security are shown below.

Data Protection Gaps and Air Gap

There are Good Data Protection Gaps that provide recovery points to a past time enabling recoverability in the future to move forward. Another good data protection gap is an Air Gap that isolates protection copies off-site or off-line so that they can not be tampered with enabling recovery from ransomware and other software defined threats. There are Bad data protection gaps including gaps in coverage where data is not protected or items are missing. Then there are Ugly data protecting gaps which include Bad gaps that result in what you think is protected are not and finding that your copies are bad when it is too late.

Data Protection Gaps Good Bad and Ugly

The following figure shows good data protection gaps including recovery points (point in time protection) along with air gaps.

Good Data Protection Gaps
Figure 9.9 Air Gaps and Data Protection

Fault / Failures To Tolerate (FTT)

FTT is how many faults or failures to tolerate for a given solution or service which in turn determines what mode of protection, or fault tolerant mode ( FTM) to use.

Fault Tolerant Mode (FTM)

FTM is the mode or technique used to enable resiliency and protect against some number of faults.

Fault / Failure Domains

Fault or Failure domains are places and things that can fail from regions, data centers or availability zones, clusters, stamps, pods, servers, networks, storage, hardware (systems, components including SSD and HDDs, power supplies, adapters). Other fault domain topics and focus areas include facility power, cooling, software including applications, databases, operating systems and hypervisors among others.

SDDI SDDC Fault Domains Zones Regions
Figure 9.5 Various Fault and Failure Domains, Regions, Locations

Clustering

Clustering is a technique and technology for enabling resiliency, as well as scaling performance, availability, and capacity. Clusters can be local, remote, or wide-area to support different data infrastructure objectives, combined with replication and other techniques.

SDDI SDDC Clustering
Figure 9.12 Clustering and Replication Examples

Another characteristic of clustering and resiliency techniques is the ability to detect and react quickly to failures to isolate and contain faults, as well as invoking automatic repair if needed. Different clustering technologies enable various approaches, from proprietary hardware and software tightly coupled to loosely coupled general-purpose hardware or software.

Clustering characteristics include:

Application, database, file system, operating system (Windows Storage Replica)
Storage systems, appliances, adapters and network devices
Hypervisors ( Hyper-V, VMware vSphere ESXi and vSAN among others)
Share everything, share some things, share nothing
Tightly or loosely coupled with common or individual system metadata
Local in a data center, campus, metro, or stretch cluster
Wide-area in different regions and availability zones
Active/active for fast fail over or restart, active/passive (standby) mode

Additional clustering considerations include:

How does performance scale as nodes are added, or what overhead exists?
How is cluster resource locking in shared environments handled?
How many (or few) nodes are needed for quorum to exist?
Network and I/O interface (and management) requirements
Cluster partition or split-brain (i.e., cluster splits into two)?
Fast-reacting fail over and resiliency vs. overhead of failing back
Locality of where applications are located vs. storage access and clustering

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Part 1 – Data Infrastructure Data Protection Fundamentals
Part 2 – Reliability, Availability, Serviceability ( RAS) Data Protection Fundamentals
Part 3 – Data Protection Access Availability RAID Erasure Codes ( EC) including LRC
Part 4 – Data Protection Recovery Points (Archive, Backup, Snapshots, Versions)
Part 5 – Point In Time Data Protection Granularity Points of Interest
Part 6 – Data Protection Security Logical Physical Software Defined
Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends
Part 8 – Data Protection Diaries Walking Data Protection Talk
Part 9 – who’s Doing What ( Toolbox Technology Tools)
Part 10 – Data Protection Resources Where to Learn More
Data Protection Diaries series
Data Infrastructure server storage I/O network Recommended Reading List Book Shelf
Software Defined Data Infrastructure Essentials (CRC 2017) Book

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What This All Means

Everything is not the same across different environments, data centers, data infrastructures and applications. There are various performance, availability, capacity economic (PACE) considerations along with service level objectives (SLO). Availability means being able to access information resources (applications, data and underlying data infrastructure resources), as well as data being consistent along with durable. Being durable means enabling data to be accessible in the event of a device, component or other fault domain item failures (hardware, software, data center).

Just as everything is not the same across different environments, there are various techniques, technologies and tools that can be used in different ways to enable availability and accessibility. These include high availability (HA), RAS, mirroring, replication, parity along with derivative erasure code (EC), LRC, RS and other RAID implementations, along with clustering. Also keep in mind that pertaining to data protection, there are good gaps (e.g. time intervals for recovery points, air gaps), bad gaps (missed coverage or lack of protection), and ugly gaps (not being able to recover from a gap in time).

Note that mirroring, replication, EC, LRC, RS or other Parity and RAID approaches are not replacements for backup, rather they are companions to time interval based recovery point protection such as snapshots, backup, checkpoints, consistency points and versioning among others (discussed in follow-up posts in this series).

Which data protection tool, technology to trend is the best depends on what you are trying to accomplish and your application workload PACE requirements along with SLOs. Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 3 Data Protection Access Availability RAID Erasure Codes (EC) including LRC.

Ok, nuff said, for now.

October 18, 2017November 26, 2023

Cloud Conversations AWS Azure Service Maps via Microsoft

server storage I/O data infrastructure trends

Updated 1/21/2018

Microsoft has created an Amazon Web Service AWS Azure Service Map. The AWS Azure Service Map is a list created by Microsoft looks at corresponding services of both cloud providers.

Image via Azure.Microsoft.com

Note that this is an evolving work in progress from Microsoft and use it as a tool to help position the different services from Azure and AWS.

Also note that not all features or services may not be available in different regions, visit Azure and AWS sites to see current availability.

As with any comparison they are often dated the day they are posted hence this is a work in progress. If you are looking for another Microsoft created why Azure vs. AWS then check out this here. If you are looking for an AWS vs. Azure, do a simple Google (or Bing) search and watch all the various items appear, some sponsored, some not so sponsored among others.

Whats In the Service Map

The following AWS and Azure services are mapped:

Marketplace (e.g. where you select service offerings)
Compute (Virtual Machines instances, Containers, Virtual Private Servers, Serverless Microservices and Management)
Storage (Primary, Secondary, Archive, Premium SSD and HDD, Block, File, Object/Blobs, Tables, Queues, Import/Export, Bulk transfer, Backup, Data Protection, Disaster Recovery, Gateways)
Network & Content Delivery (Virtual networking, virtual private networks and virtual private cloud, domain name services (DNS), content delivery network (CDN), load balancing, direct connect, edge, alerts)
Database (Relational, SQL and NoSQL document and key value, caching, database migration)
Analytics and Big Data (data warehouse, data lake, data processing, real-time and batch, data orchestration, data platforms, analytics)
Intelligence and IoT (IoT hub and gateways, speech recognition, visualization, search, machine learning, AI)
Management and Monitoring (management, monitoring, advisor, DevOps)
Mobile Services (management, monitoring, administration)
Security, Identity and Access (Security, directory services, compliance, authorization, authentication, encryption, firewall
Developer Tools (workflow, messaging, email, API management, media trans coding, development tools, testing, DevOps)
Enterprise Integration (application integration, content management)

Down load a PDF version of the service map from Microsoft here.

Where To Learn More

Learn more about related technology, trends, tools, techniques, and tips with the following links.

What’s a data infrastructure? (Via NetworkWorld)
Introduction to Azure Files (Microsoft Docs)
Public preview of Virtual Network service endpoints for Azure storage and SQL database
Azure debuts Availability Zones for high availability and resiliency (Azure)
Azure and AWS
comparison (Docs Microsoft)
Cloud Storage Considerations (via WServerNews)
Microsoft Windows Server, Azure, Nano Life cycle Updates
Overview Review of Microsoft ReFS (Reliable File System) and resource links
Software Defined Data Infrastructure Essentials (CRC Press) book companion page
Amazon Web Service AWS September 2017 Updates
AWS S3 Storage Gateway Revisited (Part I)
Cloud conversations: AWS EBS, Glacier and S3 overview (Part II S3)
Microsoft September 2017 Updates

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What This All Means

On one hand this can and will likely be used as a comparison however use caution as both Azure and AWS services are rapidly evolving, adding new features, extending others. Likewise the service regions and site of data centers also continue to evolve thus use the above as a general guide or tool to help map what service offerings are similar between AWS and Azure.

By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

Ok, nuff said, for now.

August 26, 2017November 26, 2023

Hot Popular New Trending Data Infrastructure Vendors To Watch

server storage I/O data infrastructure trends

Updated 1/21/2018

A common question I get asked is who are the hot popular new trending data infrastructure vendors to watch. This post looks at some data infrastructure vendors to watch and keep an eye on.

Keep in mind that there is a difference between industry adoption and customer deployment, the former being what the industry (e.g. Vendors, resellers, integrators, investors, consultants, analyst, press, media, analysts, bloggers or other influences) like, want and need to talk about. Then there is customer adoption and deployment which is what is being bought, installed and used.

Some Popular Trending Vendors To Watch

The following is far from an exhaustive list however here are some that come to mind that I’m watching.

Apcera – Enterprise class containers and management tools
AWS – Rolls our new services like a startup with size momentum of a legacy player
Blue Medora – Data Infrastructure insight, software defined management
Broadcom – Avago/LSI, legacy Broadcom, Emulex, Brocade acquisition interesting portfolio
Chelsio – Server, storage and data Infrastructure I/O technologies
Commvault – Data protection and backup solutions
Compuverde – Software defined storage
Data Direct Networks (DDN) – Scale out and high performance storage
Datadog – Software defined management, data infrastructure insight, analytics, reporting
Datrium – Converged software defined data infrastructure solutions
Dell EMC Code – Rexray container persistent storage management
Docker – Container and management tools
E8 Storage – NVMe based storage solutions
Elastifile – Scale out software defined storage and file system
Enmotus – MicroTiering that works with Windows, Linux and various cloud platforms
Everspin – storage class memories and NVDIMM
Excelero – NVMe based storage
Hedvig – Scale out software defined storage
Huawei – While not common in the US, in Europe and elsewhere they are gaining momentum
Intel – Watch what they do with Optane and storage class memories
Kubernetes – Container software defined management
Liqid – Stealth Colorado startup focusing on PCIe fabrics and composable infrastructure
Maxta – Hyper converged infrastructure (HCI) and software defined data infrastructure vendor
Mellanox – While not a startup, keep an eye on what they are doing with their adapters
Micron – Watch what they do with 3D XPoint storage class memory and SSD
Microsoft – Not a startup, however keep an eye on Azure, Azure Stack, Window Server with S2D, ReFS, tiering, CI/HCI as well as Linux services on Windows.
Minio – Software defined storage solutions
NetApp – While FAS/Ontap and Solidfire get the headlines, E series generates revenue, keep an eye on StorageGrid and AltaVault
Neuvector – Container management and security
Noobaa – Software defined storage and more
NVIDA – No longer just another graphics process unit based company
Pivot3 – An original HCI software defined players, granted, some of their competitors might not think so
Pluribus Networks – Software Defined Networks for Software Defined Data Infrastructures
Portwork – Container management and persistent storage
Rozo Systems – Scale out software defined storage and file system
Rubrik – Data Protection software, reminds me of a startup called Commvault 20 years ago.
ScaleMP – Composable scale out software defined servers
Storpool – Scale out software defined storage
Stratoscale – Software defined data infrastructure and hybrid solutions
SUSE – Linux distribution looking to expand their offerings, gain more insight
Tidalscale – Composable software defined data infrastructures
Turbonomic – Software Defined Management, insight, analytics and automation
Ubuntu – Known for their Linux distribution, check out their Metal as a Service (MaaS) technology
Veeam – Data protection and backup solutions
technology
Virtuozzo – Software defined storage and data infrastructure technologies
VMware – AWS, vSAN, NSX, Integrated Containers and much more
WekaIO – Scale out software defined storage and file system

Some Popular Trending Technology Trends

ARM, ASIC, FPGA, GPU servers among others
Converged Infrastructure (CI), Hyper Converged Infrastructure (HCI), Composable Infrastructure
Analytics, reporting, insight, machine learning (ML), artificial intelligence (AI), automation
Software Defined including Cloud, Virtual, Containers, Docker, kubernetes, mesos, serverless, micro services
Data protection, backup/restore, archive, security, business resiliency (BR), business continuance (BC), disaster recovery (DR)
Non-volatile memory (NMV), NVM Express (NVMe), storage class memories (SCM), persistent memory, nand flash, SSD

Where To Learn More

Learn more about related technology, trends, tools, techniques, and tips with the following links.

Server StorageIO IT Data Infrastructure Industry Links A to E
Server Storage I/O Benchmarking Performance Resource Tools
What’s a data infrastructure? (Via NetworkWorld)
Software Defined Data Infrastructure Essentials (CRC Press) book companion page (includes various images, sample figures and added content)
Object and Cloud Storage Center (www.objectstoragecenter.com)
www.thessdplace.com – NVM, flash, SSD, SCM and related topics
www.thenvmeplace.com – NVM Express (NVMe) related topics

Various IT and Cloud Infrastructure Layers including Data Infrastructures

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

What This All Means

There are always more hot popular new or trending data infrastructure vendors to watch, which ones are you keeping an eye on?

Ok, nuff said, for now.

August 14, 2017March 7, 2022

Like Data They Protect For Now Quantum Revenues Continue To Grow

For Now Quantum Revenues Continue To Grow

server storage I/O data infrastructure trends

For Now Quantum Revenues Continue To Grow. The other day following their formal announced, I received an summary update from Quantum pertaining to their recent Q1 Results (show later below).

Various IT and Cloud Infrastructure Layers including Data Infrastructures

Quantums Revenues Continue To Grow Like Data

One of the certainties in life is change and the other is continued growth in data that gets transformed into information via IT and other applications. Data Infrastructures fundamental role is to enable an environment for applications and data to be transformed into information and delivered as services. In other words, Data Infrastructures exist to protect, preserve, secure and serve information along with the applications and data they depend on. Quantums role is to provide solutions and technologies for enabling legacy and cloud or other software defined data infrastructures to protect, preserve, secure and serve data.

What caught my eye in Quantums announcements was that while not earth shattering growth numbers normally associated with a hot startup, being a legacy data infrasture and storage vendor, Quantum’s numbers are hanging in there.

At a time when some legacy as well as startups struggle with increased competition from others including cloud, Quantum appears for at least now to be hanging in there with some gains.

The other thing that caught my eye is that most of the growth not surprisingly is non tape related solutions, particular around their bulk scale out StorNext storage solutions, there is some growth in tape.

Here is the excerpt of what Quantum sent out:


Highlights for the quarter (all comparisons are to the same period a year ago):

•	Grew total revenue and generated profit for 5th consecutive quarter
•	Total revenue was up slightly to $117M, with 3% increase in branded revenue
•	Generated operating profit of $1M with earnings per share of 4 cents, up 2 cents
•	Grew scale-out tiered storage revenue 10% to $34M, with strong growth in video surveillance and technical workflows
o	Key surveillance wins included deals with an Asian government for surveillance at a presidential palace and other government facilities, with a major U.S. port and with four new police department customers
o	Established several new surveillance partnerships – one of top three resellers/integrators in China (Uniview) and two major U.S. integrators (Protection 1 and Kratos)
o	Won two surveillance awards for StorNext – Security Industry Association’s New Product Showcase award and Security Today magazine’s Platinum Govies Government Security award
o	Key technical workflow wins included deals at an international defense and aerospace company to expand StorNext archive environment, a leading biotechnology firm for 1 PB genomic sequencing archive, a top automaker involving autonomous driving research data and a U.S. technology institute involving high performance computing  
o	Announced StorNext 6, which adds new advanced data management features to StorNext’s industry-leading performance and is now shipping
o	Announced scale-out partnerships with Veritone on artificial intelligence and DataFrameworks on data visualization and management  
•	Tape automation, devices and media revenue increased 6% overall while branded revenue for this product category was up 14%
o	Strong sales of newest generation Scalar i3 and i6 tape libraries
•	Established new/enhanced data protection partnerships
o	Enhanced partnership with Veeam, making it easier for their customers to deploy 3-2-1 data protection best practices
o	Became Pure Storage alliance partner, providing our data protection and archive solutions for their customers through mutual channel partners

Where To Learn More

Learn more about related technology, trends, tools, techniques, and tips with the following links.

Modernizing Data Protection = Using new and old things in new ways
HDD topics and what to use for bulk and content storage
Like IT Data Centers Do You Take Trade Show Exhibit Infrastructure For Granted?
Zombie Technology Life after Death Tape Is Still Alive
Who Will Be At Top Of Storage World Next Decade?
NVMe related and flash SSD along with cloud, bulk, object storage topics
Software Defined Data Infrastructure Essentials (CRC Press) that provides mover coverage of NVM and NVMe among other topics.

What This All Means

Keep in mind that Data Infrastructures fundamental role is to enable an environment for applications and data to be transformed into information and delivered as services. Data Infrastructures exist to protect, preserve, secure and serve information along with the applications and data they depend on. Quantum continues to evolve their business as they have for several years from one focused on tape and related technologies to one that includes tape as well as many other solutions for legacy as well as software defined, cloud and virtual environments. For now, quantum revenues continue to grow and diversify.

Ok, nuff said, for now.
Gs

Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

July 24, 2017November 26, 2023

Zombie Technology Life after Death Tape Is Still Alive

server storage I/O data infrastructure trends

A Zombie Technology is one declared dead yet has Life after Death such as Tape which is still alive.

zombie technology
Image via StorageIO.com (licensed for use from Shutterstock.com)

Tapes Evolving Role

Sure we have heard for decade’s about the death of tape, and someday it will be dead and buried (I mean really dead), no longer used, buried, existing only in museums. Granted tape has been on the decline for some time, and even with many vendors exiting the marketplace, there remains continued development and demand within various data infrastructure environments, including software defined as well as legacy.

data infrastructures

Tape remains viable for some environments as part of an overall memory data storage hierarchy including as a portability (transportable) as well as bulk storage medium.

memory data storage hirearchy classes tiers

Keep in mind that tapes role as a data storage medium also continues to change as does its location. The following table (via Software Defined Data Infrastructure Essentials (CRC Press)) Chapter 10 shows examples of various data movements from source to destination. These movements include migration, replication, clones, mirroring, and backup, copies, among others. The source device can be a block LUN, volume, partition, physical or virtual drive, HDD or SSD, as well as a file system, object, or blob container or bucket. An example of the modes in Table 10.1 include D2D backup from local to local (or remote) disk (HDD or SSD) storage or D2D2D copy from local to local storage, then to the remote.

Mode – Description
D2D – Data gets copied (moved, migrated, replicated, cloned, backed up) from source storage (HDD or SSD) to another device or disk (HDD or SSD)-based device
D2C – Data gets copied from a source device to a cloud device.
D2T – Data gets copied from a source device to a tape device (drive or library).
D2D2D – Data gets copied from a source device to another device, and then to another device.
D2D2T – Data gets copied from a source device to another device, then to tape.
D2D2C Data gets copied from a source device to another device, then to cloud.
Data Movement Modes from Source to Destination

Note that movement from source to the target can be a copy, rsync, backup, replicate, snapshot, clone, robocopy among many other actions. Also, note that in the earlier examples there are occurrences of tape existing in clouds (e.g. its place) and use changing. Tip – In the past, “disk” usually referred to HDD. Today, however, it can also mean SSD. Think of D2D as not being just HDD to HDD, as it can also be SSD to SSD, Flash to Flash (F2F), or S2S among many other variations if you prefer (or needed).

Image via Tapestorage.org

For those still interested in tape, check out the Active Archive Alliance recent posts (here), as well as the 2017 Tape Storage Council Memo and State of their industry report (here). While lower end-tape such as LTO (which is not exactly the low-end it was a decade or so ago) continues to evolve, things may not be as persistent for tape at the high-end. With Oracle (via its Sun/StorageTek acquisition) exiting the high-end (e.g. Mainframe focused) tape business, that leaves mainly IBM as a technology provider.

Image via Tapestorage.org

With a single tape device (e.g. drive) vendor at the high-end, that could be the signal for many organizations that it is time to finally either move from tape or at least to LTO (linear tape open) as a stepping stone (e.g. phased migration). The reason not being technical rather business in that many organizations need to have a secondary or competitive offering or go through an exception process.

On the other hand, just as many exited the IBM mainframe server market (e.g. Fujitsu/Amdahl, HDS, NEC), big blue (e.g. IBM) continues to innovate and drive both revenue and margin from those platforms (hardware, software, and services). This leads me to believe that IBM will do what it can to keep its high-end tape customers supported while also providing alternative options.

Where To Learn More

Learn more about related technology, trends, tools, techniques, and tips with the following links.

Active Archive Alliance Blog Post
Changing and evolving role of tape (PDF)
HDD topics and what to use for bulk and content storage
NVMe related and flash SSD along with cloud, bulk, object storage topics
Software Defined Data Infrastructure Essentials (CRC Press)
Tape Council website and 2017 State of Tape Memo (PDF)
Tape Talk and Related Topics
Thread regarding Oracle Corp. layoffs
GDPR (General Data Protection Regulation) Resources Are You Ready?
Welcome to the Data Protection Diaries
Via Register Did Oracle just sign tape’s death warrant? Depends what ‘no comment’ means
What is a Zombie Technology?

What This All Means

I would not schedule the last tape funeral just yet, granted there will continue to be periodic wakes and send off over the coming decade. Tape remains for some environments a viable data storage option when used in new ways, as well as new locations complementing flash SSD and other persistent memories aka storage class memories along with HDD.

Personally, I have been directly tape free for over 14 years. Granted, I have data in some clouds and object storage that may exist on a very cold data storage tier possibly maybe on tape that is transparent to my use. However just because I do not physically have tape, does not mean I do not see the need why others still have to or prefer to use it for different needs.

Also, keep in mind that tape continues to be used as an economic data transport for bulk movement of data for some environments. Meanwhile for those who only want, need or wish tape to finally go away, close your eyes, click your heels together and repeat your favorite tape is not alive chant three (or more) times. Keep in mind that HDDs are keeping tape alive by off loading some functions, while SSDs are keeping HDDs alive handling tasks formerly done by spinning media. Meanwhile, tape can and is still called upon by some organizations to protect or enable bulk recovery for SSD and HDDs even in cloud environments, granted in new different ways.

What this all means is that as a zombie technology having been declared dead for decades yet still live there is life after death for tape, which is still alive, for now.

Ok, nuff said (for now…).

Cheers
Gs

PACE your Server Storage I/O decision making, its about application requirements

PACE and common applications characteristics

Common Application Pace Attributes

Data and Application PACE

Where to learn more

What this all means

Share this:

Data Protection Recovery Life Post World Backup Day Pre GDPR

Expanding Focus Data Protection and GDPR

Expanding Focus Data Protection Recovery and other Things that start with R

Data Protection Tips, Reminders and Recommendations

Where to learn more

What this all means and wrap-up

Share this:

Application Data Value Characteristics Everything Is Not The Same

Common Applications Characteristics

Performance and Activity (How Resources Get Used)

Where to learn more

What this all means and wrap-up

Share this:

Application Data Availability 4 3 2 1 Data Protection

Availability (Accessibility, Durability, Consistency)

Capacity and Space (What Gets Consumed and Occupied)

Economics (People, Budgets, Energy and other Constraints)

Where to learn more

What this all means and wrap-up

Share this:

Application Data Characteristics Types Everything Is Not The Same

Various Types of Data

Different data with various values over time

Data Value

Where to learn more

What this all means and wrap-up

Share this:

Application Data Volume Velocity Variety Everything Not The Same

Volume of Data

Variety of Data

Velocity of Data

Where to learn more

What this all means and wrap-up

Share this:

Application Data Access Life cycle Patterns Everything Is Not The Same(Part V)

Active (Hot), Static (Warm and WORM), or Dormant (Cold) Data and Lifecycles

Where to learn more

What this all means and wrap-up

Share this:

Data Infrastructure Resource Links Server Storage I/O Network

Where to learn more

What This All Means

Share this:

Volume 17, Issue 11 (November 2017)

In This Issue

Data Infrastructure and IT Industry Activity Trends

Server StorageIO Commentary in the news, tips and articles

Server StorageIOblog Data Infrastructure Posts

Server StorageIO Recommended Reading (Watching and Listening) List

Events and Activities

Share this:

Reliability, Availability, Serviceability RAS Fundamentals

Reliability, Availability, Serviceability (RAS) Data Protection Fundamentals

Data Protection Gaps and Air Gap

Fault / Failures To Tolerate (FTT)

Fault Tolerant Mode (FTM)

Fault / Failure Domains

Clustering

Where To Learn More

What This All Means

Share this:

Cloud Conversations AWS Azure Service Maps via Microsoft

Whats In the Service Map

Where To Learn More

What This All Means

Share this:

Hot Popular New Trending Data Infrastructure Vendors To Watch

Some Popular Trending Vendors To Watch

Some Popular Trending Technology Trends

Where To Learn More

What This All Means

Share this:

For Now Quantum Revenues Continue To Grow