Data Infrastructure Data Protection Diaries Fundamental Security Logical Physical

Data Infrastructure Data Protection Security Logical Physical

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is Part 6 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 5 – Point In Time Data Protection Granularity Points of Interest, and click here to view the next post Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Infrastructure and Data Protection security including logical as well as physical from chapter 10 , 13 and 14 among others.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

There are many different aspects of security pertaining to data infrastructures that span various technology domains or focus areas from higher level application software to lower level hardware, from legacy to cloud an software-defined, from servers to storage and I/O networking, logical and physical, from access control to intrusion detection, monitoring, analytics, audit, monitoring, telemetry logs, encryption, digital forensics among many others. Security should not be an after thought of something done independent of other data infrastructure, data center and IT functions, rather integrated.

Security Logical Physical Software Defined

Physical security includes locked doors of facilities, rooms, cabinets or devices to prevent un-authorized access. In addition to locked doors, physical security also includes safeguards to prevent accidental or intentional acts that would compromise the contents of a data center including data Infrastructure resources (servers, storage, I/O networks, hardware, software, services) along with the applications that they support.

Logical security includes access controls, passwords, event and access logs, encryption among others technologies, tools, techniques. Figure 10.11 shows various data infrastructure security–related items from cloud to virtual, hardware and software, as well as network services. Also shown are mobile and edge devices as well as network connectivity between on-premises and remote cloud services. Cloud services include public, private, as well as hybrid and virtual private clouds (VPC) along with virtual private networks (VPN). Access logs for telemetry are also used to track who has accessed what and when, as well as success along with failed attempts.

Certificates (public or private), Encryption, Access keys including .pem and RSA files via a service provider or self-generated with a tool such as Putty or ssh-keygen among many others. Some additional terms including Two Factor Authentication (2FA), Subordinated, Role based and delegated management, Single Sign On (SSO), Shared Access Signature (SAS) that is used by Microsoft Azure for access control, Server Side Encryption (SSE) with various Key Management System (KMS) attributes including customer managed or via a third-party.

SDDC SDDI Data Protection Security
Figure 10.11 Various physical and logical security and access controls

Also shown in figure 10.11 are encryption enabled at various layers, levels or altitude that can range from simple to complex. Also shown are iSCSI IPsec and CHAP along with firewalls, Active Directory (AD) along with Azure AD (AAD), and Domain Controllers (DC), Group Policies Objects (GPO) and Roles. Note that firewalls can exist in various locations both in hardware appliances in the network, as well as software defined network (SDN), network function virtualization (NFV), as well as higher up.

For example there are firewalls in network routers and appliances, as well as within operating systems, hypervisors, and further up in web blogs platforms such as WordPress among many others. Likewise further up the stack or higher in altitude access to applications as well as database among other resources is also controlled via their own, or in conjunction with other authentication, rights and access control including ADs among others.

A term that might be new for some is attestation which basically means to authenticate and be validated by a server or service, for example, a host guarded server attests with a attestation server. What this means is that the host guarded server (for example Microsoft Windows Server) attests with a known attestation server, that looks at the Windows server comparing it to known good fingerprints, profiles, making sure it is safe to run as a guarded resources.

Other security concerns for legacy and software defined environments include secure boot, shield VMs, host guarded servers and fabrics (networks or clusters of servers) for on-premises, as well as cloud. The following image via Microsoft shows an example of shielded VMs in a Windows Server 2016 environment along with host guarded service (HGS) components ( see how to deploy here).


Via Microsoft.com Guarded Hosts, Shielded VMs and Key Protection Services

Encryption can be done in different locations ranging from data in flight or transit over networks (local and remote), as well as data at rest or while stored. Strength of encryption is determined by different hash and cipher codes algorithms including SHA among others ranging from simple to more complex. The encryption can be done by networks, servers, storage systems, hypervisors, operating systems, databases, email, word and many other tools at granularity from device, file systems, folder, file, database, table, object or blob.

Virtual machine and their virtual disks ( VHDX and VMDK) can be encrypted, as well as migration or movements such as vMotions among other activities. Here are some VMware vSphere encryption topics, along with deep dive previews from VMworld 2016 among other resources here, VMware hardening guides here (NSX, vSphere), and a VMware security white paper (PDF) here.

Other security-related items shown in Figure 10.11 include Lightweight Direct Access Protocol (LDAP), Remote Authentication Dial-In User Service (RADIUS), and Kerberos network authentication. Also shown are VPN along with Secure Socket Layer (SSL) network security, along with security and authentication keys, credentials for SSH remote access including SSO. The cloud shown in figure 10.11 could be your own private using AzureStack, VMware (on-site, or public cloud such as IBM or AWS), OpenStack among others, or a public cloud such as AWS, Azure or Google (among others).

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

There are many different aspects, as well as layers of security from logical to physical pertaining to data centers, applications and associated data Infrastructure resources, both on-premises and cloud. Security for legacy and software defined environments needs to be integrated as part of various technology domain focus areas, as well as across them including data protection. The above is a small sampling of security related topics with more covered in various chapters of SDDI Essentials as well as in my other books, webinars, presentations and content.

From a data protection focus, security needs to be addressed from a physical who has access to primary and protection copies, what is being protected against and where, as well as who can access logically protection copes, as well as the configuration, settings, certificates involved in data protection. In other words, how are you protecting your data protection environment, configuration and deployment. Data protection copies need to be encrypted to meet regulations, compliance and other requirements to guard against loss or theft, accidental or intentional. Likewise access control needs to be managed including granting of roles, security, authentication, monitoring of access, along with revocation.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 7 Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Data Protection Diaries Tools Technologies Toolbox Buzzword Bingo Trends

Fundamental Tools, Technologies, Toolbox, Buzzword Bingo Trends

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is Part 7 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 6 Data Protection Security Logical Physical Software Defined, and click here to view the next post Part 8 Walking The Data Protection Talk What I Do.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection related tools, technologies, trends as companion to other posts in this series, as well as across various chapters from the SDDI book.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends

There are many data Infrastructure related topics, technologies, tools, trends, techniques and tips that pertain to data protection, many of which have been covered in this series of posts already, as well as in the SDDI Essentials book, and elsewhere. The following are some additional related data Infrastructure data protection topics, tools, technologies.

Buzzword Bingo is a popular industry activity involving terms, trends, tools and more, read more here, here, and here. The basic idea of buzzword bingo is when somebody starts mentioning lots of buzzwords, buzz terms, buzz trends at some point just say bingo. Sometimes you will get somebody who asks what that means, while others will know, perhaps get the point to move on to what’s relevant vs. talking the talk or showing how current they are on industry activity, trends and terms.

Just as everything is not the same across different environments, there are various size and focus from hyper-scale clouds and managed service providers (MSP) server (and storage along with applications focus), smaller and regional cloud, hosting and MSPs, as well as large enterprise, small medium enterprise (SME), small medium business (SMB), remote office branch office (ROBO), small office home office (SOHO), prosumer, consumer and client or edge. Sometimes you will hear server vs. edge or client focus, thus context is important.

Data protection just like data infrastructures span servers, storage, I/O networks, hardware, software, clouds, containers, virtual, hypervisors and related topics. Otoh, some might view data protection as unique to a particular technology focus area or domain. For example, I once had backup vendor tell me that backups and data protection was not a storage topic, can you guess which vendor did not get recommend for data protection of data stored on storage?

Data gets protected to different target media, mediums or services including HDDs, SSD, tape, cloud, bulk and object storage among others in various format from native to encapsulated in save sets, zips, tar ball among others.

Bulk storage can be on-site, on-premises low-cost tape, disk (file, block or object) as well as off-site including cloud services such as AWS S3 (buckets and objects), Microsoft Azure (containers and blobs), Google among others using various Access ( Protocols, Personalities, Front-end, Back-end) technologies. Which type of data protection storage medium, location or service is best depends on what you are trying to do, along with other requirements.

SDDC SDDI data center data protection toolbox
Data Protection Toolbox

SDDC SDDI Object Storage Architecture
Figure 3.18 Generic Object (and Blob) architecture with Buckets (and Containers)

Object Storage

Before discussing Object Storage lets take a step back and look at some context that can clarify some confusion around the term object. The word object has many different meanings and context, both inside of the IT world as well as outside. Context matters with the term object such as a verb being a thing that can be seen or touched as well as a person or thing of action or feeling directed towards.

Besides a person, place or physical thing, an object can be a software defined data structure that describes something. For example, a database record describing somebody’s contact or banking information, or a file descriptor with name, index ID, date and time stamps, permissions and access control lists along with other attributes or metadata. Another example is an object or blob stored in a cloud or object storage system repository, as well as an item in a hypervisor, operating system, container image or other application.

Besides being a verb, object can also be a noun such as disapproval or disagreement with something or someone. From an IT context perspective, object can also refer to a programming method (e.g. object oriented programming [oop], or Java [among other environments] objects and class’s) and systems development in addition to describing entities with data structures.

In other words, a data structure describes an object that can be a simple variable, constant, complex descriptor of something being processed by a program, as well as a function or unit of work. There are also objects unique or with context to specific environments besides Java or databases, operating systems, hypervisors, file systems, cloud and other things.

SDDC SDDI Object Storage Example
Figure 3.19 AWS S3 Object storage example, objects left and descriptive names on right

The role of object storage (view more at www.objectstoragecenter.com) is to provide low-cost, scalable capacity, durable availability of data including data protection copies on-premises or off-site. Note that not all object storage solutions or services are the same, some are immutable with write once read many (WORM) like attributes, while others non-immutable meaning that they can be not only appended to, also updated to page or block level granularity.

Also keep in mind that some solutions and services refer to items being stored as objects while others as blobs, and the name space those are part of as a bucket or container. Note that context is important not to confuse an object container with a docker, kubernetes or micro services container.

Many applications and storage systems as well as appliances support as back-end targets cloud access using AWS S3 API (of AWS S3 service or other solutions), as well as OpenStack Switch API among others. There are also many open source and third-party tools for working with cloud storage including objects and blobs. Learn more about object storage, cloud storage at www.objectstoragecenter.com as well as in chapters 3, 4, 13 and 14 in SDDI Essentials book.

S3 Simple Storage Service

Simple Storage Service ( S3) is the Amazon Web Service (AWS) cloud object storage service that can be used for bulk and other storage needs. The S3 service can be accessed from within AWS as well as externally via different tools. AWS S3 supports large number of buckets and objects across different regions and availability zones. Objects can be stored in a hierarchical directory structure format for compatibility with existing file systems or as a simple flat name space.

Context is important with data protection and S3 which can mean the access API, or AWS service. Likewise context is important in that some solutions, software and services support S3 API access as part of their front-end (e.g. how servers or clients access their service), as well as a back-end target (what they can store data on).

Additional AWS S3 (service) and related resources include:

Data Infrastructure Environments and Applications

Data Infrastructure environments that need to be protected include legacy, software defined (SDDC, SDDI, SDS), cloud, virtual and container based, as well as clustered, scale-out, converged Infrastructure (CI), hyper-converged Infrastructure (HCI) among others. In addition to data protection related topics already converged in the posts in this series (as well as those to follow), a related topic is Data Footprint Reduction ( DFR). DFR comprises several different technologies and techniques including archiving, compression, compaction, deduplication (dedupe), single instance storage, normalization, factoring, zip, tiering and thin provisioning among many others.

Data Footprint Reduction (DFR) Including Dedupe

There is a long-term relationship with data protection and DFR in that to reduce the impact of storing more data, traditional techniques such as compression and compaction have been used, along with archive and more recently dedupe among others. In the Software Defined Data Infrastructure Essentials book there is an entire chapter on DFR ( chapter 11), as well as related topics in chapters 8 and 13 among others. For those interested in DFR and related topics, there is additional material in my books Cloud and Virtual Data Storage Networking (CRC Press), along with in The Green and Virtual Data Center (CRC Press), as well as various posts on StorageIOblog.com and storageio.com. Figure 11.4 is from Software Defined Data Infrastructure Essentials showing big picture of various places where DFR can be implemented along with different technologies, tools and techniques.

SDDC, SDI, SDDI DFR Dedupe
Figure 11.4 Various points of interest where DFR techniques and technology can be applied

Just as everything is not the same, there are different DFR techniques along with implementations to address various application workload and data performance, availability, capacity, economics (PACE) needs. Where is the best location for DFR that depends on your objectives as well as what your particular technology can support. However in general, I recommend putting DFR as close to where the data is created and stored as possible to maximize its effectiveness which can be on the host server. That however also means leveraging DFR techniques downstream where data gets sent to be stored or protected. In other words, a hybrid DFR approach as a companion to data protection should use various techniques, technologies in different locations. Granted, your preferred vendor might only work in a given location or functionality so you can pretty much guess what the recommendations will be ;) .

Tips, Recommendations and Considerations

Additional learning experiences along with common questions (and answers), appendices, as well as tips can be found here.

General action items, tips, considerations and recommendations include:

  • Everything is not the same; different applications with SLO, PACE, FTT, FTM needs
  • Understand the 4 3 2 1 data protection rule and how to implement it.
  • Balance rebuild performance impact and time vs. storage space overhead savings.
  • Use different approaches for various applications and environments.
  • What is best for somebody else may not be best for you and your applications.
  • You cant go forward in the future after a disaster if you cant go back
  • Data protection is a shared responsibility between vendors, service providers and yourself
  • There are various aspects to data protection and data Infrastructure management

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

There are many different buzzword, buzz terms, buzz trends pertaining to data infrastructure and data protection. These technologies span legacy and emerging, software-defined, cloud, virtual, container, hardware and software. Key point is what technology is best fit for your needs and applications, as well as how to use the tools in different ways (e.g. skill craft techniques and tradecraft). Keep context in mind when looking at and discussing different technologies such as objects among others.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 8 Walking The Data Protection Talk.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Data Protection Diaries Fundamentals Walking The Data Protection Talk

Data Protection Diaries Walking The Data Protection Talk

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is Part 8 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends, and click here to view the next post who’s Doing What ( Toolbox Technology Tools).

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around what I (and Server StorageIO) does for Data Protection besides just talking the talk and is a work in progress that is being updated over time with additional insights.

Walking The Data Protection Talk What I Do

A couple of years back I did the first post as part of the Data Protection Diaries series ( view here), that included the following image showing some data protection needs and requirements, as well as what being done, along with areas for improvement. Part of what I and Server StorageIO does involves consulting (strategy, design, assessment), advising and other influencers activities (e.g. blog, write articles, create reports, webinars, seminars, videos, podcasts) pertaining to data Infrastructure topics as well as data protection.

What this means is knowing about the trends, tools, technologies, what’s old and new, who’s doing what, what should be in the data protection toolbox, as well as how to use those for different scenarios. Its one thing to talk the talk, however I also prefer to walk the talk including eating my own dog food applying various techniques, approaches, tools and technologies discussed.

The following are from a previous Data Protection Diaries post where I discuss my data protection needs (and wants) some of which have evolved since then. Note the image on the left is my Livescribe Echo digital pen and paper tablet. On the right is an example of the digital image created and imported into my computer from the Livescribe. In other words, Im able to protect my hand written notes, diagrams and figures.

Data Protection Diaries Data Protection Diaries Walking The Talk
Via my Livescribe Echo digital pen ( get your Livescribe here at Amazon.com)

My Environment and data protection is always evolving, some based on changing projects, others that are more stable. Likewise the applications along with data are varied after all, everything is not the same. My data protection includes snapshots, replication, mirror, sync, versions, backup, archive, RAID, erasure code among others technologies, tools, and techniques.

Applications range from desktop, office, email, documents, spreadsheets, presentations, video, audio and related items in support of day-to-day activities. Then there are items part of various projects that range from physical to virtual, cloud and container leveraging various tools. This means having protection copies (sync, backup, snapshots, consistency points) of virtual machines, physical machine instances, applications and databases such as SQL Server among many others. Other application workloads include web, word press blog and email among others.

The Server StorageIO environment consists of a mix of legacy on-premises technologies from servers, storage, hardware, software, networks, tools as well as software defined virtual (e.g. VMware, Hyper-V, Docker among others), as well as cloud. The StorageIO data Infrastructure environment consists of dedicated private server (DPS) that I have had for several years now that supports this blog as well as other sites and activity. I also have a passive standby site used for testing of the WordPress based blog on an AWS Lightsail server. I use tools such as Updraft Plus Premium to routinely create a complete data protection view (database, plugins, templates, settings, configuration, core) of my WordPress site (runs on DPS) that is stored in various locations, including at AWS.

Data Protection Diaries Walking The Talk
Some of my past data protection requirements (they have evolved)

Currently the Lightsail Virtual Private Server (VPS) is in passive mode, however plans are to enable it as a warm or active standby fail over site for some of the DPS functions. One of the tools I have for monitoring and insight besides those in WordPress and the DPS are AWS Route 53 alerts that I have set up to monitor endpoints. AWS Route 53 is a handy resource for monitoring your endpoints such as a website, blog among other things and have it notify you, or take action including facilitating DNS fail over if needed. For now, Im simply using Route 53 besides as a secondary DNS as a notification tool.

Speaking of AWS, I have compute instances in Elastic Cloud Compute (EC2) along with associated Elastic Block Storage (EBS) volumes as well as their snapshots. I also have AWS S3 buckets in different regions that are on various tiers from standard to infrequent access (IA), as well as some data on Glacier. Data from my DPS at Bluehost gets protected to a AWS S3 bucket that I can access from AWS EC2, as well as via other locations including Microsoft Azure as needed.

Some on-premises data also gets protected to AWS S3 (as well as to elsewhere) using various tools, for different granularity, frequency, access and retention. After all, everything is not the same, why treat it the same. Some of the data protected to AWS S3 buckets is in native format (e.g. they appear as objects to S3 or object enabled applications), as well as file to file based applications with appropriate tools.

Other data that is also protected to AWS S3 from different data protection or backup tools are stored in vendor neutral or vendor specific save set, zip, tar ball or other formats. In other words, I need the tool or compatible tool that knows the format of the saved data to retrieve individual data files, items or objects. Note that this is similar to storing data on tape, HDDs, SSD or other media in native format vs. in some type of encapsulate save set or other format.

In addition to protecting data to AWS, I also have data at Microsoft Azure among other locations. Other locations include non-cloud based off-site where encrypted removable media is periodically taken to a safe secure place as a master, gold in case of major emergency, ransomeware copy.

Why not just rely on cloud copies?

Simple, I can pull individual files or relatively small amounts of data back from the cloud sometimes faster (or easier) than from on-site copies, let alone my off-site, off-line, air gap copies. On the other hand, if I need to restore large amounts of data, without a fast network, it can be quicker to get the air gap off-line, off-site copy, do the large restore, then apply incremental or changed data via cloud. In other a hybrid approach.

Now a common question I get is why not just do one or the other and save some money. Good point, I would save some money, however by doing the above among other things, they are part of being able to test, try new and different things, gain insight, experience not to mention walk the talk vs. simply talking the talk.

Of course Im always looking for ways to streamline to make my data protection more efficient, as well as effective (along with remove complexity and costs).

  • Everything is not the same, so why treat it all the same with common SLO, RTO, RPO and retention?
  • Likewise why treat and store all data the same way, on the same tiers of technology
  • Gain insight and awareness into environment, applications, workloads, PACE needs
  • Applications, data, systems or devices are protected with different granularity and frequency
  • Apply applicable technology and tools to the task at hand
  • Any data I have in cloud has a copy elsewhere, likewise, any data on-premises has a copy in the cloud or elsewhere
  • I implement the 4 3 2 1 rule by having multiple copies, versions, data in different locations, on and off-line including cloud
  • From a security standpoint, many different things are implemented on a logical as well as physical basis including encryption
  • Ability to restore data as well as applications or image instances locally as well as into cloud environments
  • Leverage different insight and awareness, reporting, analytics and monitoring tools
  • Mix of local storage configured with different RAID and other protection
  • Test, find, fix, remediate improve the environment including leveraging lessons learned

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

Everything is not the same, thats why in my environment I use different technologies, tools and techniques to protect my data. This also means having different RTO, RPO across various applications, data and systems as well as devices. Data that is more important has more copies, versions in different locations as well as occurring more frequently as part of 4 3 2 1 data protection. Other data that does not change as frequently, or time sensitive have alternate RTO and RPO along with corresponding frequency of protection.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series Part 9 who’s Doing What (Toolbox Technology Tools).

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Data Protection Diaries Fundamentals Who Is Doing What Toolbox Technology Tools

Data Protection Toolbox Whos Doing What Technology Tools

Updated 1/17/2018

Data protection toolbox is a companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is Part 9 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 8 Walking The Data Protection Talk, and click here to view the next post Part 10 Data Protection Resources Where to Learn More.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection who’s Doing What ( Toolbox Technology Tools).

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

who’s Doing What (Toolbox Technology Tools)

SDDC SDDI data center data protection toolbox
Data Protection Toolbox

Note that this post is evolving with additional tools, technologies, techniques, hardware, software, services being added over time along with applicable industry links.

The following are a sampling of some hardware, software, solution and component vendors along with service providers involved with data protection from RAID, Erasure Codes (EC) to snapshots, backup, BC, BR, DR, archive, security, cloud, bulk object storage, HDDs, SSD, tape among others including buzzword (and buzz term trends) bingo. Acronis, Actifio, Arcserve, ATTO, AWS, Backblaze, Barracuda, Broadcom, Caringo, Chelsio (offload), Code42/Crashplan, Cray, Ceph, Cisco, Cloudian, Cohesity, Compuverde, Commvault, Datadog, Datrium, Datos IO, DDN, Dell EMC, Druva, E8, Elastifile, Exagrid, Excelero, Fujifilm, Fujutsu, Google, HPE, Huawei, Hedvig, IBM, Intel, Iomega, Iron Mountain, IBM, Jungledisk, Kinetic key value drives (Seagate), Lenovo, LTO organization, Mangstor, Maxta, Mellanox (offload), Micron, Microsoft (Azure, Windows, Storage Spaces), Microsemi, Nakivo, NetApp, NooBaa, Nexsan, Nutanix, OpenIO, OpenStack (Swift), Oracle, Panasas, Panzura, Promise, Pure, Quantum, Quest, Qumulo, Retrospect, Riverbed, Rozo, Rubrik, Samsung, Scale, Scality, Seagate (DotHill), Sony, Solarwinds, Spectralogic, Starwind, Storpool, Strongbox, Sureline, Swiftstack, Synology, Toshiba, Tintri, Turbonomics, Unitrends, Unix and Linux platforms, Vantara, Veeam, VMware, Western Digital (Amplidata, Tegile and others), WekaIO, X-IO, Zadara and Zmanda among many others.

Note if you dont see yours, or your favorite, preferred or clients listed above or in the data Infrastructure industry related links send us a note for consideration to be included in future updates, or having a link, or sponsor spot pointing to your site added. Feel free to add a non sales marketing pitch to courteous comments to the comment section below.

View additional IT, data center and data Infrastructure along with data protection related vendors, services, tools, technologies links here.

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

Part of modernizing data protection for various data center and data infrastructure environments is to know the tools, technologies and trends that are part of your data protection toolbox. The other part of modernizing data is protection is knowing the techniques of how to use different tools, technologies to meet various application workload performance, availability, capacity economic (PACE) needs.

Also keep in mind that information services requires applications (e.g. programs) and that programs are a combination of algorithms (code, rules, policies) and data structures (e.g. data and how it is organized including unstructured). What this means is that data protection needs to address not only data, also the applications, configuration settings, metadata as well as protecting the protection tools and its data.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 10 Data Protection Fundamental Resources Where to Learn More.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Data Protection Diaries Fundamental Resources Where to Learn More

Data Protection Diaries Fundamental Resources Where to Learn More

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is the last in a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Click here to view the previous post Part 9 – who’s Doing What ( Toolbox Technology Tools).

Software Defined Data Infrastructure Essentials Book SDDC

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection Resources Where to Learn More.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

Software Defined Data Infrastructure Essentials Table of Contents (TOC)

Here is a link (PDF) to the table of contents (TOC) for Software Defined Data Infrastructure Essentials.

The following is a Software Defined Data Infrastructure Essentials book TOC summary:

Chapter 1: Server Storage I/O and Data Infrastructure Fundamentals
Chapter 2: Application and IT Environments
Chapter 3: Bits, Bytes, Blobs, and Software-Defined Building Blocks
Chapter 4: Servers: Physical, Virtual, Cloud, and Containers
Chapter 5: Server I/O and Networking
Chapter 6: Servers and Storage-Defined Networking
Chapter 7: Storage Mediums and Component Devices
Chapter 8: Data Infrastructure Services: Access and Performance
Chapter 9: Data Infrastructure Services: Availability, RAS, and RAID
Chapter 10: Data Infrastructure Services: Availability, Recovery-Point Objective, and Security
Chapter 11: Data Infrastructure Services: Capacity and Data Reduction
Chapter 12: Storage Systems and Solutions (Products and Cloud)
Chapter 13: Data Infrastructure and Software-Defined Management
Chapter 14: Data Infrastructure Deployment Considerations
Chapter 15: Software-Defined Data Infrastructure Futures, Wrap-up, and Summary
Appendix A: Learning Experiences
Appendix B: Additional Learning, Tools, and tradecraft Tricks
Appendix C: Frequently Asked Questions
Appendix D: Book Shelf and Recommended Reading
Appendix E: Tools and Technologies Used in Support of This Book
Appendix F: How to Use This Book for Various Audiences
Appendix G: Companion Website and Where to Learn More
Glossary
Index

Click here to view (PDF) table of contents (TOC).

Data Protection Resources Where To Learn More

Learn more about Data Infrastructure and Data Protection related technology, trends, tools, techniques, tradecraft and tips with the following links.

The following are the various posts that are part of this data protection series:

  • Part 1Data Infrastructure Data Protection Fundamentals
  • Part 2 – Reliability, Availability, Serviceability ( RAS) Data Protection Fundamentals
  • Part 3 – Data Protection Access Availability RAID Erasure Codes ( EC) including LRC
  • Part 4 – Data Protection Recovery Points (Archive, Backup, Snapshots, Versions)
  • Part 5 – Point In Time Data Protection Granularity Points of Interest
  • Part 6 – Data Protection Security Logical Physical Software Defined
  • Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends
  • Part 8 – Data Protection Diaries Walking Data Protection Talk
  • Part 9 – who’s Doing What ( Toolbox Technology Tools)
  • Part 10Data Protection Resources Where to Learn More

  • The following are various data protection blog posts:

  • Welcome to the Data Protection Diaries
  • Until the focus expands to data protection, backup is staying alive!
  • The blame game, Does cloud storage result in data loss?
  • Loss of data access vs. data loss
  • Revisiting RAID storage remains relevant and resources
  • Only you can prevent cloud (or other) data loss
  • Data protection is a shared responsibility
  • Time for CDP (Commonsense Data Protection)?
  • Data Infrastructure Server Storage I/O Tradecraft Trends (skills, experiences, knowledge)
  • My copies were corrupted: The [4] 3-2-1 rule and more about 4 3 2 1 as well as 3 2 1 here and here
  • The following are various data protection tips and articles:

  • Via Infostor Cloud Storage Concerns, Considerations and Trends
  • Via Network World What’s a data infrastructure?
  • Via Infostor Data Protection Gaps, Some Good, Some Not So Good
  • Via Infostor Object Storage is in your future
  • Via Iron Mountain Preventing Unexpected Disasters
  • Via InfoStor – The Many Variations of RAID Storage
  • Via InfoStor – RAID Remains Relevant, Really!
  • Via WservNews Cloud Storage Considerations (Microsoft Azure)
  • Via ComputerWeekly Time to restore from backup: Do you know where your data is?
  • Via Network World Ensure your data infrastructure remains available and resilient
  • The following are various data protection related webinars and events:

  • BrightTalk Webinar Data Protection Modernization – Protect, Preserve and Serve you Information
  • BrightTalk Webinar BCDR and Cloud Backup Protect Preserve and Secure Your Data Infrastructure
  • TechAdvisor Webinar (Free with registration) All You Need To Know about ROBO data protection
  • TechAdvisor Webinar (Free with registration) Tips for Moving from Backup to Full Disaster Recovery
  • The following are various data protection tools, technologies, services, vendor and industry resource links:

  • Various Data Infrastructure related news commentary, events, tips and articles
  • Data Center and Data Infrastructure industry links (vendors, services, tools, technologies, hardware, software)
  • Data Infrastructure server storage I/O network Recommended Reading List Book Shelf
  • Software Defined Data Infrastructure Essentials (CRC 2017) Book
  • Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    What This All Means

    Everything is not the same across environments, data centers, data infrastructures including SDDC, SDX and SDDI as well as applications along with their data.

    Likewise everything is and does not have to be the same when it comes to Data Protection.

    Since everything is not the same, various data protection approaches are needed to address various application performance, availability, capacity economic (PACE) needs, as well as SLO and SLAs.

    Data protection encompasses many different hardware, software, services including cloud technologies, tools, techniques, best practices, policies and tradecraft experience skills (e.g. knowing what to use when, where, why and how).

    Software Defined Data Infrastructure Essentials Book SDDC

    Context is important as different terms have various meanings depending on what they are being discussed with. Likewise different technologies and topics such as object, blob, backup, replication, RAID, erasure code (EC), mirroring, gaps (good, bad, ugly), snapshot, checkpoint, availability, durability among others have various meanings depending on context, as well as implementation approach.

    In most cases there is no bad technology or tool, granted there are some poor or bad (even ugly) implementations, as well as deployment or configuration decisions. What this means is the best technology or approach for your needs may be different from somebody else’s and vice versa.

    Some other points include there is no such thing as an information recession with more data generated every day, granted, how that data is transformed or stored can be in a smaller footprint. Likewise there is an increase in the size of data including unstructured big data, as well as the volume (how much data), as well as velocity (speed at which it is created, moved, processed, stored). This also means there is an increased dependency on data being available, accessible and intact with consistency. Thus the fundamental role of data Infrastructures (e.g. what’s inside the data center or cloud) is to combine resources, technologies, tools, techniques, best practices, policies, people skill set, experiences (e.g. tradecraft) to protect, preserve, secure and serve information (applications and data).

    modernizing data protection including backup, availability and related topics means more than swapping out one hardware, software, service or cloud for whatever is new, and then using it in old ways.

    What this means is to start using new (and old) things in new ways, for example move beyond using SSD or HDDs like tape as targets for backup or other data protection approaches. Instead use SSD, HDDs or cloud as a tier, yet also to enable faster protection and recovery by stepping back and rethinking what to protect, when, where, why, how and apply applicable techniques, tools and technologies. Find a balance between knowing all about the tools and trends while not understanding how to use those toolbox items, as well as knowing all about the techniques of how to use the tools, yet not knowing what the tools are.

    Want to learn more, have questions about specific tools, technologies, trends, vendors, products, services or techniques discussed in this series, send a note (info at storageio dot com) or via our contact page. We can set up a time to discuss your questions or needs pertaining to Data Protection as well as data infrastructures related topics from legacy to software defined virtual, cloud, container among others. For example consulting, advisory services, architecture strategy design, technology selection and acquisition coaching, education knowledge transfer sessions, seminars, webinars, special projects, test drive lab reviews or audits, content generation, videos, podcasts, custom content, chapter excerpts, demand generation among many other things.

    Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Data Infrastructure server storage I/O network Recommended Reading #blogtober

    server storage I/O data infrastructure trends recommended reading list

    Updated 7/30/2018

    The following is an evolving recommended reading list of data infrastructure topics including, server, storage I/O, networking, cloud, virtual, container, data protection and related topics that includes books, blogs, podcast’s, events and industry links among other resources.

    Various Data Infrastructure including hardware, software, services related links:

    Links A-E
    Links F-J
    Links K-O
    Links P-T
    Links U-Z
    Other Links

    In addition to my own books including Software Defined Data Infrastructure Essentials (CRC Press 2017), the following are Server StorageIO recommended reading list items . The recommended reading list includes various IT, Data Infrastructure and related topics.

    Intel Recommended Reading List (IRRL) for developers is a good resource to check out.

    Duncan Epping (@DuncanYB), Frank Denneman (@FrankDenneman) and Neils Hagoort (@NHagoort) have released their VMware vSphere 6.7 Clustering Deep Dive book available at venues including Amazon.com. This is the latest in a series of Cluster and deep dive books from Frank and Duncan which if you are involved with VMware, SDDC and related software defined data infrastructures these should be on your bookshelf.

    Check out the Blogtober list of check out some of the blogs and posts occurring during October 2017 here.

    Preston De Guise aka @backupbear is Author of several books has an interesting new site Foolsrushin.info that looks at topics including Ethics in IT among others. Check out his new book Data Protection: Ensuring Data Availability (CRC Press 2017) and available via Amazon.com here.

    Brendan Gregg has a great site for Linux performance related topics here.

    Greg Knieriemen has a must read weekly blog, post, column collection of whats going on in and around the IT and data infrastructure related industries, Check it out here.

    Interested in file systems, CIFS, SMB, SAMBA and related topics then check out Chris Hertels book on implementing CIFS here at Amazon.com

    For those involved with VMware, check out Frank Denneman VMware vSphere 6.5 host resource guide-book here at Amazon.com.

    Docker: Up & Running: Shipping Reliable Containers in Production by Karl Matthias & Sean P. Kane via Amazon.com here.

    Essential Virtual SAN (VSAN): Administrator’s Guide to VMware Virtual SAN,2nd ed. by Cormac Hogan & Duncan Epping via Amazon.com here.

    Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale by Tom White via Amazon.com here.

    Systems Performance: Enterprise and the Cloud by Brendan Gregg Via Amazon.com here.

    Implementing Cloud Storage with OpenStack Swift by Amar Kapadia, Sreedhar Varma, & Kris Rajana Via Amazon.com here.

    The Human Face of Big Data by Rick Smolan & Jennifer Erwitt Via Amazon.com here.

    VMware vSphere 5.1 Clustering Deepdive (Vol. 1) by Duncan Epping & Frank Denneman Via Amazon.com here. Note: This is an older title, but there are still good fundamentals in it.

    Linux Administration: A Beginners Guide by Wale Soyinka Via Amazon.com here.

    TCP/IP Network Administration by Craig Hunt Via Amazon.com here.

    Cisco IOS Cookbook: Field tested solutions to Cisco Router Problems by Kevin Dooley and Ian Brown Via Amazon.com here.

    I often mention in presentations a must have for anybody involved with software defined anything, or programming for that matter which is the Niklaus Wirth classic Algorithms + Data Structures = Programs that you can get on Amazon.com here.

    Seven Databases in Seven Weeks including NoSQL

    Another great book to have is Seven Databases in Seven Weeks (here is a book review) which not only provides an overview of popular NoSQL databases such as Cassandra, Mongo, HBASE among others, lots of good examples and hands on guides. Get your copy here at Amazon.com.

    Additional Data Infrastructure and related topic sites

    In addition to those mentioned above, other sites, venues and data infrastructure related resources include:

    aiim.com – Archiving and records management trade group

    apache.org – Various open-source software

    blog.scottlowe.org – Scott Lowe VMware Networking and topics

    blogs.msdn.microsoft.com/virtual_pc_guy – Ben Armstrong Hyper-V blog

    brendangregg.com – Linux performance-related topics

    cablemap.info – Global network maps

    CMG.org – Computer Measurement Group (CMG)

    communities.vmware.com – VMware technical community and resources

    comptia.org – Various IT, cloud, and data infrastructure certifications

    cormachogan.com – Cormac Hogan VMware and vSAN related topics

    csrc.nist.gov – U.S. government cloud specifications

    dmtf.org – Distributed Management Task Force (DMTF)

    ethernetalliance.org – Ethernet industry trade group

    fibrechannel.org – Fibre Channel trade group

    github.com – Various open-source solutions and projects

    Intel Reading List – recommended reading list for developers

    ieee.org – Institute of Electrical and Electronics Engineers

    ietf.org – Internet Engineering Task Force

    iso.org – International Standards Organizations

    it.toolbox.com – Various IT and data infrastructure topics forums

    labs.vmware.com/flings – VMware Fling additional tools and software

    nist.gov – National Institute of Standards and Technology

    nvmexpress.org – NVM Express (NVMe) industry trade group

    objectstoragecenter.com – Various object and cloud storage items

    opencompute.org – Open Compute Project (OCP) servers and related topics

    opendatacenteralliance.org – Open Data Center Alliance (ODCA)

    openfabrics.org – Open-fabric software industry group

    opennetworking.org – Open Networking Foundation (ONF)

    openstack.org – OpenStack resources

    pcisig.com – Peripheral Component Interconnect (PCI) trade group

    reddit.com – Various IT, cloud, and data infrastructure topics

    scsita.org – SCSI trade association (SAS and others)

    SNIA.org – Storage Network Industry Association (SNIA)

    Speakingintech.com – Popular industry and data infrastructure podcast

    Storage Bibliography – Collection of Dr. J. Metz storage related content

    technet.microsoft.com – Microsoft TechNet data infrastructure–related topics

    thenvmeplace.com – various NVMe and related tools, topics and links

    thevpad.com – Collection of various virtualization and related sites

    thessdplace.com – various NVM, SSD, flash, 3D XPoint related topics, tools, links

    tpc.org – Transaction Performance Council benchmark site

    vmug.org – VMware User Groups (VMUG)

    wahlnetwork.com – Chris Whal Networking and related topics

    yellow-bricks.com – Duncan Epping VMware and related topics

    Additional Data Infrastructure Venues

    Additional useful data infrastructure related information can be found at BizTechMagazine, BrightTalk, ChannelProNetwork, ChannelproSMB, ComputerWeekly, Computerworld, CRN, CruxialCIO, Data Center Journal (DCJ), Datacenterknowledge, and DZone. Other good sourses include Edtechmagazine, Enterprise Storage Forum, EnterpriseTech, Eweek.com, FedTech, Google+, HPCwire, InfoStor, ITKE, LinkedIn, NAB, Network Computing, Networkworld, and nextplatform. Also check out Reddit, Redmond Magazine and Webinars, Spiceworks Forums, StateTech, techcrunch.com, TechPageOne, TechTarget Venues (various Search sites, e.g., SearchStorage, SearchSSD, SearchAWS, and others), theregister.co.uk, TheVarGuy, Tom’s Hardware, and zdnet.com, among many others.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    The above is an evolving collection of recommended reading including what I have on my physical and virtual bookshelves, as well as list of web sites, blogs and podcasts worth listening, reading or watching. Watch for more items to be added to the book shelf soon, and if you have a suggested recommendation, add it to the comments below.

    By the way, if you have not heard, its #Blogtober, check out some of the other blogs and posts occurring during October here as part of your recommended reading list.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    PCIe Fundamentals Server Storage I/O Network Essentials

    Updated 8/31/19

    PCIe Fundamentals Server Storage I/O Network Essentials

    PCIe fundamentals data infrastructure trends

    This piece looks at PCIe Fundamentals topics for server, storage, I/O network data infrastructure environments. Peripheral Computer Interconnect (PCI) Express aka PCIe is a Server, Storage, I/O networking fundamentals component. This post is an excerpt from chapter 4 (Chapter 4: Servers: Physical, Virtual, Cloud, and Containers) of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017) Available via Amazon.com and other global venues. In this post, we look various PCIe fundamentals to learn and expand or refresh your server, storage, and I/O and networking tradecraft skills experience.

    PCIe fundamentals Server Storage I/O Fundamentals

    PCIe fundamental common server I/O component

    Common to all servers is some form of a main system board, which can range from a few square meters in supercomputers, data center rack, tower, and micro towers converged or standalone, to small Intel NUC (Next Unit of Compute), MSI and Kepler-47 footprint, or Raspberry Pi-type desktop servers and laptops. Likewise, PCIe is commonly found in storage and networking systems, appliances among other devices.

    For example, a blade server will have multiple server blades or modules, each with its motherboard, which shares a common back plane for connectivity. Another variation is a large server such as an IBM “Z” mainframe, Cray, or another supercomputer that consists of many specialized boards that function similar to a smaller-sized motherboard on a larger scale.

    Some motherboards also have mezzanine or daughter boards for attachment of additional I/O networking or specialized devices. The following figure shows a generic example of a two-socket, with eight-memory-channel-type server architecture.

    PCIe fundamentals SDDC, SDI, SDDI Server fundamentals
    Generic computer server hardware architecture. Source: Software Defined Data Infrastructure Essentials (CRC Press 2017)

    The above figure shows several PCIe, USB, SAS, SATA, 10 GbE LAN, and other I/O ports. Different servers will have various combinations of processor, and Dual Inline Memory Module (DIMM) Dynamic RAM (DRAM) sockets along with other features. What will also vary are the type and some I/O and storage expansion ports, power and cooling, along with management tools or included software.

    PCIe, Including Mini-PCIe, NVMe, U.2, M.2, and GPU

    At the heart of many servers I/O and connectivity solutions are the PCIe industry-standard interface (see PCIsig.com). PCIe is used to communicate with CPUs and the outside world of I/O networking devices. The importance of a faster and more efficient PCIe bus is to support more data moving in and out of servers while accessing fast external networks and storage.

    For example, a server with a 40-GbE NIC or adapter would have to have a PCIe port capable of 5 GB per second. If multiple 40-GbE ports are attached to a server, you can see where the need for faster PCIe interfaces come into play.

    As more VM are consolidated onto PM, as applications place more performance demand either regarding bandwidth or activity (IOPS, frames, or packets) per second, more 10-GbE adapters will be needed until the price of 40-GbE (also 25, 50 or 100 Gbe) becomes affordable. It is not if, but rather when you will grow into the performance needs on either a bandwidth/throughput basis or to support more activity and lower latency per interface.

    PCIe is a serial interface specified for how servers communicate between CPUs, memory, and motherboard-mounted as well as AiC devices. This communication includes support attachment of onboard and host bus adapter (HBA) server storage I/O networking devices such as Ethernet, Fibre Channel, InfiniBand, RapidIO, NVMe (cards, drives, and fabrics), SAS, and SATA, among other interfaces.

    In addition to supporting attachment of traditional LAN, SAN, MAN, and WAN devices, PCIe is also used for attaching GPU and video cards to servers. Traditionally, PCIe has been focused on being used inside of a given server chassis. Today, however, PCIe is being deployed on servers spanning nodes in dual, quad, or CiB, CI, and HCI or Software Defined Storage (SDS) deployments. Another variation of PCIe today is that multiple servers in the same rack or proximity can attach to shared devices such as storage via PCIe switches.

    PCIe components (hardware and software) include:

    • Hardware chipsets, cabling, connectors, endpoints, and adapters
    • Root complex and switches, risers, extenders, retimers, and repeaters
    • Software drivers, BIOS, and management tools
    • HBAs, RAID, SSD, drives, GPU, and other AiC devices
    • Mezzanine, mini-PCIe, M.2, NVMe U.2 (8639 drive form factor)

    There are many different implementations of PCIe, corresponding to generations representing speed improvements as well as physical packing options. PCIe can be deployed in various topologies, including a traditional model where an AiC such as GbE or Fibre Channel HBA connects the server to a network or storage device.

    Another variation is for a server to connect to a PCIe switch, or in a shared PCIe configuration between two or more servers. In addition to different generations and topologies, there are also various PCIe form factors and physical connectors (see the following figure), ranging from AiC of various length and height, as well as M.2 small-form-factor devices and U.2 (8639) drive form-factor device for NVMe, among others.

    Note that the presence of M.2 does not guarantee PCIe NVMe, as it also supports SATA.

    Likewise, different NVMe devices run at various PCIe speeds based on the number of lanes. For example, in the following figure, the U.2 (8639) device (looks like a SAS device) shown is a PCIe x4.

    SDDC, SDI, SDDI PCIe NVMe U.2 8639 drive fundamentals
    PCIe devices NVMe U.2, M.2, and NVMe AiC. (Source: StorageIO Labs.)

    PCIe leverages multiple serial unidirectional point-to-point links, known as lanes, compared to traditional PCI, which used a parallel bus design. PCIe interfaces can have one (x1), four (x4), eight (x8), sixteen (x16), or thirty-two (x32) lanes for data movement. Those PCIe lanes can be full-duplex, meaning data is sent and received at the same time, providing improved effective performance.

    PCIe cards are upward-compatible, meaning that an x4 can work in an x8, an x8 in an x16, and so forth. Note, however, that the cards will not perform any faster than their specified speed; an x4 in an x8 slot will only run at x8. PCIe cards can also have single, dual, or multiple external ports and interfaces. Also, note that there are still some motherboards with legacy PCI slots that are not interoperable with PCIe cards and vice versa.

    Note that PCIe cards and slots can be mechanically x1, x4, x8, x16, or x32, yet electrically (or signal) wired to a slower speed, based on the type and capabilities of the processor sockets and corresponding chipsets being used. For example, you can have a PCIe x16 slot (mechanical) that is wired for x8, which means it will only run at x8 speed.

    In addition to the differences between electrical and mechanical slots, also pay attention to what generation the PCIe slots are, such as Gen 2 or Gen 3 or higher. Also, some motherboards or servers will advertise multiple PCIe slots, but those are only active with a second or additional processor socket occupied by a CPU. For example, a PCIe card that has dual x4 external PCIe ports requiring full PCIe bandwidth will need at least PCIe x8 attachment in the server slot. In other words, for full performance, the external ports on a PCIe card or device need to match the external electrical and mechanical card type and vice versa.

    Recall big “B” as in Bytes vs. little “b” as in bits; for example, a PCIe Gen 3 x4 electrical could provide up to 4 GB/s bandwidth (your mileage and performance will vary), which translates to 8 × 4 GB or 32 Gbits/s. In the following table below, there is a mix of Big “B” Bytes per second and small “b” bits per second.

    Each generation of PCIe has improved on the previous one by increasing the effective speed of the links. Some of the speed improvements have come from faster clock rates while implementing lower overhead encoding (e.g., from 8 b/10 b to 128 b/130 b).

    For example, PCIe Gen 3 raw bit or line rate is 8 GT/s or 8 Gbps or about 2 GBps by using a 128 b/130 b encoding scheme that is very efficient compared to PCIe Gen 2 or Gen 1, which used an 8 b/10 b encoding scheme. With 8 b/10 b, there is a 20% overhead vs. a 1.5% overhead with 128 b/130 b (i.e., of 130 bits sent, 128 bits contain data, and 2 bits are for overhead).

    PCIe Gen 1

    PCIe Gen 2

    PCIe Gen 3

    PCIe Gen 4

    PCIe Gen 5

    Raw bit rate

    2.5 GT/s

    5 GT/s

    8 GT/s

    16 GT/s

    32 GT/s

    Encoding

    8 b/10 b

    8 b/10 b

    128 b/130 b

    128 b/130 b

    128 b/130 b

    x1 Lane bandwidth

    2 Gb/s

    4 Gb/s

    8 Gb/s

    16 Gb/s

    32 Gb/s

    x1 Single lane (one-way)

    ~250 MB/s

    ~500 MB/s

    ~1 GB/s

    ~2 GB/s

    ~4GB/s

    x16 Full duplex (both ways)

    ~8 GB/s

    ~16 GB/s

    ~32 GB/s

    ~64 GB/s

    ~128 GB/s

    Above Table: PCIe Generation and Sample Lane Comparison

    Note that PCIe Gen 3 is the currently generally available shipping technology with PCIe Gen 4 appearing in the not so distant future, with PCIe Gen 5 in the wings appearing a few more years down the road.

    By contrast, older generations of Fibre Channel and Ethernet also used 8 b/10 b, having switched over to 64 b/66 b encoding with 10 Gb and higher. PCIe, like other serial interfaces and protocols, can support full-duplex mode, meaning that data can be sent and received concurrently.

    PCIe Bit Rate, Encoding, Giga Transfers, and Bandwidth

    Let’s clarify something about data transfer or movement both internal and external to a server. At the core of a server, there is data movement within the sockets of the processors and its cores, as well as between memory and other devices (internal and external). For example, the QPI bus is used for moving data between some Intel processors whose performance is specified in giga transfers (GT).

    PCIe is used for moving data between processors, memory, and other devices, including internal and external facing devices. Devices include host bus adapters (HBAs), host channel adapters (HCAs), converged network adapters (CNAs), network interface cards (NICs) or RAID cards, and others. PCIe performance is specified in multiple ways, given that it has a server processor focus which involves GT for raw bit rate as well as effective bandwidth per lane.

    Note to keep in perspective PCIe mechanical as well as electrical lanes in that a card or slot may be advertised as say x8 mechanical (e.g., its physical slot form factor) yet only be x4 electrical (how many of those lanes are used or enabled). Also in the case of an adapter that has two or more ports, if the device is advertised as x8 does that mean it is x8 per port or x4 per port with an x8 connection to the PCIe bus.

    Effective bandwidth per lane can be specified as half- or full-duplex (data moving in one or both directions for send and receive). Also, effective bandwidth can be specified as a single lane (x1), four lanes (x4), eight lanes (x8), sixteen lanes (x16), or 32 lanes (x32), as shown in the above table. The difference in speed or bits moved per second between the raw bit or line rate, and the effective bandwidth per lane in a single direction (i.e., half-duplex) is the encoding that is common to all serial data transmissions.

    When data gets transmitted, the serializer/deserializer, or serdes, convert the bytes into a bit stream via encoding. There are different types of encoding, ranging from 8 b/10 b to 64 b/66 b and 128 b//130 b, shown in the following table.

    Single 1542-byte frame

    64 × 1542-byte frames

    Encoding Scheme

    Overhead

    Data Bits

    Encoding Bits

    Bits Transmitted

    Data Bits

    Encoding Bits

    Bits Transferred

    8 b/10 b

    20%

    12,336

    3,084

    15,420

    789,504

    197,376

    986,880

    64 b/66 b

    3%

    12,336

    386

    12,738

    789,504

    24,672

    814,176

    128 b/130 b

    1.5%

    12,336

    194

    12,610

    789,504

    12,336

    801,840

    Above Table: Low-Level Serial Encoding Data Transmit Efficiency

    In these encoding schemes, the smaller number represents the amount of data being sent, and the difference is the overhead. Note that this is different yet related to what occurs at a higher level with the various network protocols such as TCP/IP (IP). With IP, there is a data payload plus addressing and other integrity and management features in a given packet or frame.

    The 8-b/10-b, 64-b/66-b or 128-b/130-b encoding is at the lower physical layer. Thus, a small change there has a big impact and benefit when optimized. Table 4.2 shows comparisons of various encoding schemes using the example of moving a single 1542-byte packet or frame, as well as sending (or receiving) 64 packets or frames that are 1542 bytes in size.

    Why 1542? That is a standard IP packet including data and protocol framing without using jumbo frames (MTU or maximum transmission units).

    What does this have to do with PCIe? GbE, 10-GbE, 40-GbE, and other physical interfaces that are used for moving TCP/IP packets and frames interface with servers via PCIe.

    This encoding is important as part of server storage I/O tradecraft regarding understanding the impact of performance and network or resource usage. It also means understanding why there are fewer bits per second of effective bandwidth (independent of compression or deduplication) vs. line rate in either half- or full-duplex mode.

    Another item to note is that looking at encoding such as the example given in the above table shows how a relatively small change at a large scale can have a big effective impact benefit. If the bits and bytes encoding efficiency and effectiveness scenario in Table 4.2 do not make sense, then try imagining 13 MINI Cooper automobiles each with eight people in it (yes, that would be a tight fit) end to end on the same road.

    Now imagine a large bus that takes up much less length on the road than the 13 MINI Coopers. The bus holds 128 people, who would still be crowded but nowhere near as cramped as eight people in a MINI, plus 24 additional people can be carried on the bus. That is an example of applying basic 8-b/10-b encoding (the MINI) vs. applying 128-b/130-b encoding (the bus) and is also similar to PCIe G3 and G4, which use 128-b/130-b encoding for data movement.

    PCIe Topologies

    The basic PCIe topology configuration has one or more devices attached to the root complex shown in the following figure via an AiC or onboard device connector. Examples of AiC and motherboard-mounted devices that attach to PCIe root include LAN or SAN HBA, networking, RAID, GPU, NVM or SSD, among others. At system start-up, the server initializes the PCIe bus and enumerates the devices found with their addresses.

    PCIe devices attach (shown in the following figure) to a bus that communicates with the root complex that connects with processor CPUs and memory. At the other end of a PCIe device is an end-point target, a PCIe switch that in turn has end-point targets attached. From a software standpoint, hypervisor or operating system device drivers communicate with the PCI devices that in turn send or receive data or perform other functions.

    SDDC, SDI, SDDI PCIe fundamentals
    Basic PCIe root complex with a PCIe switch or expander.

    Note that in addition to PCIe AiC such as HBAs, GPU, and NVM SSD, among others that install into PCIe slots, servers also have converged storage or disk drive enclosures that support a mix of SAS, SATA, and PCIe. These enclosure backplanes have a connector that attaches to a SAS or SATA onboard port, or a RAID card, as well as to a PCIe riser card or motherboard connector. Depending on what type of drive is installed in the connector, either the SAS, SATA, or NVMe (AiC, U.2, and M2) using PCIe communication paths are used.

    In addition to traditional and switched PCIe, using PCIe switches as well as nontransparent bridging (NTB), various other configurations can be deployed. These include server to server for clustering, failover, or device sharing as well as fabrics. Note that this also means that while traditionally found inside a server, PCIe can today use an extender, retimer, and repeaters extended across servers within a rack or cabinet.

    A nontransparent bridge (NTB) is a point-to-point connection between two PCIe-based systems that provide electrical isolation yet functions as a transport bridge between two different address domains. Hosts on either side of the NTB see their respective memory or I/O address space. The NTB presents an endpoint exposed to the local system where writes are mirrored to memory on the remote system to allow the systems to communicate and share devices using associated device drivers. For example, in the following figure, two servers, each with a unique PCIe root complex, address, and memory map, are shown using NTB to any communication between the systems while maintaining data integrity.

    SDDC, SDI, SDDI PCIe two server fundamentals
    PCIe dual server example using NTB along with switches.

    General PCIe considerations (slots and devices) include:

    • Power consumption (and heat dissipation)
    • Physical and software plug-and-play (good interoperability)
    • Drivers (in-the-box, built into the OS, or add-in)
    • BIOS, UEFI, and firmware being current versions
    • Power draw per card or adapters
    • Type of processor, socket, and support chip (if not an onboard processor)
    • Electrical signal (lanes) and mechanical form factor per slot
    • Nontransparent bridge and root port (RP)
    • PCI multi-root (MR), single-root (SR), and hot plug
    • PCIe expansion chassis (internal or external)
    • External PCIe shared storage

    Various operating system and hypervisor commands are available for viewing and managing PCIe devices. For example, on Linux, the “lspci” and “lshw–c pci” commands displays PCIe devices and associated information. On a VMware ESXi host, the “esxcli hardware pci list” command will show various PCIe devices and information, while on Microsoft Windows systems, “device manager” (GUI) or “devcon” (command line) will show similar information.

    Who Are Some PCIe Fundamentals Vendors and Service Providers

    While not an exhaustive list, here is a sampling of some vendors and service providers involved in various ways with PCIe from solutions to components to services to trade groups include Amphenol (connectors and cables), AWS (cloud data infrastructure services), Broadcom (PCIe components), Cisco (servers), DataOn (servers), Dell EMC (servers, storage, software), E8 (storage software), Excelero (storage software), HPE (storage, servers), Huawei (storage, servers), IBM, Intel (storage, servers, adapters), Keysight (test equipment and tools).

    Others include Lenovo (servers), Liqid (composable data infrastructure), Mellanox (server and storage adapters), Micron (storage devices), Microsemi (PCIe components), Microsoft (Cloud and Software including S2D), Molex (connectors, cables), NetApp, NVMexpress.org (NVM Express trade group organizations), Open Compute Project (server, storage, I/O network industry group), Oracle, PCISIG (PCIe industry trade group), Samsung (storage devices), ScaleMP (composable data infrastructure), Seagate (storage devices), SNIA (industry trade group), Supermicro (servers), Tidal (composable data infrastructure), Vantar (formerly known as HDS), VMware (Software including vSAN), and WD among others.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    PCIe fundamentals are resources for building legacy and software-defined data infrastructures (SDDI), software-defined infrastructures (SDI), data centers and other deployments from laptop to large scale, hyper-scale cloud service providers. Learn more about Servers: Physical, Virtual, Cloud, and Containers in chapter 4 of my new book Software Defined Data Infrastructure Essentials (CRC Press 2017) Available via Amazon.com and other global venues. Meanwhile, PCIe fundamentals continues to evolve as a Server, Storage, I/O networking fundamental component.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Amazon Web Service AWS September 2017 Software Defined Data Infrastructure Updates

    Amazon Web Service AWS September 2017 Software Defined Data Infrastructure Updates

    server storage I/O data infrastructure trends

    Amazon Web Service AWS September 2017 Software Defined Data Infrasture Updates

    September was a busy month pertaining to software defined data infrastructure including cloud and related AWS announcements. One of the announcements included VMware partnering to deliver vSphere, vSAN and NSX data infrastructure components for creating software defined data centers (SDDC) also known as multi cloud, and hybrid cloud leveraging AWS elastic bare metal servers (read more here in a companion post). Unlike traditional partner software defined solutions that relied on AWS Elastic Cloud Compute (EC2) instances, VMware is being deployed using private bare metal AWS elastic servers.

    What this means is that VMware vSphere (e.g. ESXi) hypervisor, vCenter, software defined storage (vSAN), storage defined network (NSX) and associated vRealize tools are deployed on AWS data infrastructure that can be used for deploying hybrid software defined data centers (e.g. connecting to your existing VMware environment). Learn more about VMware on AWS here or click on the following image.

    VMware on AWS via Amazon.com

    Additional AWS Updates

    Amazon Web Services (AWS) updates include, coinciding with VMworld, the initial availability of VMware on AWS (using virtual private servers e.g. think along the lines of Lightsail, not EC2 instances) was announced. Amazon Web Services (AWS) continues its expansion into database and table services with Relational Data Services (RDS) including various engines (Amazon Auora,MariaDB, MySQL, Oracle, PostgreSQL,and SQL Server along with Database Migration Service (DMS). Note that these RDS are in addition to what you can install and run your self on Elastic Cloud Compute (EC2) virtual machine instances, Lambda serverless containers, or Lightsail Virtual Private Servers (VPS).

    AWS has published a guide to database testing on Amazon RDS for Oracle plotting latency and IOPs for OLTP workloads here using SLOB. If you are not familiar with SLOB (Silly Little Oracle Benchmark) here is a podcast with its creator Kevin Closson discussing database performance and related topics. Learn more about SLOB and step by step installation for AWS RDS Oracle here, and for those who are concerned or think that you can not run workloads to evaluate Oracle platforms, have a look at this here.

    EC2 enhancements include charging by the second (previous by the hour) for some EC2 instances (see details here including what is or is not currently available) which is a growing trend by private cloud vendors aligning with how serverless containers have been billed. New large memory EC2 instances that for example support up to 3,904GB of DDR4 RAM have been added by AWS. Other EC2 enhancements include updated network performance for some instances, OpenCL development environment to leverage AWS F1 FPGA enabled instances, along with new Elastic GPU enabled instances. Other server and network enhancements include Network Load Balancer for Elastic Load Balancer announced, as well as application load balancer now supports load balancing to IP address as targets for AWS and on premises (e.g. hybrid) resources.

    Other updates and announces include data protection backups to AWS via Commvault and AWS Storage Gateway VTL announced. IBM has announced their Spectrum Scale (e.g. formerly known as SONAS aka GPFS) Scale Out Storage solution for high performance compute (HPC) quick start on AWS. Additional AWS enhancements include new edge location in Boston and a third Seattle site, while Direct Connect sites have been added in Boston and Houston along with Canberra Australia. View more AWS announcements and enhancements here.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    What This All Means

    AWS continues to grow and expand, both in terms of number of services, also the extensiveness of them. Likewise AWS continues to add more regions and data center availability zones, enhanced connectivity, along with earlier mentioned service features. The partnership with VMware should enable enterprise organizations to move towards hybrid cloud data infrastructures, while giving AWS an additional reach into those data centers. Overall a good set of enhancements by AWS who continues to evolve their cloud and software defined data infrastructure portfolio of solution offerings.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    August 2017 Server StorageIO Data Infrastructure Update Newsletter



    Server StorageIO August 2017 Data Infrastructure Update Newsletter

    Volume 17, Issue VII (Pre VMworld 2017)

    Hello and welcome to the August 2017 issue of the Server StorageIO update newsletter.

    Its end of summer season here in north america which means wrapping up holidays, vacations, back to school shopping (and going to school), as well as the start of the fall IT technology conference season. VMworld 2017 USA is this week in Las Vegas and there will be several announcements coming out of that event. Given all of the activity so far this month, I’m going to cover the VMworld and related topics in a special early September issue of this newsletter.

    Speaking of VMworld 2017, if you are going to be there in Las Vegas, stop by the book store located in the community village area on Tuesday at 1PM I will be doing a book signing, meet and greet, stop by and say hello.

    Thanks to all who participated in the recent thevPad top 100 vBloggers event, I am honored to have StorageIOblog listed in the top 100 vBlogs. Also congratulations to new and returning fellow Microsoft MVPs and VMware vExperts. There is a lot going on in the industry, lets get to it in this Server StorageIO Data Infrastructure Update Newsletter.

    In This Issue

    Enjoy this edition of the Server StorageIO update newsletter (pre VMworld edition).

    Cheers GS

    Data Infrastructure and IT Industry Activity Trends

    Acronis announced True Image 2018 for home based data protection (backup), while Crashplan aka code42 announced they were getting out of the consumer, small office home office (SOHO) backup and data protection space to focus on the enterprise.

    Cisco bought software defined storage converged infrastructure software vendor Springpath for about $320M USD. Cisco and Swiftstack (object storage software) also announced interoperability news with the UCS S32600 storage server platform.

    GPU vendor NVIDIA announced Quadro Virtual Data Center workstation technology.

    Meanwhile ioFABRIC announced their new Vicinity 3.0 software defined management solution.

    Microsemi (remember PMC Sierra) announced release of its Flashtec PCIe controllers to help speed adoption deployment of SSDs including NVMe based.

    Microsoft bought Cycle Computing to enhance Azure services, while also making Azure Blob storage tiering available as part of an ongoing public preview. For those not aware, Azure Blob is similar to what other services call objects. Get in on the public preview here. For those who live in a hybrid world where your environment and experience include both Windows and Linux, check out Windows Services for Linux here. With this service which can install onto an Windows 10 system along side Win32 (e.g. it co-exists, its not a virtual machine), you can choose from the Windows Store which Linux distro you want (e.g. Centos, Ubuntu, etc).

    Need to learn, refresh or simply gain a better understanding of Microsoft PowerShell for software defined management of Windows, Azure and other environments? Check out this great post from Microsoft Blogs.

    For those who work in a Windows or Azure environment, here are some useful icons for Powerpoint, Visio, PNG and SVG from Microsoft. With Microsoft Ignite coming up in September, watch for some interesting update enhancements to Windows Server from a server storage I/O perspective.

    NextPlatform.com has an interesting article on Exascale Timeline for Storage and I/O systems worth a read. Panzura global name space and scale out software defined storage management software announced mobile client file sharing. After dropping their own cloud business, Verizon is now a virtual network services partner with Amazon.

    Over at all flash array (AFA) SSD vendor Pure, revenues are growing closer to an annual $1B USD rate despite loss per share, Pure also announced a change in leadership with current CEO Scott Dietzen stepping aside for Charles Giancarlo to take the lead spot.

    VMware has been talking about the continued increase in customer adoption and deployment of VSAN now they are showing they eat their own dog food. Check out this post here from VMware that shows how many and what size VSAN clusters they are using for various internal operations. Also on the VMware storage front, learn more about enhancements for large and small file allocation blocks with vSphere VMFS6.

    With all of the pre and post VMworld related announcements, remember to check out the tools available over at the VMware flings site including vSphere HTML5 Web Client, HCIBench, vRealize Operations Export, VisualEsxtop, ESXi Embedded Host Client, VMware OS Optimization Tool and many others. Watch for VMworld coverage in the September newsletter along with posts at www.storageioblog.com

    Check out other industry news, comments, trends perspectives here.

    Server StorageIO Commentary in the news

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via EnterpriseStorageForum: Comments on Who Will Rule the Storage World?
    Via InfoGoto: Comments on Google Cloud Platform Gaining Data Storage Momentum
    Via InfoGoto: Comments on Singapore High Rise Data Centers
    Via InfoGoto: Comments on New Tape Storage Capacity

    View more Server, Storage and I/O trends and perspectives comments here

    Server StorageIOblog Posts

    Recent and popular Server StorageIOblog posts include:

    In Case You Missed It #ICYMI

    View other recent as well as past StorageIOblog posts here

    Server StorageIO Data Infrastructure Tips and Articles

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via NetworkWorld: Do you have an IT trade craft skills gap?

    View more Server, Storage and I/O trends and perspectives comments here

    Events and Activities

    Recent and upcoming event activities.

    Sep. 21, 2017 – MSP CMG – Minneapolis MN
    Sep. 20, 2017 – Redmond Data Protection and Backup – Webinar
    Sep. 14, 2017 – Fujifilm IT Executive Summit – Seattle WA
    Sep. 12, 2017 – SNIA Software Developers Conference (SDC) – Santa Clara CA
    Sep. 7, 2017 – WiPro – Planning Your Software Defined Journey – New York City
    August 29, 2017 – VMworld – Las Vegas

    See more webinars and activities on the Server StorageIO Events page here.

    Server StorageIO Industry Resources and Links

    Useful links and pages:
    Microsoft TechNet – Various Microsoft related from Azure to Docker to Windows
    storageio.com/links – Various industry links (over 1,000 with more to be added soon)
    objectstoragecenter.com – Cloud and object storage topics, tips and news items
    OpenStack.org – Various OpenStack related items
    storageio.com/protect – Various data protection items and topics
    thenvmeplace.com – Focus on NVMe trends and technologies
    thessdplace.com – NVM and Solid State Disk topics, tips and techniques
    storageio.com/converge – Various CI, HCI and related SDS topics
    storageio.com/performance – Various server, storage and I/O benchmark and tools
    VMware Technical Network – Various VMware related items

    Ok, nuff said, for now.

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Who Will Be At Top Of Storage World Next Decade?

    Who Will Be At Top Of Storage World Next Decade?

    server storage I/O data infrastructure trends

    Data Storage regardless of if hardware, legacy, new, emerging, cloud service or various software defined storage (SDS) approaches are all fundamental resource components of data infrastructures along with compute server, I/O networking as well as management tools, techniques, processes and procedures.

    fundamental Data Infrastructure resource components
    Fundamental Data Infrastructure resources

    Data infrastructures include legacy along with software defined data infrastructures (SDDI), along with software defined data centers (SDDC), cloud and other environments to support expanding workloads more efficiently as well as effectively (e.g. boosting productivity).

    Data Infrastructures and workloads
    Data Infrastructure and other IT Layers (stacks and altitude levels)

    Various data infrastructures resource components spanning server, storage, I/O networks, tools along with hardware, software, services get defined as well as composed into solutions or services which may in turn be further aggregated into more extensive higher altitude offerings (e.g. further up the stack).

    IT and Data Infrastructure Stack Layers
    Various IT and Data Infrastructure Stack Layers (Altitude Levels)

    Focus on Data Storage Present and Future Predictions

    Drew Robb (@Robbdrew) has a good piece over at Enterprise Storage Forum looking at the past, present and future of who will rule the data storage world that includes several perspective predictions comments from myself as well as others. Some of the perspectives and predictions by others are more generic and technology trend and buzzword bingo focus which should not be a surprise. For example including the usual performance, Cloud and Object Storage, DPDK, RDMA/RoCE, Software-Defined, NVM/Flash/SSD, CI/HCI, NVMe among others.

    Here are some excerpts from Drews piece along with my perspective and prediction comments of who may rule the data storage roost in a decade:

    Amazon Web Services (AWS) – AWS includes cloud and object storage in the form of S3. However, there is more to storage than object and S3 with AWS also having Elastic File Services (EFS), Elastic Block Storage (EBS), database, message queue and on-instance storage, among others. for traditional, emerging and storage for the Internet of Things (IoT).

    It is difficult to think of AWS not being a major player in a decade unless they totally screw up their execution in the future. Granted, some of their competitors might be working overtime putting pins and needles into Voodoo Dolls (perhaps bought via Amazon.com) while wishing for the demise of Amazon Web Services, just saying.

    Voodoo Dolls via Amazon.com
    Voodoo Dolls and image via Amazon.com

    Of course, Amazon and AWS could follow the likes of Sears (e.g. some may remember their catalog) and ignore the future ending up on the where are they now list. While talking about Amazon and AWS, one will have to wonder where Wall Mart will end up in a decade with or without a cloud of their own?

    Microsoft – With Windows, Hyper-V and Azure (including Azure Stack), if there is any company in the industry outside of AWS or VMware that has quietly expanded its reach and positioning into storage, it is Microsoft, said Schulz.

    Microsoft IMHO has many offerings and capabilities across different dimensions as well as playing fields. There is the installed base of Windows Servers (and desktops) that have the ability to leverage Software Defined Storage including Storage Spaces Direct (S2D), ReFS, cache and tiering among other features. In some ways I’m surprised by the number of people in the industry who are not aware of Microsoft’s capabilities from S2D and the ability to configure CI as well as HCI (Hyper Converged Infrastructure) deployments, or of Hyper-V abilities, Azure Stack to Azure among others. On the other hand, I run into Microsoft people who are not aware of the full portfolio offerings or are just focused on Azure. Needless to say, there is a lot in the Microsoft storage related portfolio as well as bigger broader data infrastructure offerings.

    NetApp – Schulz thinks NetApp has the staying power to stay among the leading lights of data storage. Assuming it remains as a freestanding company and does not get acquired, he said, NetApp has the potential of expanding its portfolio with some new acquisitions. “NetApp can continue their transformation from a company with a strong focus on selling one or two products to learning how to sell the complete portfolio with diversity,” said Schulz.

    NetApp has been around and survived up to now including via various acquisitions, some of which have had mixed results vs. others. However assuming NetApp can continue to reinvent themselves, focusing on selling the entire solution portfolio vs. focus on specific products, along with good execution and some more acquisitions, they have the potential for being a top player through the next decade.

    Dell EMC – Dell EMC is another stalwart Schulz thinks will manage to stay on top. “Given their size and focus, Dell EMC should continue to grow, assuming execution goes well,” he said.

    There are some who I hear are or have predicted the demise of Dell EMC, granted some of those predicted the demise of Dell and or EMC years ago as well. Top companies can and have faded away over time, and while it is possible Dell EMC could be added to the where are they now list in the future, my bet is that at least while Michael Dell is still involved, they will be a top player through the next decade, unless they mess up on execution.

    Cloud and software defined storage data infrastructure
    Various Data Infrastructures and Resources involving Data Storage

    Huawei – Huawei is one of the emerging giants from China that are steadily gobbling up market share. It is now a top provider in many categories of storage, and its rapid ascendancy is unlikely to stop anytime soon. “Keep an eye on Huawei, particularly outside of the U.S. where they are starting to hit their stride,” said Schulz.

    In the US, you have to look or pay attention to see or hear what Huawei is doing involving data storage, however that is different in other parts of the world. For example, I see and hear more about them in Europe than in the US. Will Huawei do more in the US in the future? Good question, keep an eye on them.

    VMware – A decade ago, Storage Networking World (SNW) was by far the biggest event in data storage. Everyone who was anyone attended this twice yearly event. And then suddenly, it lost its luster. A new forum known as VMworld had emerged and took precedence. That was just one of the indicators of the disruption caused by VMware. And Schulz expects the company to continue to be a major force in storage. “VMware will remain a dominant player, expanding its role with software-defined storage,” said Schulz.

    VMware has a dominant role in data storage not just because of the relationship with Dell EMC, or because of VSAN which continues to gain in popularity, or the soon to be released VMware on AWS solution options among others. Sure all of those matters, however, keep in mind that VMware solutions also tie into and work with other legacies as well as software-defined storage solution, services as well as tools spanning block, file, object for virtual machines as well as containers.

    "Someday soon, people are going to wake up like they did with VMware and AWS," said Schulz. "That’s when they will be asking ‘When did Microsoft get into storage like this in such a big way.’"

    What the above means is that some environments may not be paying attention to what AWS, Microsoft, VMware among others are doing, perhaps discounting them as the old or existing while focusing on new, emerging what ever is trendy in the news this week. On the other hand, some environments may see the solution offerings from those mentioned as not relevant to their specific needs, or capable of scaling to their requirements.

    Keep in mind that it was not that long ago, just a few years that VMware entered the market with what by today’s standard (e.g. VSAN and others) was a relatively small virtual storage appliance offering, not to mention many people discounted and ignored VMware as a practical storage solution provider. Things and technology change, not to mention there are different needs and solution requirements for various environments. While a solution may not be applicable today, give it some time, keep an eye on them to avoid being surprised asking the question, how and when did a particular vendor get into storage in such a big way.

    Is Future Data Storage World All Cloud?

    Perhaps someday everything involving data storage will be in or part of the cloud.

    Does this mean everything is going to the cloud, or at least in the next ten years? IMHO the simple answer is no, even though I see more workloads, applications, and data residing in the cloud, there will also be an increase in hybrid deployments.

    Note that those hybrids will span local and on-premises or on-site if you prefer, as well as across different clouds or service providers. Granted some environments are or will become all in on clouds, while others are or will become a hybrid or some variation. Also when it comes to clouds, do not be scared, be prepared. Also keep an eye on what is going on with containers, orchestration, management among other related areas involving persistent storage, a good example is Dell EMCcode RexRay among others.

    Server Storage I/O resources
    Various data storage focus areas along with data infrastructures.

    What About Other Vendors, Solutions or Services?

    In addition to those mentioned above, there are plenty of other existing, new and emerging vendors, solutions, and services to keep an eye on, look into, test and conduct a proof of concept (PoC) trial as part of being an informed data infrastructure and data storage shopper (or seller).

    Keep in mind that component suppliers some of whom like Cisco also provides turnkey solutions that are also part of other vendors offerings (e.g. Dell EMC VxBlock, NetApp FlexPod among others), Broadcom (which includes Avago/LSI, Brocade Fibre Channel, among others), Intel (servers, I/O adapters, memory and SSDs), Mellanox, Micron, Samsung, Seagate and many others.

    E8, Excelero, Elastifile (software defined storage), Enmotus (micro-tiering, read Server StorageIOlab report here), Everspin (persistent and storage class memories including NVDIMM), Hedvig (software defined storage), NooBaa, Nutanix, Pivot3, Rozo (software defined storage), WekaIO (scale out elastic software defined storage, read Server StorageIO report here).

    Some other software defined management tools, services, solutions and components I’m keeping an eye on, exploring, digging deeper into (or plan to) include Blue Medora, Datadog, Dell EMCcode and RexRay docker container storage volume management, Google, HPE, IBM Bluemix Cloud aka IBM Softlayer, Kubernetes, Mangstor, OpenStack, Oracle, Retrospect, Rubrix, Quest, Starwind, Solarwinds, Storpool, Turbonomic, Virtuozzo (software defined storage) among many others

    What about those not mentioned? Good question, some of those I have mentioned in earlier Server StorageIO Update newsletters, as well as many others mentioned in my new book "Software Defined Data Infrastructure Essentials" (CRC Press). Then there are those that once I hear something interesting from on a regular basis will get more frequent mentions as well. Of course, there is also a list to be done someday that is basically where are they now, e.g. those that have disappeared, or never lived up to their full hype and marketing (or technology) promises, let’s leave that for another day.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Data Infrastructures and workloads
    Data Infrastructures Resources (Servers, Storage, I/O Networks) enabling various services

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    It is safe to say that each new year will bring new trends, techniques, technologies, tools, features, functionality as well as solutions involving data storage as well as data infrastructures. This means a usual safe bet is to say that the current year is the most exciting and has the most new things than in the past when it comes to data infrastructures along with resources such as data storage. Keep in mind that there are many aspects to data infrastructures as well as storage all of which are evolving. Who Will Be At Top Of Storage World Next Decade? What say you?

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book "Software-Defined Data Infrastructure Essentials" (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    GDPR (General Data Protection Regulation) Resources Are You Ready?

    server storage I/O data infrastructure trends
    Updated 6/29/17

    What Is GDPR

    If your initial response is that you are not in Europe and do not need to be concerned about GDPR you might want to step back and review that thought. While it is possible that some organizations may not be affected by GDPR in Europe directly, there might be indirect considerations. For example, GDPR, while focused on Europe, has ties to other initiatives in place or being planned for elsewhere in the world. Likewise unlike earlier regulatory compliance that tended to focus on specific industries such as healthcare (HIPPA and HITECH) or financial (SARBOX, Dodd/Frank among others), these new regulations can be more far-reaching.

    Where To Learn More

    Acronis GDPR Resources

    • Acronis Outlines GDPR position

    Quest GDPR Resources

    Microsoft and Azure Cloud GDPR Resources

    Do you have or know of relevant GDPR information and resources? Feel free to add them via comments or send us an email, however please watch the spam and sales pitches as they will be moderated.

    What This All Means

    Now is the time to start planning, preparing for GDPR if you have not done so and need to, as well as becoming more generally aware of it and other initiatives. One of the key takeaways is that while the word compliance is involved, there is much more to GDPR than just compliance as we have seen in the part. With GDPR and other initiatives data protection becomes the focus including privacy, protect, preserve, secure, serve as well as manage, have insight, awareness along with associated reporting.

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Dell EMC World 2017 Day One news announcement summary

    server storage I/O trends

    Dell EMC World 2017 Day One news announcement summary

    This is the first day of the first combined Dell EMC World 2017 being held in Las Vegas Nevada. Last year’s event in Las Vegas was the end of the EMC World, while this being the first of the combined Dell EMC World events that succeeded its predecessors.

    What this means is an expanded focus because of the new Dell EMC that has added servers among other items to the event focus. Granted, EMC had been doing servers via its VCE and converged divisions, however with the Dell EMC integration completed as of last fall, the Dell Server group is now part of the Dell EMC organization.

    The central theme of this Dell EMC world is REALIZE with a focus on four pillars:

    • Digital Transformation (Pivotal focus) of applications
    • IT Transformation (Dell EMC, Virtustream, VMware) data center modernization
    • Workforce transformation (Dell Client Solutions) devices from mobile to IoT
    • Information Security (RSA and Secureworks)

    software defined data infrastructures SDDI and SDDC

    What Did Dell EMC Announce Today

    Note that while there are focus areas of the different Dell Technologies business units aligned to the pillars, there is also leveraging across those areas and groups. For example, VMware NSX spans into security, and  PowerEdge servers span into other pillars as a core data infrastructure building block.

    What Dell EMC and Dell Technologies announced today.

    • Wave of Innovations to help customers realize digital transformation
    • New 14th generation PowerEdge Servers that are core building blocks for data infrastructures
    • Flexible consumption models (financing and more) from desktop to data center
    • Hyper-Converged Infrastructure (HCI), Converged (CI) and Cloud like systems
    • New All-Flash (ADA) SSD Storage Systems (VMAX, XtremIO X2, Unity, SC, Isilon)
    • Integrated Data Protection Appliance (IDPA) and Cloud Protection solutions
    • Using Gen14 servers several Software Defined Storage (SDS) enhancements
    • Open Networking and software-defined networks (SDN) with 25G
    • Last week Dell EMC announced Microsoft Azure Stack hybrid cloud solutions

    New 14th generation PowerEdge Servers that are core building blocks for data infrastructures

    Dell EMC has announced the 14th generation of Intel-powered Dell EMC PowerEdge server portfolio systems. These includes servers that get defined with software for software-defined data centers (SDDC), software-defined data infrastructures (SDDI) for the cloud, virtual, the container as well as storage among other applications. Target application workloads and environments range from high-performance compute (HPC), and high-productivity (or profitability) compute (the other HPC), super compute (SC), little data and big data analytics, legacy and emerging business applications as well as cloud and beyond. Enhancements besides new Intel processor technology includes enhanced iDRAC, OpenManage, REST interface, QuickSync, Secure Boot among other management, automation, security, performance, and capacity updates.

    Other Dell EMC enhancements with Gen14 include support for various NVDIMM to enable persistent memory also known as storage class memories such as 3D Xpoint among others. Note at this time, Dell EMC is not saying much about speeds, feeds and other details, stay tuned for more information on these in the weeks and months to come.

    Dell EMC has also been leaders with deploying NVMe from PCIe flash cards to 8639 U.2 devices such as 2.5” drives. Thus it makes sense to see continued adoption and deployment of those devices along with SAS, SATA support. Note that Broadcom (formerly known as Avago) recently announced the release of their PCIe SAS, SATA and NVMe based adapters.

    The reason this is worth mentioning is that in the past Dell has OEM sourced Avago (formerly known as LSI) based adapters. Given Dell EMC use of NVMe drives, it only makes sense to put two and two together.

    Let’s wait a few months to see what the speeds, feeds, and specifications are to put the rest of the puzzle together. Speaking of NVMe, also look for Dell EMC to also supporting PCIe AIC and U.2 (8639) NVMe devices, also leverage M.2 Next Generation Form Factor (NGFF) aka Gum sticks as boot devices.

    While these are all Intel focused, I would expect Dell EMC not to sit back, instead, watch for what they do with other processors and servers including ARMs among others.

    Increased support for more GPUs to support VDI and other graphic intensive workloads such as video rendering, imaging among others. Part of enhanced GPU support is improvements (multi-vector cooling) to power and cooling including sensing the type of PCIe card, and then adjusting cooling fans and subsequent power draw accordingly. The benefit should be more proper cooling to reduce power to support more work and productivity.

    Flexible consumption models (financing and more) from desktop to data center

    Dell Technologies has announced several financing, procurement, and consumption models with cloud-like flexible options for different IT and data center, along with mobile device technologies. These range from licensing to deployment as a service, consumption and other options via Dell Financial Services (DFS).

    Highlights include:

    • DFS Flex on Demand is available now in select countries globally.
    • DFS Cloud Flex for HCI is available now for Dell EMC VxRail and Dell EMC XC Series and has planned availability for Q3 2017 in Dell EMC VxRack Systems.
    • PC as a Service is available now in select countries globally.
    • Dell EMC VDI Complete Solutions are available now in select countries globally.
    • DFS Flex on Demand is available now in select countries globally.
    • DFS Cloud Flex for HCI is available now for Dell EMC VxRail and Dell EMC XC Series and has planned VxRack systems in Q3 2017.
    • PC as a Service solution is available now in select countries globally.
    • Dell EMC VDI Complete Solutions are available now in select countries.
    • Dell Technologies transformation license agreement (TLA) is available now in select countries

    Hyper-Converged Infrastructure (HCI), Converged (CI) and Cloud like systems

    Enhancements to VxRail system, VxRACK Systems, and XC Series leveraging Del EMC Gen14 PowerEdge servers along with other improvements. Note that this also includes continued support for VMware, Microsoft as well as Nutanix software-defined solutions.

    New All-Flash (ADA) SSD Storage Systems (VMAX, XtremIO X2, Unity, SC, Isilon)

    Storage system enhancements include from high-end (VMAX and XtremIO) to mid-range (Unity and SC) along with scale-out NAS (Isilon)

    Highlights of the announcements include:

    • New VMAX 950F all flash array (AFA)
    • New XtremIO X2 with enhanced software, more powerful hardware
    • New Unity AFA systems
    • New SC5020 midrange hybrid storage
    • New generation of Isilon storage with improved performance, capacity, density

    Integrated Data Protection Appliance (IDPA) and Cloud Protection solutions

    Data protection enhancement highlights include:

    • New Turnkey Integrated Data Protection Appliance (IDPA) with four models (DP5300, DP5800, DP8300, and DP8800) starting at 34 TB usable scaling up to 1PB usable. Data services including encryption, data footprint reduction such as dedupe, remote monitoring, Maintenance service dispatch, along with application integration. Application integration includes MongoDB, Hadoop, MySQL.

    • Enhanced cloud capabilities powered by Data Domain virtual edition (DD VE 3.1) along with data protection suite enable data to be protected too, and restored from Amazon Web Services (AWS) Simple Storage Service (S3) as well as Microsoft Azure.

    Open Networking and software-defined networks (SDN) with 25G

    Dell EMC Open Networking highlights include:

    • Dell EMCs first 25GbE open networking top of rack (TOR) switch including S5100-ON series (With OS10 enterprise edition software) complimenting new PowerEdge Gen14 servers with native 25GbE support. Switches support 100GbE uplinks fabric connectivity for east-west (management) network traffic. Also announced is the S4100-ON series and N1100-ON series that are in addition to recently announce N3100-ON and N2100-ON switches.

    • Dell EMCs first optimized Open Networking platform for unified storage network switching including support for 16Gb/32GB Fibre Channel

    • New Network Function Virtualization (NFV) and IoT advisory consulting services

    Note that Dell EMC is announcing the availability of these networking solutions in Dell Technologies 2018 fiscal year which occurs before the traditional calendar year.

    Using Gen14 servers, several Software Defined Storage (SDS) enhancements

    Dell EMC announced enhancements to their Software Defined Storage (SDS) portfolio that leveraging the PowerEdge 14th generation server portfolio. These improvements include ScaleIO, Elastic Cloud Storage (ECS), IsilonSD Edge and Preview of Project Nautilus.

    Where to learn more

    What this all means

    This is a summary of what has been announced so far on the first morning of the first day of the first new Dell EMC world. Needless to say, there is more detail to look at for the above announcements from speeds, feeds, functionality and related topics that will get addressed in subsequent posts. Overall this is a good set of announcements expanding capabilities of the combined Dell EMC while enhancing existing systems as well as well as solutions.

    Ok, nuff said (for now…)

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book "Software-Defined Data Infrastructure Essentials" (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Broadcom aka Avago aka LSI announces SAS SATA NVMe Adapters with RAID

    server storage I/O trends

    Broadcom aka Avago aka LSI announces SAS SATA NVMe Adapters with RAID

    In case you missed it, Broadcom formerly known as Avago who bought the LSI adapter and RAID card business announced shipping new SAS, SATA and NVMe devices.

    While SAS and SATA are well established continuing to be deployed for both HDD as well as flash SSD, NVMe continues to evolve with a bright future. Likewise, while there is a focus on software-defined storage (SDS), software defined data centers (SDDC) and software defined data infrastructures (SDDI) along with advanced parity RAID including erasure codes, object storage among other technologies, there is still a need for adapter cards including traditional RAID.

    Keep in mind that while probably not meeting the definition of some software-defined aficionados, the many different variations, permutations along with derivatives of RAID from mirror and replication to basic parity to advanced erasure codes (some based on Reed Solomon aka RAID 2) rely on software. Granted, some of that software is run on regular primary server processors, some on packaged in silicon via ASICs or FPGAs, or System on Chips (SOC), RAID on Chip (RoC) as well as BIOS, firmware, drivers as well as management tools.

    SAS, SATA and NVMe adapters

    For some environments cards such as those announced by Broadcom are used in passthru mode effectively as adapters for attaching SAS, SATA and NVMe storage devices to servers. Those servers may be deployed as converged infrastructures (CI), hyper-converged infrastructures (HCI), Cluster or Cloud in Box (CiB) among other variations. To name names you might find the above (or in the not so distant future) in VMware vSAN or regular vSphere based environments, Microsoft Windows Server, Storage Spaces Direct (S2D) or Azure Stack, OpenStack among other deployments (check your vendors Hardware Compatibility Lists aka HCLs). In some cases these cards may be adapters in passthru mode, or using their RAID (support various by different software stacks). Meanwhile in other environments, the more traditional RAID features are still used spanning Windows to Linux among others.

    Who Is Broadcom?

    Some of you may know of Broadcom having been around for many years with a focus on networking related technologies. However some may not realize that Avago bought Broadcom and changed their name to Broadcom. Here is a history that includes more recent acquisitions such as Brocade, PLX, Emulex as well as LSI. Some of you may recall Avago buying LSI (the SAS, SATA, PCIe HBA, RAID and components) business not sold to NetApp as part of Engenio. Also recall that Avago sold the LSI flash SSD business unit to Seagate a couple of years ago as part of its streamlining. That’s how we get to where we are at today with Broadcom aka formerly known as Avago who bought the LSI adapter and RAID business announcing new SAS, SATA, NVMe cards.

    What Was Announced?

    Broadcom has announced cards that are multi-protocol supporting Serial Attached SCSI (SAS), SATA/AHCI as well as NVM Express (NVMe) as basic adapters for attaching storage (HDD, SSD, storage systems) along with optional RAID as well as cache support. These cards can be used in application servers for traditional, as well as virtualized SDDC environments, as well as storage systems or appliances for software-defined storage among other uses. The basic functionality of these cards is to provide high performance (IOPs and other activity, as well as bandwidth) along with low latency combined with data protection as well as dense connectivity.

    Specific features include:

    • Broadcom’s Tri-Mode SerDes Technology enables the operation of NVMe, SAS or SATA devices in a single drive bay, allowing for endless design flexibility.
    • Management software including LSI Storage Authority (LSA), StorCLI, HII (UEFI)
    • Optional CacheVault(R) flash cache protection
    • Physical dimension Low Profile 6.127” x 2.712”
    • Host bus type x8 lane PCIe Express 3.1
    • Data transfer rates SAS-3 12Gbs; NVMe up to 8 GT/s PCIe Gen 3
    • Various OS and hypervisors host platform support
    • Warranty 3 yrs, free 5×8 phone support, advanced replacement option
    • RAID levels 0, 1, 5, 6, 10, 50, and 60

    Note that some of the specific feature functionality may be available at a later date, check with your preferred vendors HCL

    Specification

    9480 8i8e

    9440 8i

    9460 8i

    9460 16i

    Image

    Internal Ports

    8

     

    8

    16

    Internal Connectors

    2 x Mini-SAS HD x4 SFF-8643

    2 x Mini-SAS HD x4 SFF-8643

    2 x Mini-SAS HD x4 SFF-8643

    4 Mini-SAS HD x4
    SFF-8643

    External Ports

    8

     

     

     

    External Connectors

    2 x Mini-SAS HD SFF8644

     

     

     

    Cache Protection

    CacheVault CVPM05

     

    CacheVault CVPM05

    CacheVault CVPM05

    Cache Memory

    2GB 2133 MHz DDR4 SDRAM

     

    2GB 2133 MHz DDR4 SDRAM

    4GB 2133 MHz DDR4 SDRAM

    Devices Supported

    SAS/SATA: 255, NVMe: 4 x4, up to 24 x2 or x4*

    SAS/SATA: 63, NVMe: 4 x4, up to 24 x2 or x4*

    SAS/SATA: 255, NVMe: 4 x4, up to 24 x2 or x4*

    SAS/SATA: 255, NVMe: 4 x4, up to 24 x2 or x4*

    I/O Processors (SAS Controller)

    SAS3516 dual-core RAID-on-Chip (ROC)

    SAS3408 I/O controller (IOC)

    SAS3508 dual-core RAID-on-Chip (ROC)

    SAS3516 dual-core RAID-on-Chip (ROC)

    In case you need a refresher on SFF cable types, click on the following two images which take you to Amazon.com where you can learn more, as well as order various cable options. PC Pit Stop has a good selection of cables (See other SFF types), connectors and other accessories that I have used, along with those from Amazon.com and others.

    Available via Amazon.com sff 8644 8643 sas mini hd cable
    Left: SFF 8644 Mini SAS HD (External), Right SFF-8643 Mini SAS HD (internal) Image via Amazon.com

    Available via Amazon.com sff 8644 8642 sas mini hd cable
    Left: SFF 8643 Mini SAS HD (Internal), Right SFF-8642 SATA with power (internal) Image via Amazon.com

    Wait, Doesnt NVMe use PCIe

    For those who are not familiar with NVMe and in particular U.2 aka SFF 8639 based devices, physically they look the same (almost) as a SAS device connector. The slight variation is if you look at a SAS drive, there is a small tab to prevent plugging into a SATA port (recall you can plug SATA into SAS. For SAS drives that tab is blank, however on the NVMe 8639 aka U.2 drives (below left) that tab has several connectors which are PCIe x4 (single or dual path).

    What this means is that the PCIe x4 bus electrical signals are transferred via a connector, to backplane chassis to 8639 drive slot to the drive. Those same 8639 drive slots can also have a SAS SATA connection using their traditional connectors enabling a converged or hybrid drive slot so to speak. Learn more about NVMe here (If the Answer is NVMe, then what were and are the questions?) as well as at www.thenvmeplace.com.

    NVMe U.2 8639 driveNVMe U.2 8639 sas sata nvme drive
    Left NVMe U.2 drive showing PCIe x4 connectors, right, NVMe U.2 8639 connector

    Who Is This For?

    These cards are applicable for general purpose IT and other data infrastructure environments in traditional servers among others uses. They are also applicable for systems builders, integrators and OEMs whom you may be buying your current systems from, or future ones.

    Where to Learn More

    The following are additional resources to learn more about vSAN and related technologies.

    What this all means

    Even as the industry continues to talk and move towards more software-defined focus, even for environments that are serverless, there is still need for hardware somewhere. These adapters are a good sign of the continued maturing cycle of NVMe to be well positioned into the next decade and beyond, while also being relevant today. Likewise, even though the future involves NVMe, there is a still a place for SAS along with SATA to coexist in many environments. For some environment there is a need for traditional RAID while for others simply the need for attachment of SAS, SATA and NVMe devices. Overall, a good set of updates, enhancements and new technology for today and tomorrow, now, when do I get some to play with? ;).

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book "Software-Defined Data Infrastructure Essentials" (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    VMware vSAN V6.6 Part IV (HCI scaling ROBO and data centers today)

    server storage I/O trends

    VMware vSAN V6.6 Part IV (HCI scaling ROBO and data centers today)

    In case you missed it, VMware announced vSAN v6.6 hyper-converged infrastructure (HCI) software defined data infrastructure solution. This is the fourth of a five-part series about VMware vSAN V6.6. View Part I here, Part II (just the speeds feeds please) is located here, part III (reducing cost and complexity) located here, as well as part V here (VMware vSAN evolution, where to learn more and summary).

    VMware vSAN 6.6
    Image via VMware

    For those who are not aware, vSAN is a VMware virtual Storage Area Network (e.g. vSAN) that is software-defined, part of being a software-defined data infrastructure (SDDI) and software-defined data center (SDDC). Besides being software-defined vSAN is HCI combining compute (server), I/O networking, storage (space and I/O) along with hypervisors, management, and other tools.

    Scaling HCI for ROBO and data centers today and for tomorrow

    Scaling with stability for today and tomorrow. This includes addressing your applications Performance, Availability, Capacity and Economics (PACE) workload requirements today and for the future. By scaling with stability means boosting performance, availability (data protection, security, resiliency, durable, FTT), effective capacity without one of those attributes compromising another.

    VMware vSAN data center scaling
    Image via VMware

    Scaling today for tomorrow also means adapting to today’s needs while also flexible to evolve with new application workloads, hardware as well as a cloud (public, private, hybrid, inter and intra-cloud). As part of continued performance improvements, enhancements to optimize for higher performance flash SSD including NVMe based devices.

    VMware vSAN cloud analytics
    Image via VMware

    Part of scaling with stability means enhancing performance (as well as productivity) or the effectiveness of a solution. Keep in mind that efficiency is often associated with storage (or server or network) space capacity savings or reductions. In that context then effectiveness means performance and productivity or how much work can be done with least overhead impact. With vSAN, V6.6 performance enhancements include reduced checksum overhead, enhanced compression, and deduplication, along with destaging optimizations.

    Other enhancements that help collectively contribute to vSAN performance improvements include VMware object handling (not to be confused with cloud or object storage S3 or Swift objects) as well as faster iSCSI for vSAN. Also improved are more accurate refined cache sizing guidelines. Keep in mind that a little bit of NAND flash SSD or SCM in the right place can have a significant benefit, while a lot of flash cache costs much cash.

    Part of enabling and leveraging new technology today includes support for larger capacity 1.6TB flash SSD drives for cache, as well as lower read latency with 3D XPoint and NVMe drives such as those from Intel among others. Refer to the VMware vSAN HCL for current supported devices which continue evolve along with the partner ecosystem. Future proofing is also enabled where you can grow from today to tomorrow as new storage class memories (SCM) among other flash SSD as well as NVMe enhanced storage among other technologies are introduced into the market as well as VMware vSAN HCL.

    VMware vSAN and data center class applications
    Image via VMware

    Traditional CI and in particular many HCI solutions have been optimized or focused on smaller application workloads including VDI resulting in the perception that HCI, in general, is only for smaller environments, or larger environment non-mission critical workloads. With vSAN V6.6 VMware is addressing and enabling larger environment mission critical applications including Intersystem Cache medical health management software among others. Other application workload extensions including support for higher performance demanding Hadoop big data analytics, a well as extending virtual desktop infrastructure (VDI) workspace with XenDesktop/XenApp, along with Photon 1.1 container support.

    What about VMware vSAN 6.6. Packaging and License Options

    As part of vSAN 6.6 VMware several solution bundle packaged options for the data center as well as smaller ROBO environment. Contact your VMware representative or partner to learn more about specific details.

    VMware vSAN cloud analytics
    Image via VMware

    VMware vSAN cloud analytics
    Image via VMware

    Where to Learn More

    The following are additional resources to find out more about vSAN and related technologies.

    What this all means

    Continue reading more about VMware vSAN 6.6 in part I here, part II (just the speeds feeds please) is located here, part III (reducing cost and complexity) located here as well as part V here (VMware vSAN evolution, where to learn more and summary).

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the Spring 2017 release of his new book “Software-Defined Data Infrastructure Essentials” (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.