Category: Lab report

Posted on

Data Protection Diaries Fundamental Point In Time Granularity Points of Interest

Data Protection Diaries Fundamental Point In Time Granularity

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulz – www.storageioblog.com November 26, 2017

This is Part 5 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 4 Data Protection Recovery Points (Archive, Backup, Snapshots, Versions), and click here to view the next post Part 6 Data Protection Security Logical Physical Software Defined.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection points of granularity, addressing different layers and stack altitude (higher application and lower system level) Chapter 10 . among others.

Point-in-Time Protection Granularity Points of Interest

SDDC SDDI Data Protection Recovery consistency points
Figure 10.1 Recovery and consistency points

Figure 10.1 above is a refresh from previous posts about the role and importance of having various recovery points at different time intervals to enable data protection (and restoration). Building upon figure 10.1, figure 10.5 looks at different granularity of where and how data should be protected. Keep in mind that everything is not the same, so why treat everything the same with the same type of protection?

Figure 10.5 shows backup and Data Protection focus, granularity, and coverage. For example, at the top left is less frequent protection of the operating system, hypervisors, and BIOS, UEFI settings. At the middle left is volume, or device level protection (full, incremental, differential), along with various views on the right ranging from protecting everything, to different granularity such as file system, database, database logs and journals, and operating system (OS) and application software, along with settings.

SDDC SDDI Different Protection Granularity
Figure 10.5 Backup and data protection focus, granularity, and coverage

In Figure 10.5, note that the different recovery point focus and granularity also take into consideration application and data consistency (as well as checkpoints), along with different frequencies and coverage (e.g. full, partial, incremental, incremental forever, differential) as well as retention.

Tip – Some context is needed about object backup and backing up objects, which can mean different things. As mentioned elsewhere, objects refer to many different things, including cloud and object storage buckets, containers, blobs, and objects accessed via S3 or Swift, among other APIs. There are also database objects and entities, which are different from cloud or object storage objects.

Another context factor is that an object backup can refer to protecting different systems, servers, storage devices, volumes, and entities that collectively comprise an application such as accounting, payroll, or engineering, vs. focusing on the individual components. An object backup may, in fact, be a collection of individual backups, PIT copies, and snapshots that combined represent what’s needed to restore an application or system.

On the other hand, the content of a cloud or object storage repository ( buckets, containers, blobs, objects, and metadata) can be backed up, as well as serve as a destination target for protection.

Backups can be cold and off-line like archives, as well as on-line and accessible. However, the difference between the two, besides intended use and scope, is granularity. Archives are intended to be coarser and less frequently accessed, while backups can be more frequently and granular accessed. Can you use a backup for an archive and vice versa? A qualified yes, as an archive could be a master gold copy such as an annual protection copy, in addition to functioning in its role as a compliance and retention copy. Likewise, a full backup set to long-term retention can provide and enable some archive functions.

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

A common theme in this series as well as in my books, webinars, seminars and general approach to data infrastructures, data centers and IT in general is that everything is not the same, why treat it all the same? What this means is that there are differences across various environments, data centers, data infrastructures, applications, workloads and data. There are also different threat risks scenarios (e.g. threat vectors and attack surface if you like vendor industry talk) to protect against.

Rethinking and modernizing data protection means using new (and old) tools in new ways, stepping back and rethinking what to protect, when, where, why, how, with what. This also means protecting in different ways at various granularity, time intervals, as well as multiple layers or altitude (higher up the application stack, or lower level).

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 6 Data Protection Security Logical Physical Software Defined.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

Data Infrastructure Data Protection Diaries Fundamental Security Logical Physical

Data Infrastructure Data Protection Security Logical Physical

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulz – www.storageioblog.com November 26, 2017

This is Part 6 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 5 – Point In Time Data Protection Granularity Points of Interest, and click here to view the next post Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Infrastructure and Data Protection security including logical as well as physical from chapter 10 , 13 and 14 among others.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

There are many different aspects of security pertaining to data infrastructures that span various technology domains or focus areas from higher level application software to lower level hardware, from legacy to cloud an software-defined, from servers to storage and I/O networking, logical and physical, from access control to intrusion detection, monitoring, analytics, audit, monitoring, telemetry logs, encryption, digital forensics among many others. Security should not be an after thought of something done independent of other data infrastructure, data center and IT functions, rather integrated.

Security Logical Physical Software Defined

Physical security includes locked doors of facilities, rooms, cabinets or devices to prevent un-authorized access. In addition to locked doors, physical security also includes safeguards to prevent accidental or intentional acts that would compromise the contents of a data center including data Infrastructure resources (servers, storage, I/O networks, hardware, software, services) along with the applications that they support.

Logical security includes access controls, passwords, event and access logs, encryption among others technologies, tools, techniques. Figure 10.11 shows various data infrastructure security–related items from cloud to virtual, hardware and software, as well as network services. Also shown are mobile and edge devices as well as network connectivity between on-premises and remote cloud services. Cloud services include public, private, as well as hybrid and virtual private clouds (VPC) along with virtual private networks (VPN). Access logs for telemetry are also used to track who has accessed what and when, as well as success along with failed attempts.

Certificates (public or private), Encryption, Access keys including .pem and RSA files via a service provider or self-generated with a tool such as Putty or ssh-keygen among many others. Some additional terms including Two Factor Authentication (2FA), Subordinated, Role based and delegated management, Single Sign On (SSO), Shared Access Signature (SAS) that is used by Microsoft Azure for access control, Server Side Encryption (SSE) with various Key Management System (KMS) attributes including customer managed or via a third-party.

SDDC SDDI Data Protection Security
Figure 10.11 Various physical and logical security and access controls

Also shown in figure 10.11 are encryption enabled at various layers, levels or altitude that can range from simple to complex. Also shown are iSCSI IPsec and CHAP along with firewalls, Active Directory (AD) along with Azure AD (AAD), and Domain Controllers (DC), Group Policies Objects (GPO) and Roles. Note that firewalls can exist in various locations both in hardware appliances in the network, as well as software defined network (SDN), network function virtualization (NFV), as well as higher up.

For example there are firewalls in network routers and appliances, as well as within operating systems, hypervisors, and further up in web blogs platforms such as WordPress among many others. Likewise further up the stack or higher in altitude access to applications as well as database among other resources is also controlled via their own, or in conjunction with other authentication, rights and access control including ADs among others.

A term that might be new for some is attestation which basically means to authenticate and be validated by a server or service, for example, a host guarded server attests with a attestation server. What this means is that the host guarded server (for example Microsoft Windows Server) attests with a known attestation server, that looks at the Windows server comparing it to known good fingerprints, profiles, making sure it is safe to run as a guarded resources.

Other security concerns for legacy and software defined environments include secure boot, shield VMs, host guarded servers and fabrics (networks or clusters of servers) for on-premises, as well as cloud. The following image via Microsoft shows an example of shielded VMs in a Windows Server 2016 environment along with host guarded service (HGS) components ( see how to deploy here).


Via Microsoft.com Guarded Hosts, Shielded VMs and Key Protection Services

Encryption can be done in different locations ranging from data in flight or transit over networks (local and remote), as well as data at rest or while stored. Strength of encryption is determined by different hash and cipher codes algorithms including SHA among others ranging from simple to more complex. The encryption can be done by networks, servers, storage systems, hypervisors, operating systems, databases, email, word and many other tools at granularity from device, file systems, folder, file, database, table, object or blob.

Virtual machine and their virtual disks ( VHDX and VMDK) can be encrypted, as well as migration or movements such as vMotions among other activities. Here are some VMware vSphere encryption topics, along with deep dive previews from VMworld 2016 among other resources here, VMware hardening guides here (NSX, vSphere), and a VMware security white paper (PDF) here.

Other security-related items shown in Figure 10.11 include Lightweight Direct Access Protocol (LDAP), Remote Authentication Dial-In User Service (RADIUS), and Kerberos network authentication. Also shown are VPN along with Secure Socket Layer (SSL) network security, along with security and authentication keys, credentials for SSH remote access including SSO. The cloud shown in figure 10.11 could be your own private using AzureStack, VMware (on-site, or public cloud such as IBM or AWS), OpenStack among others, or a public cloud such as AWS, Azure or Google (among others).

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

There are many different aspects, as well as layers of security from logical to physical pertaining to data centers, applications and associated data Infrastructure resources, both on-premises and cloud. Security for legacy and software defined environments needs to be integrated as part of various technology domain focus areas, as well as across them including data protection. The above is a small sampling of security related topics with more covered in various chapters of SDDI Essentials as well as in my other books, webinars, presentations and content.

From a data protection focus, security needs to be addressed from a physical who has access to primary and protection copies, what is being protected against and where, as well as who can access logically protection copes, as well as the configuration, settings, certificates involved in data protection. In other words, how are you protecting your data protection environment, configuration and deployment. Data protection copies need to be encrypted to meet regulations, compliance and other requirements to guard against loss or theft, accidental or intentional. Likewise access control needs to be managed including granting of roles, security, authentication, monitoring of access, along with revocation.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 7 Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

Data Protection Diaries Tools Technologies Toolbox Buzzword Bingo Trends

Fundamental Tools, Technologies, Toolbox, Buzzword Bingo Trends

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

This is Part 7 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 6 Data Protection Security Logical Physical Software Defined, and click here to view the next post Part 8 Walking The Data Protection Talk What I Do.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection related tools, technologies, trends as companion to other posts in this series, as well as across various chapters from the SDDI book.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends

There are many data Infrastructure related topics, technologies, tools, trends, techniques and tips that pertain to data protection, many of which have been covered in this series of posts already, as well as in the SDDI Essentials book, and elsewhere. The following are some additional related data Infrastructure data protection topics, tools, technologies.

Buzzword Bingo is a popular industry activity involving terms, trends, tools and more, read more here, here, and here. The basic idea of buzzword bingo is when somebody starts mentioning lots of buzzwords, buzz terms, buzz trends at some point just say bingo. Sometimes you will get somebody who asks what that means, while others will know, perhaps get the point to move on to what’s relevant vs. talking the talk or showing how current they are on industry activity, trends and terms.

Just as everything is not the same across different environments, there are various size and focus from hyper-scale clouds and managed service providers (MSP) server (and storage along with applications focus), smaller and regional cloud, hosting and MSPs, as well as large enterprise, small medium enterprise (SME), small medium business (SMB), remote office branch office (ROBO), small office home office (SOHO), prosumer, consumer and client or edge. Sometimes you will hear server vs. edge or client focus, thus context is important.

Data protection just like data infrastructures span servers, storage, I/O networks, hardware, software, clouds, containers, virtual, hypervisors and related topics. Otoh, some might view data protection as unique to a particular technology focus area or domain. For example, I once had backup vendor tell me that backups and data protection was not a storage topic, can you guess which vendor did not get recommend for data protection of data stored on storage?

Data gets protected to different target media, mediums or services including HDDs, SSD, tape, cloud, bulk and object storage among others in various format from native to encapsulated in save sets, zips, tar ball among others.

Bulk storage can be on-site, on-premises low-cost tape, disk (file, block or object) as well as off-site including cloud services such as AWS S3 (buckets and objects), Microsoft Azure (containers and blobs), Google among others using various Access ( Protocols, Personalities, Front-end, Back-end) technologies. Which type of data protection storage medium, location or service is best depends on what you are trying to do, along with other requirements.

SDDC SDDI data center data protection toolbox
Data Protection Toolbox

SDDC SDDI Object Storage Architecture
Figure 3.18 Generic Object (and Blob) architecture with Buckets (and Containers)

Object Storage

Before discussing Object Storage lets take a step back and look at some context that can clarify some confusion around the term object. The word object has many different meanings and context, both inside of the IT world as well as outside. Context matters with the term object such as a verb being a thing that can be seen or touched as well as a person or thing of action or feeling directed towards.

Besides a person, place or physical thing, an object can be a software defined data structure that describes something. For example, a database record describing somebody’s contact or banking information, or a file descriptor with name, index ID, date and time stamps, permissions and access control lists along with other attributes or metadata. Another example is an object or blob stored in a cloud or object storage system repository, as well as an item in a hypervisor, operating system, container image or other application.

Besides being a verb, object can also be a noun such as disapproval or disagreement with something or someone. From an IT context perspective, object can also refer to a programming method (e.g. object oriented programming [oop], or Java [among other environments] objects and class’s) and systems development in addition to describing entities with data structures.

In other words, a data structure describes an object that can be a simple variable, constant, complex descriptor of something being processed by a program, as well as a function or unit of work. There are also objects unique or with context to specific environments besides Java or databases, operating systems, hypervisors, file systems, cloud and other things.

SDDC SDDI Object Storage Example
Figure 3.19 AWS S3 Object storage example, objects left and descriptive names on right

The role of object storage (view more at www.objectstoragecenter.com) is to provide low-cost, scalable capacity, durable availability of data including data protection copies on-premises or off-site. Note that not all object storage solutions or services are the same, some are immutable with write once read many (WORM) like attributes, while others non-immutable meaning that they can be not only appended to, also updated to page or block level granularity.

Also keep in mind that some solutions and services refer to items being stored as objects while others as blobs, and the name space those are part of as a bucket or container. Note that context is important not to confuse an object container with a docker, kubernetes or micro services container.

Many applications and storage systems as well as appliances support as back-end targets cloud access using AWS S3 API (of AWS S3 service or other solutions), as well as OpenStack Switch API among others. There are also many open source and third-party tools for working with cloud storage including objects and blobs. Learn more about object storage, cloud storage at www.objectstoragecenter.com as well as in chapters 3, 4, 13 and 14 in SDDI Essentials book.

S3 Simple Storage Service

Simple Storage Service ( S3) is the Amazon Web Service (AWS) cloud object storage service that can be used for bulk and other storage needs. The S3 service can be accessed from within AWS as well as externally via different tools. AWS S3 supports large number of buckets and objects across different regions and availability zones. Objects can be stored in a hierarchical directory structure format for compatibility with existing file systems or as a simple flat name space.

Context is important with data protection and S3 which can mean the access API, or AWS service. Likewise context is important in that some solutions, software and services support S3 API access as part of their front-end (e.g. how servers or clients access their service), as well as a back-end target (what they can store data on).

Additional AWS S3 (service) and related resources include:

Data Infrastructure Environments and Applications

Data Infrastructure environments that need to be protected include legacy, software defined (SDDC, SDDI, SDS), cloud, virtual and container based, as well as clustered, scale-out, converged Infrastructure (CI), hyper-converged Infrastructure (HCI) among others. In addition to data protection related topics already converged in the posts in this series (as well as those to follow), a related topic is Data Footprint Reduction ( DFR). DFR comprises several different technologies and techniques including archiving, compression, compaction, deduplication (dedupe), single instance storage, normalization, factoring, zip, tiering and thin provisioning among many others.

Data Footprint Reduction (DFR) Including Dedupe

There is a long-term relationship with data protection and DFR in that to reduce the impact of storing more data, traditional techniques such as compression and compaction have been used, along with archive and more recently dedupe among others. In the Software Defined Data Infrastructure Essentials book there is an entire chapter on DFR ( chapter 11), as well as related topics in chapters 8 and 13 among others. For those interested in DFR and related topics, there is additional material in my books Cloud and Virtual Data Storage Networking (CRC Press), along with in The Green and Virtual Data Center (CRC Press), as well as various posts on StorageIOblog.com and storageio.com. Figure 11.4 is from Software Defined Data Infrastructure Essentials showing big picture of various places where DFR can be implemented along with different technologies, tools and techniques.

SDDC, SDI, SDDI DFR Dedupe
Figure 11.4 Various points of interest where DFR techniques and technology can be applied

Just as everything is not the same, there are different DFR techniques along with implementations to address various application workload and data performance, availability, capacity, economics (PACE) needs. Where is the best location for DFR that depends on your objectives as well as what your particular technology can support. However in general, I recommend putting DFR as close to where the data is created and stored as possible to maximize its effectiveness which can be on the host server. That however also means leveraging DFR techniques downstream where data gets sent to be stored or protected. In other words, a hybrid DFR approach as a companion to data protection should use various techniques, technologies in different locations. Granted, your preferred vendor might only work in a given location or functionality so you can pretty much guess what the recommendations will be ;) .

Tips, Recommendations and Considerations

Additional learning experiences along with common questions (and answers), appendices, as well as tips can be found here.

General action items, tips, considerations and recommendations include:

    • Everything is not the same; different applications with SLO, PACE, FTT, FTM needs
    • Understand the 4 3 2 1 data protection rule and how to implement it.
    • Balance rebuild performance impact and time vs. storage space overhead savings.
    • Use different approaches for various applications and environments.
    • What is best for somebody else may not be best for you and your applications.
    • You cant go forward in the future after a disaster if you cant go back
    • Data protection is a shared responsibility between vendors, service providers and yourself
    • There are various aspects to data protection and data Infrastructure management

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

There are many different buzzword, buzz terms, buzz trends pertaining to data infrastructure and data protection. These technologies span legacy and emerging, software-defined, cloud, virtual, container, hardware and software. Key point is what technology is best fit for your needs and applications, as well as how to use the tools in different ways (e.g. skill craft techniques and tradecraft). Keep context in mind when looking at and discussing different technologies such as objects among others.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 8 Walking The Data Protection Talk.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

Data Protection Diaries Fundamentals Walking The Data Protection Talk

Data Protection Diaries Walking The Data Protection Talk

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulz – www.storageioblog.com November 26, 2017

This is Part 8 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends, and click here to view the next post who’s Doing What ( Toolbox Technology Tools).

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around what I (and Server StorageIO) does for Data Protection besides just talking the talk and is a work in progress that is being updated over time with additional insights.

Walking The Data Protection Talk What I Do

A couple of years back I did the first post as part of the Data Protection Diaries series ( view here), that included the following image showing some data protection needs and requirements, as well as what being done, along with areas for improvement. Part of what I and Server StorageIO does involves consulting (strategy, design, assessment), advising and other influencers activities (e.g. blog, write articles, create reports, webinars, seminars, videos, podcasts) pertaining to data Infrastructure topics as well as data protection.

What this means is knowing about the trends, tools, technologies, what’s old and new, who’s doing what, what should be in the data protection toolbox, as well as how to use those for different scenarios. Its one thing to talk the talk, however I also prefer to walk the talk including eating my own dog food applying various techniques, approaches, tools and technologies discussed.

The following are from a previous Data Protection Diaries post where I discuss my data protection needs (and wants) some of which have evolved since then. Note the image on the left is my Livescribe Echo digital pen and paper tablet. On the right is an example of the digital image created and imported into my computer from the Livescribe. In other words, Im able to protect my hand written notes, diagrams and figures.

Data Protection Diaries Data Protection Diaries Walking The Talk
Via my Livescribe Echo digital pen ( get your Livescribe here at Amazon.com)

My Environment and data protection is always evolving, some based on changing projects, others that are more stable. Likewise the applications along with data are varied after all, everything is not the same. My data protection includes snapshots, replication, mirror, sync, versions, backup, archive, RAID, erasure code among others technologies, tools, and techniques.

Applications range from desktop, office, email, documents, spreadsheets, presentations, video, audio and related items in support of day-to-day activities. Then there are items part of various projects that range from physical to virtual, cloud and container leveraging various tools. This means having protection copies (sync, backup, snapshots, consistency points) of virtual machines, physical machine instances, applications and databases such as SQL Server among many others. Other application workloads include web, word press blog and email among others.

The Server StorageIO environment consists of a mix of legacy on-premises technologies from servers, storage, hardware, software, networks, tools as well as software defined virtual (e.g. VMware, Hyper-V, Docker among others), as well as cloud. The StorageIO data Infrastructure environment consists of dedicated private server (DPS) that I have had for several years now that supports this blog as well as other sites and activity. I also have a passive standby site used for testing of the WordPress based blog on an AWS Lightsail server. I use tools such as Updraft Plus Premium to routinely create a complete data protection view (database, plugins, templates, settings, configuration, core) of my WordPress site (runs on DPS) that is stored in various locations, including at AWS.

Data Protection Diaries Walking The Talk
Some of my past data protection requirements (they have evolved)

Currently the Lightsail Virtual Private Server (VPS) is in passive mode, however plans are to enable it as a warm or active standby fail over site for some of the DPS functions. One of the tools I have for monitoring and insight besides those in WordPress and the DPS are AWS Route 53 alerts that I have set up to monitor endpoints. AWS Route 53 is a handy resource for monitoring your endpoints such as a website, blog among other things and have it notify you, or take action including facilitating DNS fail over if needed. For now, Im simply using Route 53 besides as a secondary DNS as a notification tool.

Speaking of AWS, I have compute instances in Elastic Cloud Compute (EC2) along with associated Elastic Block Storage (EBS) volumes as well as their snapshots. I also have AWS S3 buckets in different regions that are on various tiers from standard to infrequent access (IA), as well as some data on Glacier. Data from my DPS at Bluehost gets protected to a AWS S3 bucket that I can access from AWS EC2, as well as via other locations including Microsoft Azure as needed.

Some on-premises data also gets protected to AWS S3 (as well as to elsewhere) using various tools, for different granularity, frequency, access and retention. After all, everything is not the same, why treat it the same. Some of the data protected to AWS S3 buckets is in native format (e.g. they appear as objects to S3 or object enabled applications), as well as file to file based applications with appropriate tools.

Other data that is also protected to AWS S3 from different data protection or backup tools are stored in vendor neutral or vendor specific save set, zip, tar ball or other formats. In other words, I need the tool or compatible tool that knows the format of the saved data to retrieve individual data files, items or objects. Note that this is similar to storing data on tape, HDDs, SSD or other media in native format vs. in some type of encapsulate save set or other format.

In addition to protecting data to AWS, I also have data at Microsoft Azure among other locations. Other locations include non-cloud based off-site where encrypted removable media is periodically taken to a safe secure place as a master, gold in case of major emergency, ransomeware copy.

Why not just rely on cloud copies?

Simple, I can pull individual files or relatively small amounts of data back from the cloud sometimes faster (or easier) than from on-site copies, let alone my off-site, off-line, air gap copies. On the other hand, if I need to restore large amounts of data, without a fast network, it can be quicker to get the air gap off-line, off-site copy, do the large restore, then apply incremental or changed data via cloud. In other a hybrid approach.

Now a common question I get is why not just do one or the other and save some money. Good point, I would save some money, however by doing the above among other things, they are part of being able to test, try new and different things, gain insight, experience not to mention walk the talk vs. simply talking the talk.

Of course Im always looking for ways to streamline to make my data protection more efficient, as well as effective (along with remove complexity and costs).

  • Everything is not the same, so why treat it all the same with common SLO, RTO, RPO and retention?
  • Likewise why treat and store all data the same way, on the same tiers of technology
  • Gain insight and awareness into environment, applications, workloads, PACE needs
  • Applications, data, systems or devices are protected with different granularity and frequency
  • Apply applicable technology and tools to the task at hand
  • Any data I have in cloud has a copy elsewhere, likewise, any data on-premises has a copy in the cloud or elsewhere
  • I implement the 4 3 2 1 rule by having multiple copies, versions, data in different locations, on and off-line including cloud
  • From a security standpoint, many different things are implemented on a logical as well as physical basis including encryption
  • Ability to restore data as well as applications or image instances locally as well as into cloud environments
  • Leverage different insight and awareness, reporting, analytics and monitoring tools
  • Mix of local storage configured with different RAID and other protection
  • Test, find, fix, remediate improve the environment including leveraging lessons learned

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

Everything is not the same, thats why in my environment I use different technologies, tools and techniques to protect my data. This also means having different RTO, RPO across various applications, data and systems as well as devices. Data that is more important has more copies, versions in different locations as well as occurring more frequently as part of 4 3 2 1 data protection. Other data that does not change as frequently, or time sensitive have alternate RTO and RPO along with corresponding frequency of protection.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series Part 9 who’s Doing What (Toolbox Technology Tools).

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

Data Protection Diaries Fundamentals Who Is Doing What Toolbox Technology Tools

Data Protection Toolbox Whos Doing What Technology Tools

Updated 1/17/2018

Data protection toolbox is a companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulz – www.storageioblog.com November 26, 2017

This is Part 9 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 8 Walking The Data Protection Talk, and click here to view the next post Part 10 Data Protection Resources Where to Learn More.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection who’s Doing What ( Toolbox Technology Tools).

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

who’s Doing What (Toolbox Technology Tools)

SDDC SDDI data center data protection toolbox
Data Protection Toolbox

Note that this post is evolving with additional tools, technologies, techniques, hardware, software, services being added over time along with applicable industry links.

The following are a sampling of some hardware, software, solution and component vendors along with service providers involved with data protection from RAID, Erasure Codes (EC) to snapshots, backup, BC, BR, DR, archive, security, cloud, bulk object storage, HDDs, SSD, tape among others including buzzword (and buzz term trends) bingo. Acronis, Actifio, Arcserve, ATTO, AWS, Backblaze, Barracuda, Broadcom, Caringo, Chelsio (offload), Code42/Crashplan, Cray, Ceph, Cisco, Cloudian, Cohesity, Compuverde, Commvault, Datadog, Datrium, Datos IO, DDN, Dell EMC, Druva, E8, Elastifile, Exagrid, Excelero, Fujifilm, Fujutsu, Google, HPE, Huawei, Hedvig, IBM, Intel, Iomega, Iron Mountain, IBM, Jungledisk, Kinetic key value drives (Seagate), Lenovo, LTO organization, Mangstor, Maxta, Mellanox (offload), Micron, Microsoft (Azure, Windows, Storage Spaces), Microsemi, Nakivo, NetApp, NooBaa, Nexsan, Nutanix, OpenIO, OpenStack (Swift), Oracle, Panasas, Panzura, Promise, Pure, Quantum, Quest, Qumulo, Retrospect, Riverbed, Rozo, Rubrik, Samsung, Scale, Scality, Seagate (DotHill), Sony, Solarwinds, Spectralogic, Starwind, Storpool, Strongbox, Sureline, Swiftstack, Synology, Toshiba, Tintri, Turbonomics, Unitrends, Unix and Linux platforms, Vantara, Veeam, VMware, Western Digital (Amplidata, Tegile and others), WekaIO, X-IO, Zadara and Zmanda among many others.

Note if you dont see yours, or your favorite, preferred or clients listed above or in the data Infrastructure industry related links send us a note for consideration to be included in future updates, or having a link, or sponsor spot pointing to your site added. Feel free to add a non sales marketing pitch to courteous comments to the comment section below.

View additional IT, data center and data Infrastructure along with data protection related vendors, services, tools, technologies links here.

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

Part of modernizing data protection for various data center and data infrastructure environments is to know the tools, technologies and trends that are part of your data protection toolbox. The other part of modernizing data is protection is knowing the techniques of how to use different tools, technologies to meet various application workload performance, availability, capacity economic (PACE) needs.

Also keep in mind that information services requires applications (e.g. programs) and that programs are a combination of algorithms (code, rules, policies) and data structures (e.g. data and how it is organized including unstructured). What this means is that data protection needs to address not only data, also the applications, configuration settings, metadata as well as protecting the protection tools and its data.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 10 Data Protection Fundamental Resources Where to Learn More.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

Data Protection Diaries Fundamental Resources Where to Learn More

Data Protection Diaries Fundamental Resources Where to Learn More

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulz – www.storageioblog.com November 26, 2017

This is the last in a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Click here to view the previous post Part 9 – who’s Doing What ( Toolbox Technology Tools).

Software Defined Data Infrastructure Essentials Book SDDC

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection Resources Where to Learn More.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

Software Defined Data Infrastructure Essentials Table of Contents (TOC)

Here is a link (PDF) to the table of contents (TOC) for Software Defined Data Infrastructure Essentials.

The following is a Software Defined Data Infrastructure Essentials book TOC summary:

Chapter 1: Server Storage I/O and Data Infrastructure Fundamentals
Chapter 2: Application and IT Environments
Chapter 3: Bits, Bytes, Blobs, and Software-Defined Building Blocks
Chapter 4: Servers: Physical, Virtual, Cloud, and Containers
Chapter 5: Server I/O and Networking
Chapter 6: Servers and Storage-Defined Networking
Chapter 7: Storage Mediums and Component Devices
Chapter 8: Data Infrastructure Services: Access and Performance
Chapter 9: Data Infrastructure Services: Availability, RAS, and RAID
Chapter 10: Data Infrastructure Services: Availability, Recovery-Point Objective, and Security
Chapter 11: Data Infrastructure Services: Capacity and Data Reduction
Chapter 12: Storage Systems and Solutions (Products and Cloud)
Chapter 13: Data Infrastructure and Software-Defined Management
Chapter 14: Data Infrastructure Deployment Considerations
Chapter 15: Software-Defined Data Infrastructure Futures, Wrap-up, and Summary
Appendix A: Learning Experiences
Appendix B: Additional Learning, Tools, and tradecraft Tricks
Appendix C: Frequently Asked Questions
Appendix D: Book Shelf and Recommended Reading
Appendix E: Tools and Technologies Used in Support of This Book
Appendix F: How to Use This Book for Various Audiences
Appendix G: Companion Website and Where to Learn More
Glossary
Index

Click here to view (PDF) table of contents (TOC).

Data Protection Resources Where To Learn More

Learn more about Data Infrastructure and Data Protection related technology, trends, tools, techniques, tradecraft and tips with the following links.

The following are the various posts that are part of this data protection series:

  • Part 1 – Data Infrastructure Data Protection Fundamentals
  • Part 2 – Reliability, Availability, Serviceability ( RAS) Data Protection Fundamentals
  • Part 3 – Data Protection Access Availability RAID Erasure Codes ( EC) including LRC
  • Part 4 – Data Protection Recovery Points (Archive, Backup, Snapshots, Versions)
  • Part 5 – Point In Time Data Protection Granularity Points of Interest
  • Part 6 – Data Protection Security Logical Physical Software Defined
  • Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends
  • Part 8 – Data Protection Diaries Walking Data Protection Talk
  • Part 9 – who’s Doing What ( Toolbox Technology Tools)
  • Part 10 – Data Protection Resources Where to Learn More


    • The following are various data protection blog posts:

  • Welcome to the Data Protection Diaries
  • Until the focus expands to data protection, backup is staying alive!
  • The blame game, Does cloud storage result in data loss?
  • Loss of data access vs. data loss
  • Revisiting RAID storage remains relevant and resources
  • Only you can prevent cloud (or other) data loss
  • Data protection is a shared responsibility
  • Time for CDP (Commonsense Data Protection)?
  • Data Infrastructure Server Storage I/O Tradecraft Trends (skills, experiences, knowledge)
  • My copies were corrupted: The [4] 3-2-1 rule and more about 4 3 2 1 as well as 3 2 1 here and here

    • The following are various data protection tips and articles:

  • Via Infostor Cloud Storage Concerns, Considerations and Trends
  • Via Network World What’s a data infrastructure?
  • Via Infostor Data Protection Gaps, Some Good, Some Not So Good
  • Via Infostor Object Storage is in your future
  • Via Iron Mountain Preventing Unexpected Disasters
  • Via InfoStor – The Many Variations of RAID Storage
  • Via InfoStor – RAID Remains Relevant, Really!
  • Via WservNews Cloud Storage Considerations (Microsoft Azure)
  • Via ComputerWeekly Time to restore from backup: Do you know where your data is?
  • Via Network World Ensure your data infrastructure remains available and resilient

    • The following are various data protection related webinars and events:

  • BrightTalk Webinar Data Protection Modernization – Protect, Preserve and Serve you Information
  • BrightTalk Webinar BCDR and Cloud Backup Protect Preserve and Secure Your Data Infrastructure
  • TechAdvisor Webinar (Free with registration) All You Need To Know about ROBO data protection
  • TechAdvisor Webinar (Free with registration) Tips for Moving from Backup to Full Disaster Recovery

    • The following are various data protection tools, technologies, services, vendor and industry resource links:

  • Various Data Infrastructure related news commentary, events, tips and articles
  • Data Center and Data Infrastructure industry links (vendors, services, tools, technologies, hardware, software)
  • Data Infrastructure server storage I/O network Recommended Reading List Book Shelf
  • Software Defined Data Infrastructure Essentials (CRC 2017) Book

    • Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    What This All Means

    Everything is not the same across environments, data centers, data infrastructures including SDDC, SDX and SDDI as well as applications along with their data.

    Likewise everything is and does not have to be the same when it comes to Data Protection.

    Since everything is not the same, various data protection approaches are needed to address various application performance, availability, capacity economic (PACE) needs, as well as SLO and SLAs.

    Data protection encompasses many different hardware, software, services including cloud technologies, tools, techniques, best practices, policies and tradecraft experience skills (e.g. knowing what to use when, where, why and how).

    Software Defined Data Infrastructure Essentials Book SDDC

    Context is important as different terms have various meanings depending on what they are being discussed with. Likewise different technologies and topics such as object, blob, backup, replication, RAID, erasure code (EC), mirroring, gaps (good, bad, ugly), snapshot, checkpoint, availability, durability among others have various meanings depending on context, as well as implementation approach.

    In most cases there is no bad technology or tool, granted there are some poor or bad (even ugly) implementations, as well as deployment or configuration decisions. What this means is the best technology or approach for your needs may be different from somebody else’s and vice versa.

    Some other points include there is no such thing as an information recession with more data generated every day, granted, how that data is transformed or stored can be in a smaller footprint. Likewise there is an increase in the size of data including unstructured big data, as well as the volume (how much data), as well as velocity (speed at which it is created, moved, processed, stored). This also means there is an increased dependency on data being available, accessible and intact with consistency. Thus the fundamental role of data Infrastructures (e.g. what’s inside the data center or cloud) is to combine resources, technologies, tools, techniques, best practices, policies, people skill set, experiences (e.g. tradecraft) to protect, preserve, secure and serve information (applications and data).

    modernizing data protection including backup, availability and related topics means more than swapping out one hardware, software, service or cloud for whatever is new, and then using it in old ways.

    What this means is to start using new (and old) things in new ways, for example move beyond using SSD or HDDs like tape as targets for backup or other data protection approaches. Instead use SSD, HDDs or cloud as a tier, yet also to enable faster protection and recovery by stepping back and rethinking what to protect, when, where, why, how and apply applicable techniques, tools and technologies. Find a balance between knowing all about the tools and trends while not understanding how to use those toolbox items, as well as knowing all about the techniques of how to use the tools, yet not knowing what the tools are.

    Want to learn more, have questions about specific tools, technologies, trends, vendors, products, services or techniques discussed in this series, send a note (info at storageio dot com) or via our contact page. We can set up a time to discuss your questions or needs pertaining to Data Protection as well as data infrastructures related topics from legacy to software defined virtual, cloud, container among others. For example consulting, advisory services, architecture strategy design, technology selection and acquisition coaching, education knowledge transfer sessions, seminars, webinars, special projects, test drive lab reviews or audits, content generation, videos, podcasts, custom content, chapter excerpts, demand generation among many other things.

    Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Microsoft Azure September 2017 Software Defined Data Infrastructure Updates

    Microsoft Azure September 2017 Software Defined Data Infrastructure Updates

    server storage I/O data infrastructure trends

    Microsoft and Azure September 2017 Software Defined Data infrastructure Updates

    September was a busy month for data infrastructure topics as well as Microsoft in terms of new and enhanced technologies. Wrapping up September was Microsoft Ignite where Azure, Azure Stack, Windows, O365, AI, IoT, development tools announcements occurred, along with others from earlier in the month. As part of the September announcements, Microsoft released a new version of Windows server (e.g. 1709) that has a focus for enhanced container support. Note that if you have deployed Storage Spaces Direct (S2D) and are looking to upgrade to 1709, do your homework as there are some caveats that will cause you to wait for the next release. Note that there had been new storage related enhancements slated for the September update, however those were announced at Ignite to being pushed to the next semi-annual release. Learn more here and also here.

    Azure Files and NFS

    Microsoft made several Azure file storage related announcements and public previews during September including Native NFS based file sharing as companion to existing Azure Files, along with public preview of new Azure File Sync Service. Native NFS based file sharing (public preview announced, service is slated to be available in 2018) is a software defined storage deployment of NetApp OnTAP running on top of Azure data infrastructure including virtual machines and leverage Azure underlying storage.

    Note that the new native NFS is in addition to the earlier native Azure Files accessed via HTTP REST and SMB3 enabling sharing of files inside Azure public cloud, as well as accessible externally from Windows based and Linux platforms including on premises. Learn more about Azure Storage and Azure Files here.

    Azure File Sync (AFS)

    Azure File Sync AFS

    Azure File Sync (AFS) has now entered public preview. While users of Windows-based systems have been able to access and share Azure Files in the past, AFS is something different. I have used AFS for some time now during several private preview iterations having seen how it has evolved, along with how Microsoft listens incorporating feedback into the solution.

    Lets take a look at what is AFS, what it does, how it works, where and when to use it among other considerations. With AFS, different and independent systems can now synchronize file shares through Azure. Currently in the AFS preview Windows Server 2012 and 2016 are supported including bare metal, virtual, and cloud based. For example I have had bare metal, virtual (VMware), cloud (Azure and AWS) as part of participating in a file sync activities using AFS.

    Not to be confused with some other storage related AFS including Andrew File System among others, the new Microsoft Azure File Sync service enables files to be synchronized across different servers via Azure. This is different then the previous available Azure File Share service that enables files stored in Azure cloud storage to be accessed via Windows and Linux systems within Azure, as well as natively by Windows platforms outside of Azure. Likewise this is different from the recently announced Microsoft Azure native NFS file sharing serving service in partnership with NetApp (e.g. powered by OnTAP cloud).

    AFS can be used to synchronize across different on premise as well as cloud servers that can also function as cache. What this means is that for Windows work folders served via different on premise servers, those files can be synchronized across Azure to other locations. Besides providing a cache, cloud tiering and enterprise file sync share (EFSS) capabilities, AFS also has robust optimization for data movement to and from the cloud and across sites, along with management tools. Management tools including diagnostics, performance and activity monitoring among others.

    Check out the AFS preview including planning for an Azure File Sync (preview) deployment (Docs Microsoft), and for those who have Yammer accounts, here is the AFS preview group link.

    Microsoft Azure Blob Events via Microsoft

    Azure Blob Storage Tiering and Event Triggers

    Two other Azure storage features that are in public preview include blob tiering (for cold archiving) and event triggers for events. As their names imply, blob tiering enables automatic migration from active to cold inactive storage of dormant date. Event triggers are policies rules (code) that get executed when a blob is stored to do various functions or tasks. Here is an overview of blob events and a quick start from Microsoft here.

    Keep in mind that not all blob and object storage are the same, a good example is Microsoft Azure that has page, block and append blobs. Append blobs are similar to what you might be familiar with other services objects. Here is a Microsoft overview of various Azure blobs including what to use when.

    Project Honolulu and Windows Server Enhancements

    Microsoft has evolved from command prompt (e.g. early MSDOS) to GUI with Windows to command line extending into PowerShell that left some thinking there is no longer need for GUI. Even though Microsoft has extended its CLI with PowerShell spanning WIndows platforms and Azure, along with adding Linux command shell, there are those who still want or need a GUI. Project Honolulu is the effort to bring GUI based management back to Windows in a simplified way for what had been headless, and desktop less deployments (e.g. Nano, Server Core). Microsoft had Server Management Tools (SMT) accessible via the Azure Portal which has been discontinued.


    Project Honolulu Image via Microsoft.com

    This is where project Honolulu comes into play for managing Windows Server platforms. What this means is that for those who dont want to rely on or have a PowerShell dependency have an alternative option. Learn more about Project Honolulu here and here, including download the public preview here.

    Storage Spaces Direct (S2D) Kepler Appliance

    Data Infrastructure provider DataOn has announced a new turnkey Windows Server 2016 Storage Spaces Direct (S2D) powered Hyper-Converged Infrastructure (e.g. productization of project Kepler-47) solution with two node small form factor servers (partner with MSI). How small? Think suitcase or airplane roller board carry on luggage size.

    What this means is that you can get into the converged, hyper-converged software defined storage game with Windows-based servers supporting Hyper-V virtual machines (Windows and Linux) including hardware for around $10,000 USD (varies by configuration and other options).

    Azure and Microsoft Networking News

    Speaking of Microsoft Azure public cloud, ever wonder what the network that enables the service looks like and some of the software defined networking (SDN) along with network virtualization function (NFV) objectives are, have a look at this piece from over at Data Center Knowledge.

    In related Windows, Azure and other focus areas, Microsoft, Facebook and Telxius have completed the installation of a high-capacity subsea cable (network) to cross the atlantic ocean. Whats so interesting from a data infrastructure, cloud or legacy server storage I/O and data center focus perspective? The new network was built by the combined companies vs. in the past by a Telco provider consortium with the subsequent bandwidth sold or leased to others.

    This new network is also 4,000 miles long including in depths of 11,000 feet, supports with current optics 160 terabits (e.g. 20 TeraBytes) per second capable of supporting 71 million HD videos streamed simultaneous. To put things into perspective, some residential Fiber Optic services can operate best case up to 1 gigabit per second (line speed) and in an asymmetrical fashion (faster download than uploads). Granted there are some 10 Gbit based services out there more common with commercial than residential. Simply put, there is a large amount of bandwidth increased across the atlantic for Microsoft and Facebook to support growing demands.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    What This All Means

    Microsoft announced a new release of Windows Server at Ignite as part of its new semi-annual release cycle. This latest version of Windows server is optimized for containers. In addition to Windows server enhancements, Microsoft continues to extend Azure and related technologies for public, private and hybrid cloud as well as software defined data infrastructures.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Posted on

    August 2017 Server StorageIO Data Infrastructure Update Newsletter



    Server StorageIO August 2017 Data Infrastructure Update Newsletter

    Volume 17, Issue VII (Pre VMworld 2017)

    Hello and welcome to the August 2017 issue of the Server StorageIO update newsletter.

    Its end of summer season here in north america which means wrapping up holidays, vacations, back to school shopping (and going to school), as well as the start of the fall IT technology conference season. VMworld 2017 USA is this week in Las Vegas and there will be several announcements coming out of that event. Given all of the activity so far this month, I’m going to cover the VMworld and related topics in a special early September issue of this newsletter.

    Speaking of VMworld 2017, if you are going to be there in Las Vegas, stop by the book store located in the community village area on Tuesday at 1PM I will be doing a book signing, meet and greet, stop by and say hello.

    Thanks to all who participated in the recent thevPad top 100 vBloggers event, I am honored to have StorageIOblog listed in the top 100 vBlogs. Also congratulations to new and returning fellow Microsoft MVPs and VMware vExperts. There is a lot going on in the industry, lets get to it in this Server StorageIO Data Infrastructure Update Newsletter.

    In This Issue

    Enjoy this edition of the Server StorageIO update newsletter (pre VMworld edition).

    Cheers GS

    Data Infrastructure and IT Industry Activity Trends

    Acronis announced True Image 2018 for home based data protection (backup), while Crashplan aka code42 announced they were getting out of the consumer, small office home office (SOHO) backup and data protection space to focus on the enterprise.

    Cisco bought software defined storage converged infrastructure software vendor Springpath for about $320M USD. Cisco and Swiftstack (object storage software) also announced interoperability news with the UCS S32600 storage server platform.

    GPU vendor NVIDIA announced Quadro Virtual Data Center workstation technology.

    Meanwhile ioFABRIC announced their new Vicinity 3.0 software defined management solution.

    Microsemi (remember PMC Sierra) announced release of its Flashtec PCIe controllers to help speed adoption deployment of SSDs including NVMe based.

    Microsoft bought Cycle Computing to enhance Azure services, while also making Azure Blob storage tiering available as part of an ongoing public preview. For those not aware, Azure Blob is similar to what other services call objects. Get in on the public preview here. For those who live in a hybrid world where your environment and experience include both Windows and Linux, check out Windows Services for Linux here. With this service which can install onto an Windows 10 system along side Win32 (e.g. it co-exists, its not a virtual machine), you can choose from the Windows Store which Linux distro you want (e.g. Centos, Ubuntu, etc).

    Need to learn, refresh or simply gain a better understanding of Microsoft PowerShell for software defined management of Windows, Azure and other environments? Check out this great post from Microsoft Blogs.

    For those who work in a Windows or Azure environment, here are some useful icons for Powerpoint, Visio, PNG and SVG from Microsoft. With Microsoft Ignite coming up in September, watch for some interesting update enhancements to Windows Server from a server storage I/O perspective.

    NextPlatform.com has an interesting article on Exascale Timeline for Storage and I/O systems worth a read. Panzura global name space and scale out software defined storage management software announced mobile client file sharing. After dropping their own cloud business, Verizon is now a virtual network services partner with Amazon.

    Over at all flash array (AFA) SSD vendor Pure, revenues are growing closer to an annual $1B USD rate despite loss per share, Pure also announced a change in leadership with current CEO Scott Dietzen stepping aside for Charles Giancarlo to take the lead spot.

    VMware has been talking about the continued increase in customer adoption and deployment of VSAN now they are showing they eat their own dog food. Check out this post here from VMware that shows how many and what size VSAN clusters they are using for various internal operations. Also on the VMware storage front, learn more about enhancements for large and small file allocation blocks with vSphere VMFS6.

    With all of the pre and post VMworld related announcements, remember to check out the tools available over at the VMware flings site including vSphere HTML5 Web Client, HCIBench, vRealize Operations Export, VisualEsxtop, ESXi Embedded Host Client, VMware OS Optimization Tool and many others. Watch for VMworld coverage in the September newsletter along with posts at www.storageioblog.com

    Check out other industry news, comments, trends perspectives here.

    Server StorageIO Commentary in the news

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via EnterpriseStorageForum: Comments on Who Will Rule the Storage World?
    Via InfoGoto: Comments on Google Cloud Platform Gaining Data Storage Momentum
    Via InfoGoto: Comments on Singapore High Rise Data Centers
    Via InfoGoto: Comments on New Tape Storage Capacity

    View more Server, Storage and I/O trends and perspectives comments here

    Server StorageIOblog Posts

    Recent and popular Server StorageIOblog posts include:

    In Case You Missed It #ICYMI

    View other recent as well as past StorageIOblog posts here

    Server StorageIO Data Infrastructure Tips and Articles

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via NetworkWorld: Do you have an IT trade craft skills gap?

    View more Server, Storage and I/O trends and perspectives comments here

    Events and Activities

    Recent and upcoming event activities.

    Sep. 21, 2017 – MSP CMG – Minneapolis MN
    Sep. 20, 2017 – Redmond Data Protection and Backup – Webinar
    Sep. 14, 2017 – Fujifilm IT Executive Summit – Seattle WA
    Sep. 12, 2017 – SNIA Software Developers Conference (SDC) – Santa Clara CA
    Sep. 7, 2017 – WiPro – Planning Your Software Defined Journey – New York City
    August 29, 2017 – VMworld – Las Vegas

    See more webinars and activities on the Server StorageIO Events page here.

    Server StorageIO Industry Resources and Links

    Useful links and pages:
    Microsoft TechNet – Various Microsoft related from Azure to Docker to Windows
    storageio.com/links – Various industry links (over 1,000 with more to be added soon)
    objectstoragecenter.com – Cloud and object storage topics, tips and news items
    OpenStack.org – Various OpenStack related items
    storageio.com/protect – Various data protection items and topics
    thenvmeplace.com – Focus on NVMe trends and technologies
    thessdplace.com – NVM and Solid State Disk topics, tips and techniques
    storageio.com/converge – Various CI, HCI and related SDS topics
    storageio.com/performance – Various server, storage and I/O benchmark and tools
    VMware Technical Network – Various VMware related items

    Ok, nuff said, for now.

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Posted on

    Announcing Software Defined Data Infrastructure Essentials Book by Greg Schulz

    New SDDI Essentials Book by Greg Schulz of Server StorageIO

    Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft

    server storage I/O data infrastructure trends

    Update 1/21/2018
    Over the past several months I have posted, commenting, presenting and discussing more about Software Defined Data Infrastructure Essentials aka SDDI or SDDC and SDI. Now it is time to announce my new book (my 4th solo project), Software Defined Data Infrastructure Essentials Book (CRC Press). Software Defined Data Infrastructure Essentials is now generally available at various global venues in hardcopy, hardback print as well as various electronic versions including via Amazon and CRC Press among others. For those attending VMworld 2017 in Las Vegas, I will be doing a book signing, meet and greet at 1PM Tuesday August 29 in the VMworld book store, as well as presenting at various other fall industry events.

    Software Defined Data Infrastructure Essentials Book Announcement

    (Via Businesswire) Stillwater, Minnesota – August 23, 2017  – Server StorageIO, a leading independent IT industry advisory and consultancy firm, in conjunction with publisher CRC Press, a Taylor and Francis imprint, announced the release and general availability of “Software-Defined Data Infrastructure Essentials,” a new book by Greg Schulz, noted author and Server StorageIO founder.

    Software Defined Data Infrastructure Essentials

    The Software Defined Data Infrastructure Essentials book covers physical, cloud, converged (and hyper-converged), container, and virtual server storage I/O networking technologies, revealing trends, tools, techniques, and tradecraft skills.

    Data Infrastructures Protect Preserve Secure and Serve Information
    Various IT and Cloud Infrastructure Layers including Data Infrastructures

    From cloud web scale to enterprise and small environments, IoT to database, software-defined data center (SDDC) to converged and container servers, flash solid state devices (SSD) to storage and I/O networking,, the book helps develop or refine hardware, software, services and management experiences, providing real-world examples for those involved with or looking to expand their data infrastructure education knowledge and tradecraft skills.

    Software Defined Data Infrastructure Essentials book topics include:

      • Cloud, Converged, Container, and Virtual Server Storage I/O networking
      • Data protection (archive, availability, backup, BC/DR, snapshot, security)
      • Block, file, object, structured, unstructured and data value
      • Analytics, monitoring, reporting, and management metrics
      • Industry trends, tools, techniques, decision making
      • Local, remote server, storage and network I/O troubleshooting
      • Performance, availability, capacity and  economics (PACE)

    Where To Purchase Your Copy

    Order via Amazon.com and CRC Press along with Google Books among other global venues.

    What People Are Saying About Software Defined Data Infrastructure Essentials Book

    “From CIOs to operations, sales to engineering, this book is a comprehensive reference, a must-read for IT infrastructure professionals, beginners to seasoned experts,” said Tom Becchetti, advisory systems engineer.

    “We had a front row seat watching Greg present live in our education workshop seminar sessions for ITC professionals in the Netherlands material that is in this book. We recommend this amazing book to expand your converged and data infrastructure knowledge from beginners to industry veterans.”

    Gert and Frank Brouwer – Brouwer Storage Consultancy

    “Software-Defined Data Infrastructures provides the foundational building blocks to improve your craft in several areas including applications, clouds, legacy, and more.  IT professionals, as well as sales professionals and support personal, stand to gain a great deal by reading this book.”

    Mark McSherry- Oracle Regional Sales Manager

    “Greg Schulz has provided a complete ‘toolkit’ for storage management along with the background and framework for the storage or data infrastructure professional (or those aspiring to become one).”
    Greg Brunton – Experienced Storage and Data Management Professional

    “Software-defined data infrastructures are where hardware, software, server, storage, I/O networking and related services converge inside data centers or clouds to protect, preserve, secure and serve applications and data,” said Schulz.  “Both readers who are new to data infrastructures and seasoned pros will find this indispensable for gaining and expanding their knowledge.”

    SDDI and SDDC components

    More About Software Defined Data Infrastructure Essentials
    Software Defined Data Infrastructures (SDDI) Essentials provides fundamental coverage of physical, cloud, converged, and virtual server storage I/O networking technologies, trends, tools, techniques, and tradecraft skills. From webscale, software-defined, containers, database, key-value store, cloud, and enterprise to small or medium-size business, the book is filled with techniques, and tips to help develop or refine your server storage I/O hardware, software, Software Defined Data Centers (SDDC), Software Data Infrastructures (SDI) or Software Defined Anything (SDx) and services skills. Whether you are new to data infrastructures or a seasoned pro, you will find this comprehensive reference indispensable for gaining as well as expanding experience with technologies, tools, techniques, and trends.

    Software Defined Data Infrastructure Essentials SDDI SDDC content

    This book is the definitive source providing comprehensive coverage about IT and cloud Data Infrastructures for experienced industry experts to beginners. Coverage of topics spans from higher level applications down to components (hardware, software, networks, and services) that get defined to create data infrastructures that support business, web, and other information services. This includes Servers, Storage, I/O Networks, Hardware, Software, Management Tools, Physical, Software Defined Virtual, Cloud, Docker, Containers (Docker and others) as well as Bulk, Block, File, Object, Cloud, Virtual and software defined storage.

    Additional topics include Data protection (Availability, Archiving, Resiliency, HA, BC, BR, DR, Backup), Performance and Capacity Planning, Converged Infrastructure (CI), Hyper-Converged, NVM and NVMe Flash SSD, Storage Class Memory (SCM), NVMe over Fabrics, Benchmarking (including metrics matter along with tools), Performance Capacity Planning and much more including whos doing what, how things work, what to use when, where, why along with current and emerging trends.

    Book Features

    ISBN-13: 978-1498738156
    ISBN-10: 149873815X
    Hardcover: 672 pages
    (Available in Kindle and other electronic formats)
    Over 200 illustrations and 70 plus tables
    Frequently asked Questions (and answers) along with many tips
    Various learning exercises, extensive glossary and appendices
    Publisher: Auerbach/CRC Press Publications; 1 edition (June 19, 2017)
    Language: English

    SDDI and SDDC toolbox

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Data Infrastructures Protect Preserve Secure and Serve Information
    Various IT and Cloud Infrastructure Layers including Data Infrastructures

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Data Infrastructures exist to protect, preserve, secure and serve information along with the applications and data they depend on. With more data being created at a faster rate, along with the size of data becoming larger, increased application functionality to transform data into information means more demands on data infrastructures and their underlying resources.

    Software-Defined Data Infrastructure Essentials: Cloud, Converged, and Virtual Fundamental Server Storage I/O Tradecraft is for people who are currently involved with or looking to expand their knowledge and tradecraft skills (experience) of data infrastructures. Software-defined data centers (SDDC), software data infrastructures (SDI), software-defined data infrastructure (SDDI) and traditional data infrastructures are made up of software, hardware, services, and best practices and tools spanning servers, I/O networking, and storage from physical to software-defined virtual, container, and clouds. The role of data infrastructures is to enable and support information technology (IT) and organizational information applications.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    Everything is not the same in business, organizations, IT, and in particular servers, storage, and I/O. This means that there are different audiences who will benefit from reading this book. Because everything and everybody is not the same when it comes to server and storage I/O along with associated IT environments and applications, different readers may want to focus on various sections or chapters of this book.

    If you are looking to expand your knowledge into an adjacent area or to understand whats under the hood, from converged, hyper-converged to traditional data infrastructures topics, this book is for you. For experienced storage, server, and networking professionals, this book connects the dots as well as provides coverage of virtualization, cloud, and other convergence themes and topics.

    This book is also for those who are new or need to learn more about data infrastructure, server, storage, I/O networking, hardware, software, and services. Another audience for this book is experienced IT professionals who are now responsible for or working with data infrastructure components, technologies, tools, and techniques.

    Learn more here about Software Defined Data Infrastructure (SDDI) Essentials book along with cloud, converged, and virtual fundamental server storage I/O tradecraft topics, order your copy from Amazon.com or CRC Press here, and thank you in advance for learning more about SDDI and related topics.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    July 2017 Server StorageIO Data Infrastructures Update Newsletter

    Volume 17, Issue VII

    Hello and welcome to the July 2017 issue of the Server StorageIO update newsletter.

    It has been busy time with a lot going on, so let’s get right to this months topics which include software defined, data infrastructures, server, I/O networking, storage and related topics.

    Software-Defined Data Infrastructure Essentials SDDI SDDC

    A quick update following up from the June newsletter is that my new book is now available via Amazon.com, CRC Press and other venues in hardcopy hardcover as well as electronic versions. Visit storageio.com/book4 to learn more including view table of contents, preface, how organized among other items.

    In This Issue

    Enjoy this edition of the Server StorageIO update newsletter.

    Cheers GS

    Data Infrastructure and IT Industry Activity Trends

    Some recent Industry Activities, Trends, News and Announcements include:

    Arcserve who gained their independence from CA a few years ago has now acquired Zetta to expand its data protection solution portfolio for legacy, software defined virtual and cloud data infrastructure environments.

    Software Defined Data Infrastructure (SDDI) solution provider Compuverde has been awarded a US Patent for optimal server selection in software defined data centers (SDDC).

    The InfiniBand Trade Association (IBTA) announced its April 2017 integrators list including RDMA over Converged Ethernet (RoCE) along with other recent plugfest updates. Learn more about IBTA and related topics at www.infinibandta.org

    Next Platform has an interesting piece about a slump in the High Performance Computing (HPC) also known as High Profit Compute (for financial firms among others) along with Super Compute (SC) industry sector. Some of the vendors in the HPC and SC space include AMD, Cisco, Cray, DDN, Dell EMC, Fujifilm, IBM, Intel, HPE/SGI, Mellanox, NetApp, Oracle, Seagate and Western Digital (WD) among others. Check out the piece over at Next Platform here.

    To beef up its storage solution offerings, Cray has done a deal with Seagate who is one of their suppliers of HDDs as well as software defined storage software (ClusterStore) and related systems. In this new deal, Cray will get (e.g. now own) Lustre based ClusterStore storage solution which besides the intellectual property (IP) including software, around 100 or so workers will shift from being Seagate to becoming Cray employees. Note that Seagate acquired ClusterStore solution as part of its acquisition a few years ago of Xyratex (enclosures and Lustre storage system software).

    Meanwhile Seagate CEO Stephen Luczo is steeping down from his top leadership position after posting poor earnings and more job cuts. We have heard that HDDs are dead for decades, yet they continue to be consumed, granted with ups and downs, lately more downs for Seagate than its competitor Western Digital (WD). Seagate has dabbled in flash solid state devices (SSDs) ranging from drives to cards including acquisition of the flash business from LSI/Avago a few years ago, along with some partnerships. As a component supplier, does Seagate need to take a bold move and get closer or do a deal with one of the nand flash or other persistent memory chip makers? Or, does Seagate need to continue to move up the stack with solutions building on their previous deals buying Xyratex (enclosure and ClusterStor), Evault (data protection software), DotHill (storage systems), LSI/Avago flash (flash cards)? Lets see what happens with Seagate both from a big picture as well as more focused product, technology, portfolio perspective.

    Watch for a flurry of NVMe, along with flash SSD and other non-volatile memory (NVM), persistent memories along with storage class memories over the next few weeks. Keep in mind that If Answer is NVMe, what are the questions? Also watch for a flurry of software defined, cloud, virtual and related news towards the end of the month and into September.

    Just for fun, Mike Acton (@mike_acton) has an interesting read on Introductory bullshit detection for non-technical managers that you can check out here. Over at Coding Horror there is another interesting read on the notion of paying down your technical debt, check it out here.

    Check out other industry news, comments, trends perspectives here.

    Server StorageIO Commentary in the news

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via EnterpriseStorageForum: Comments on Who Will Rule the Storage World?
    Via SearchDataCenter: Comments on Dell EMC Cisco VxBlock news

    View more Server, Storage and I/O trends and perspectives comments here

    Server StorageIO Featured White Paper Report

    This months featured Server StorageIO Data Infrastructure content is a new white paper industry trends perspective report looking at Weka.IO Matrix Multi-Dimensional Software Based Storage (e.g. software defined storage). In this report, we look at WekaIO Matrix, an elastic, flexible, highly scalable easy to use (and manage) software-defined (e.g. software-based) storage solution. WekaIO Matrix enables radically simple software defined storage that is flexible elastic scaling with stability and without compromise.

    Weka.io Matrix software defined storage

    • Matrix is a new storage solution that:
    • Installs on bare metal, virtual or cloud servers
    • Has POSIX, NFS, SMB, and HDFS storage access
    • Adaptable performance for little and big data
    • Tiering of flash SSD and cloud object storage
    • Distributed resilience without compromise
    • Removes complexity of traditional storage

    Read more in this StorageIO Industry Trends and Perspective (ITP) Report compliments of WekaIO by clicking here. Visit www.weka.io to learn more about WekaIO and their Matrix solution.

    View more Server StorageIO Data Infrastructure White Papers, Lab Reports, Solution Briefs, Research, Polls, Surveys and additional content here on our portfolio page.

     

    Server StorageIOblog Posts

    Recent and popular Server StorageIOblog posts include:

    • Intel Announces New Xeon Processors for Software Defined Data Infrastructures
      Intel announced a new family of Xeon Scalable Processors (aka Purely) that for some workloads Intel claims to be on average of 1.65x faster than their predecessors. Note your real improvement will vary based on workload, configuration, benchmark testing, type of processor, memory, and many other server storage I/O performance considerations.

    • Who Will Be At Top Of Storage World Next Decade?
      It is safe to say that each new year will bring new trends, techniques, technologies, tools, features, functionality as well as solutions involving data storage as well as data infrastructures. This means a usual safe bet is to say that the current year is the most exciting and has the most new things than in the past when it comes to data infrastructures along with resources such as data storage.

    • Zombie Technology Life after Death Tape Is Still Alive
      A Zombie Technology is one declared dead yet has Life after Death such as Tape which is still alive, despite having declared dead for decades.

    In Case You Missed It #ICYMI

    View other recent as well as past StorageIOblog posts here

    Server StorageIO Data Infrastructure Tips and Articles

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via IDG/NetworkWorld: Ensure your data infrastructure remains available and resilient
    Via IDG/NetworkWorld: Whats a data infrastructure?
    Via InfoStor: Object Storage Is In Your Future
    Via InfoStor: Cloud Storage Concerns, Considerations and Trends
    Via InfoStor: SSD Trends, Tips and Topics

    View more Server, Storage and I/O trends and perspectives comments here

    Events and Activities

    Recent and upcoming event activities.

    Sep. 21, 2017 – MSP CMG – Minneapolis MN

    Sep. 14, 2017 – Fujifilm IT Executive Summit – Seattle WA

    Sep. 12, 2017 – SNIA Software Developers Conference (SDC) – Santa Clara CA

    Sep. 7, 2017 – TBA – Enabling, Planning and Executing Your Software Defined Journey

    August 28-30, 2017 – VMworld – Las Vegas

    June 22, 2017 – Webinar – GDPR and Microsoft Environments

    May 11, 2017 – Webinar – Email Archiving, Compliance and Ransomware

    See more webinars and activities on the Server StorageIO Events page here.

    Server StorageIO Industry Resources and Links

    Useful links and pages:
    Microsoft TechNet – Various Microsoft related from Azure to Docker to Windows
    storageio.com/links – Various industry links (over 1,000 with more to be added soon)
    objectstoragecenter.com – Cloud and object storage topics, tips and news items
    OpenStack.org – Various OpenStack related items
    storageio.com/protect – Various data protection items and topics
    thenvmeplace.com – Focus on NVMe trends and technologies
    thessdplace.com – NVM and Solid State Disk topics, tips and techniques
    storageio.com/converge – Various CI, HCI and related SDS topics
    storageio.com/performance – Various server, storage and I/O benchmark and tools
    VMware Technical Network – Various VMware related items

    Ok, nuff said, for now.

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Posted on

    New family of Intel Xeon Scalable Processors enable software defined data infrastructures (SDDI) and SDDC

    Intel Xeon Scalable Processors SDDI and SDDC

    server storage I/O data infrastructure trends

    Today Intel announced a new family of Xeon Scalable Processors (aka Purely) that for some workloads Intel claims to be on average of 1.65x faster than their predecessors. Note your real improvement will vary based on workload, configuration, benchmark testing, type of processor, memory, and many other server storage I/O performance considerations.

    Intel Scalable Xeon Processors
    Image via Intel.com

    In general the new Intel Xeon Scalable Processors enable legacy and software defined data infrastructures (SDDI), along with software defined data centers (SDDC), cloud and other environments to support expanding workloads more efficiently as well as effectively (e.g. boosting productivity).

    Data Infrastructures and workloads

    Some target application and environment workloads Intel is positioning these new processors for includes among others:

    • Machine Learning (ML), Artificial Intelligence (AI), advanced analytics, deep learning and big data
    • Networking including software defined network (SDN) and network function virtualization (NFV)
    • Cloud and Virtualization including Azure Stack, Docker and Kubernetes containers, Hyper-V, KVM, OpenStack VMware vSphere, KVM among others
    • High Performance Compute (HPC) and High Productivity Compute (e.g. the other HPC)
    • Storage including legacy and emerging software defined storage software deployed as appliances, systems or server less deployment modes.

    Features of the new Intel Xeon Scalable Processors include:

    • New core micro architecture with interconnects and on die memory controllers
    • Sockets (processors) scalable up to 28 cores
    • Improved networking performance using Quick Assist and Data Plane Development Kit (DPDK)
    • Leverages Intel Quick Assist Technology for CPU offload of compute intensive functions including I/O networking, security, AI, ML, big data, analytics and storage functions. Functions that benefit from Quick Assist include cryptography, encryption, authentication, cipher operations, digital signatures, key exchange, loss less data compression and data footprint reduction along with data at rest encryption (DARE).
    • Optane Non-Volatile Dual Inline Memory Module (NVDIMM) for storage class memory (SCM) also referred to by some as Persistent Memory (PM), not to be confused with Physical Machine (PM).
    • Supports Advanced Vector Extensions 512  (AVX-512) for HPC and other workloads
    • Optional Omni-Path Fabrics in addition to 1/10Gb Ethernet among other I/O options
    • Six memory channels supporting up to 6TB of RDIMM with multi socket systems
    • From two to eight  sockets per node (system)
    • Systems support PCIe 3.x (some supporting x4 based M.2 interconnects)

    Note that exact speeds, feeds, slots and watts will vary by specific server model and vendor options. Also note that some server system solutions have two or more nodes (e.g. two or more real servers) in a single package not to be confused with two or more sockets per node (system or motherboard). Refer to the where to learn more section below for links to Intel benchmarks and other resources.

    Software Defined Data Infrastructures, SDDC, SDX and SDDI

    What About Speeds and Feeds

    Watch for and check out the various Intel partners who have or will be announcing their new server compute platforms based on Intel Xeon Scalable Processors. Each of the different vendors will have various speeds and feeds options that build on the fundamental Intel Xeon Scalable Processor capabilities.

    For example Dell EMC announced their 14G server platforms at the May 2017 Dell EMC World event with details to follow (e.g. after the Intel announcements).

    Some things to keep in mind include the amount of DDR4 DRAM (or Optane NVDIMM) will vary by vendors server platform configuration, motherboards, several sockets and DIMM slots. Also keep in mind the differences between registered (e.g. buffered RDIMM) that give good capacity and great performance, and load reduced DIMM (LRDIMM) that have great capacity and ok performance.

    Various nvme options

    What about NVMe

    It’s there as these systems like previous Intel models support NVMe devices via PCIe 3.x slots, and some vendor solutions also supporting M.2 x4 physical interconnects as well.

    server storageIO flash and SSD
    Image via Software Defined Data Infrastructure Essentials (CRC)

    Note that Broadcom formerly known as Avago and LSI recently announced PCIe based RAID and adapter cards that support NVMe attached devices in addition to SAS and SATA.

    server storage data infrastructure sddi

    What About Intel and Storage

    In case you have not connected the dots yet, the Intel Xeon Scalable Processor based server (aka compute) systems are also a fundamental platform for storage systems, services, solutions, appliances along with tin-wrapped software.

    What this means is that the Intel Xeon Scalable Processors based systems can be used for deploying legacy as well as new and emerging software-defined storage software solutions. This also means that the Intel platforms can be used to support SDDC, SDDI, SDX, SDI as well as other forms of legacy and software-defined data infrastructures along with cloud, virtual, container, server less among other modes of deployment.

    Image Via Intel.com

    Moving beyond server and compute platforms, there is another tie to storage as part of this recent as well as other Intel announcements. Just a few weeks ago Intel announced 64 layer triple level cell (TLC) 3D NAND solutions positioned for the client market (laptop, workstations, tablets, thin clients). Intel with that announcement increased the traditional aerial density (e.g. bits per square inch or cm) as well as boosting the number of layers (stacking more bits as well).

    The net result is not only more bits per square inch, also more per cubic inch or cm. This is all part of a continued evolution of NAND flash including from 2D to 3D, MCL to TLC, 32 to 64 layer.  In other words, NAND flash-based Solid State Devices (SSDs) are very much still a relevant and continue to be enhanced technology even with the emerging 3D XPoint and Optane (also available via Amazon in M.2) in the wings.

    server memory evolution
    Via Intel and Micron (3D XPoint launch)

    Keep in mind that NAND flash-based technologies were announced almost 20 years ago (1999), and are still evolving. 3D XPoint announced two years ago, along with other emerging storage class memories (SCM), non-volatile memory (NVM) and persistent memory (PM) devices are part of the future as is 3D NAND (among others). Speaking of 3D XPoint and Optane, Intel had announcements about that in the past as well.

    Where To Learn More

    Learn more about Intel Xeon Scalable Processors along with related technology, trends, tools, techniques and tips with the following links.

    What This All Means

    Some say the PC is dead and IMHO that depends on what you mean or define a PC as. For example if you refer to a PC generically to also include servers besides workstations or other devices, then they are alive. If however your view is that PCs are only workstations and client devices, then they are on the decline.

    However if your view is that a PC is defined by the underlying processor such as Intel general purpose 64 bit x86 derivative (or descendent) then they are very much alive. Just as older generations of PCs leveraging general purpose Intel based x86 (and its predecessors) processors were deployed for many uses, so to are today’s line of Xeon (among others) processors.

    Even with the increase of ARM, GPU and other specialized processors, as well as ASIC and FPGAs for offloads, the role of general purpose processors continues to increase, as does the technology evolution around. Even with so called server less architectures, they still need underlying compute server platforms for running software, which also includes software defined storage, software defined networks, SDDC, SDDI, SDX, IoT among others.

    Overall this is a good set of announcements by Intel and what we can also expect to be a flood of enhancements from their partners who will use the new family of Intel Xeon Scalable Processors in their products to enable software defined data infrastructures (SDDI) and SDDC.

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book "Software-Defined Data Infrastructure Essentials" (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Posted on

    AWS S3 Storage Gateway Revisited (Part I)

    server storage I/O trends

    AWS S3 Storage Gateway Revisited (Part I)

    This Amazon Web Service (AWS) Storage Gateway Revisited posts is a follow-up to the AWS Storage Gateway test drive and review I did a few years ago (thus why it’s called revisited). As part of a two-part series, the first post looks at what AWS Storage Gateway is, how it has improved since my last review of AWS Storage Gateway along with deployment options. The second post in the series looks at a sample test drive deployment and use.

    If you need an AWS primer and overview of various services such as Elastic Cloud Compute (EC2), Elastic Block Storage (EBS), Elastic File Service (EFS), Simple Storage Service (S3), Availability Zones (AZ), Regions and other items check this multi-part series (Cloud conversations: AWS EBS, Glacier and S3 overview (Part I) ).

    AWS

    As a quick refresher, S3 is the AWS bulk, high-capacity unstructured and object storage service along with its companion deep cold (e.g. inactive) Glacier. There are various S3 storage service classes including standard, reduced redundancy storage (RRS) along with infrequent access (IA) that have different availability durability, performance, service level and cost attributes.

    Note that S3 IA is not Glacier as your data always remains on-line accessible while Glacier data can be off-line. AWS S3 can be accessed via its API, as well as via HTTP rest calls, AWS tools along with those from third-party’s. Third party tools include NAS file access such as S3FS for Linux that I use for my Ubuntu systems to mount S3 buckets and use similar to other mount points. Other tools include Cloudberry, S3 Motion, S3 Browser as well as plug-ins available in most data protection (backup, snapshot, archive) software tools and storage systems today.

    AWS S3 Storage Gateway and What’s New

    The Storage Gateway is the AWS tool that you can use for accessing S3 buckets and objects via your block volume, NAS file or tape based applications. The Storage Gateway is intended to give S3 bucket and object access to on-premises applications and data infrastructures functions including data protection (backup/restore, business continuance (BC), business resiliency (BR), disaster recovery (DR) and archiving), along with storage tiering to cloud.

    Some of the things that have evolved with the S3 Storage Gateway include:

    • Easier, streamlined download, installation, deployment
    • Enhanced Virtual Tape Library (VTL) and Virtual Tape support
    • File serving and sharing (not to be confused with Elastic File Services (EFS))
    • Ability to define your own bucket and associated parameters
    • Bucket options including Infrequent Access (IA) or standard
    • Options for AWS EC2 hosted, or on-premises VMware as well as Hyper-V gateways (file only supports VMware and EC2)

    AWS Storage Gateway Three Functions

    AWS Storage Gateway can be deployed for three basic functions:

      AWS Storage Gateway File Architecture via AWS.com

    • File Gateway (NFS NAS) – Files, folders, objects and other items are stored in AWS S3 with a local cache for low latency access to most recently used data. With this option, you can create folders and subdirectory similar to a regular file system or NAS device as well as configure various security, permissions, access control policies. Data is stored in S3 buckets that you specify policies such as standard or Infrequent Access (IA) among other options. AWS hosted via EC2 as well as VMware Virtual Machine (VM) for on-premises file gateway.

      Also, note that AWS cautions on multiple concurrent writers to S3 buckets with Storage Gateway so check the AWS FAQs which may have changed by the time you read this. Current file share limits (subject to change) include 1 file gateway share per S3 bucket (e.g. a one to one mapping between file share and a bucket). There can be 10 file shares per gateway (e.g. multiple shares each with its own bucket per gateway) and a maximum file size of 5TB (same as maximum S3 object size). Note that you might hear about object storage systems supporting unlimited size objects which some may do, however generally there are some constraints either on their API front-end, or what is currently tested. View current AWS Storage Gateway resource and specification limits here.

      • AWS Storage Gateway Non-Cached Volume Architecture via AWS.com

      AWS Storage Gateway Cached Volume Architecture via AWS.com

    • Volume Gateway (Block iSCSI) – Leverages S3 with a point in time backup as an AWS EBS snapshot. Two options exist including Cached volumes with low-latency access to most recently used data (e.g. data is stored in AWS, with a local cache copy on disk or SSD). The other option is Stored Volumes (e.g. non-cached) where primary copy is local and periodic snapshot backups are sent to AWS. AWS provides EC2 hosted, as well as VMs for VMware and various Hyper-V Windows Server based VMs.

      Current Storage Gateway volume limits (subject to change) include maximum size of a cached volume 32TB, maximum size of a stored volume 16TB. Note that snapshots of cached volumes larger than 16TB can only be restored to a storage gateway volume, they can not be restored as an EBS volume (via EC2). There are a maximum of 32 volumes for a gateway with total size of all volumes for a gateway (cached) of 1,024TB (e.g. 1PB). The total size of all volumes for a gateway (stored volume) is 512TB. View current AWS Storage Gateway resource and specification limits here.

      • AWS Storage Gateway VTL Architecture via AWS.com

    • Virtual Tape Library Gateway (VTL) – Supports saving your data for backup/BC/DR/archiving into S3 and Glacier storage tiers. Being a Virtual Tape Library (e.g. VTL) you can specify emulation of tapes for compatibility with your existing backup, archiving and data protection software, management tools and processes.

      Storage Gateway limits for tape include minimum size of a virtual tape 100GB, maximum size of a virtual tape 2.5TB, maximum number of virtual tapes for a VTL is 1,500 and total size of all tapes in a VTL is 1PB. Note that the maximum number of virtual tapes in an archive is unlimited and total size of all tapes in an archive is also unlimited. View current AWS Storage Gateway resource and specification limits here.

      AWS

    Where To Learn More

    What This All Means

    As to which gateway function and mode (cached or non-cached for Volumes) depends on what it is that you are trying to do. Likewise choosing between EC2 (cloud hosted) or on-premises Hyper-V and VMware VMs depends on what your data infrastructure support requirements are. Overall I like the progress that AWS has put into evolving the Storage Gateway, granted it might not be applicable for all usage cases. Continue reading more and view images from the AWS Storage Gateway Revisited test drive in part two located here.

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Posted on

    Part II Revisting AWS S3 Storage Gateway (Test Drive Deployment)

    server storage I/O trends

    Part II Revisiting AWS S3 Storage Gateway (Test Drive Deployment)

    This Amazon Web Service (AWS) Storage Gateway Revisited posts is a follow-up to the AWS Storage Gateway test drive and review I did a few years ago (thus why it’s called revisited). As part of a two-part series, the first post looks at what AWS Storage Gateway is, how it has improved since my last review of AWS Storage Gateway along with deployment options. The second post in the series looks at a sample test drive deployment and use.

    What About Storage Gateway Costs?

    Costs vary by region, type of storage being used (files stored in S3, Volume Storage, EBS Snapshots, Virtual Tape storage, Virtual Tape storage archive), as well as type of gateway host, along with how access and used. Request pricing varies including data written to AWS storage by gateway (up to maximum of $125.00 per month), snapshot/volume delete, virtual tape delete, (prorate fee for deletes within 90 days of being archived), virtual tape archival, virtual tape retrieval. Note that there are also various data transfer fees that also vary by region and gateway host. Learn more about pricing here.

    What Are Some Storage Gateway Alternatives

    AWS and S3 storage gateway access alternatives include those from various third-party (including that are in the AWS marketplace), as well as via data protection tools (e.g. backup/restore, archive, snapshot, replication) and more commonly storage systems. Some tools include Cloudberry, S3FS, S3 motion, S3 Browser among many others.

    Tip is when a vendor says they support S3, ask them if that is for their back-end (e.g. they can access and store data in S3), or front-end (e.g. they can be accessed by applications that speak S3 API). Also explore what format the application, tool or storage system stores data in AWS storage, for example, are files mapped one to one to S3 objects along with corresponding directory hierarchy, or are they stored in a save set or other entity.

    AWS Storage Gateway Deployment and Management Tips

    Once you have created your AWS account (if you did not already have one) and logging into the AWS console (note the link defaults to US East 1 Region), go to the AWS Services Dashboard and select Storage Gateway (or click here which goes to US East 1). You will be presented with three options (File, Volume or VTL) modes.

    What Does Storage Gateway and Install Look Like

    The following is what installing a AWS Storage Gateway for file and then volume looks like. First, access the AWS Storage Gateway main landing page (it might change by time you read this) to get started. Scroll down and click on the Get Started with AWS Storage Gateway button or click here.

    AWS Storage Gateway Landing Page

    Select type of gateway to create, in the following example File is chosen.

    Select type of AWS storage gateway

    Next select the type of file gateway host (EC2 cloud hosted, or on-premises VMware). If you choose VMware, an OVA will be downloaded (follow the onscreen instructions) that you deploy on your ESXi system or with vCenter. Note that there is a different VMware VM gateway OAV for File Gateway and another for Volume Gateway. In the following example VMware ESXi OVA is selected and downloaded, then accessed via VMware tools such as vSphere Web Client for deployment.

    AWS Storage Gateway select download

    Once your VMware OVA file is downloaded from AWS, install using your preferred VMware tool, in this case I used the vSphere Web Client.

    AWS Storage Gateway VM deploy

    Once you have deployed the VMware VM for File Storage Gateway, it is time to connect to the gateway using the IP address assigned (static or DHCP) for the VM. Note that you may need to allocate some extra VMware storage to the VM if prompted (this mainly applies to Volume Gateway). Also follow directions about setting NTP time, using paravirtual adapters, thick vs. thin provisioning along with IP settings. Also double-check to make sure your VM and host are set for high-performance power setting. Note that the default username is sguser and password is sgpassword for the gateway.

    AWS Storage Gateway Connect

    Once you successfully connect to the gateway, next step will be to configure file share settings.

    AWS Storage Gateway Configure File Share

    Configure file share by selecting which gateway to use (in case you have more than one), name of an S3 bucket name to create, type of storage (S3 Standard or IA), along with Access Management security controls.

    AWS Storage Gateway Create Share

    Next step is to complete file share creation, not the commands provided for Linux and Windows for accessing the file share.

    AWS Storage Gateway Review Share Settings

    Review file share settings

    AWS Storage Gateway access from Windows

    Now lets use the file share by accessing and mounting to a Windows system, then copy some files to the file share.

    AWS Storage Gateway verify Bucket Items

    Now let’s go to the AWS console (or in our example use S3 Browser or your favorite tool) and look at the S3 bucket for the file share and see what is there. Note that each file is an object, and the objects simply appear as a file. If there were sub-directory those would also exist. Note that there are other buckets that I have masked out as we are only interested in the one named awsgwydemo that is configured using S3 Standard storage.

    AWS Storage Gateway Volume

    Now lets look at using the S3 Storage Gateway for Volumes. Similar to deploying for File Gateway, start out at the AWS Storage Gateway page and select Volume Gateway, then select what type of host (EC2 cloud, VMware or Hyper-V (2008 R2 or 2012) for on-premises deployment). Lets use the VMware Gateway, however as mentioned above, this is a different OVA/OVF than the File Gateway.

    AWS Storage Gateway Configure Volume

    Download the VMware OVA/OVF from AWS, and then install using your preferred VMware tools making sure to configure the gateway per instructions. Note that the Volume Gateway needs a couple of storage devices allocated to it. This means you will need to make sure that a SCSI adapter exists (or add one) to the VM, along with the disks (HDD or SSD) for local storage. Refer to AWS documentation about how to size, for my deployment I added a couple of small 80GB drives (you can choose to put on HDD or SSD including NVMe). Note that when connecting to the gateway if you get an error similar to below, make sure that you are in fact using the Volume Gateway and not mistakenly using the File Gateway OVA (VM). Note that the default username is sguser and password is sgpassword for the gateway.

    AWS Storage Gateway Connect To Volume

    Now connect to the local Volume Storage Gateway and notice the two local disks allocated to it.

    AWS Storage Gateway Cached Volume Deploy

    Next its time to create the Gateway which are deploying a Volume Cached below.

    AWS Storage Gateway Volume Create

    Next up is creating a volume, along with its security and access information.

    AWS Storage Gateway Volume Settings

    Volume configuration continued.

    AWS Storage Gateway Volume CHAP

    And now some additional configuration of the volume including iSCSI CHAP security.

    AWS Storage Gateway Windows Access

    Which leads us up to some Windows related volume access and configuration.

    AWS Storage Gateway Using iSCSI Volume

    Now lets use the new iSCSI based AWS Storage Gateway Volume. On the left you can see various WIndows command line activity, along with corresponding configuration information on the right.

    AWS Storage Gateway Being Used by Windows

    And there you have it, a quick tour of AWS Storage Gateway, granted there are more options that you can try yourself.

    AWS

    Where To Learn More

    What This All Means

    Overall I like the improvements that AWS has made to the Storage Gateway along with the different options it provides. Something to keep in mind is that if you are planning to use the AWS Storage Gateway File serving sharing mode that there are caveats to multiple concurrent writers to the same bucket. I would not be surprised if some other gateway or software based tool vendors tried to throw some fud towards the Storage Gateway, however ask them then how they coordinate multiple concurrent updates to a bucket while preserving data integrity.

    Which Storage Gateway variant from AWS to use (e.g. File, Volume, VTL) depends on what your needs are, same with where the gateway is placed (Cloud hosted or on-premises with VMware or Hyper-V). Keep an eye on your costs, and more than just the storage space capacity. This means pay attention to your access and requests fees, as well as different service levels, along with data transfer fees.

    You might wonder what about EFS and why you would want to use AWS Storage Gateway? Good question, at the time of this post EFS has evolved from being internal (e.g. within AWS and across regions) to having an external facing end-point however there is a catch. That catch which might have changed by time you read this is that the end-point can only be accessed from AWS Direct Connect locations.

    This means that if your servers are not in a AWS Direct Connect location, without some creative configuration, EFS is not an option. Thus Storage Gateway File mode might be an option in place of EFS as well as using AWS storage access tools from others. For example I have some of my S3 buckets mounted on Linux systems using S3FS for doing rsync or other operations from local to cloud. In addition to S3FS, I also have various backup tools that place data into S3 buckets for backup, BC and DR as well as archiving.

    Check out AWS Storage Gateway yourself and see what it can do or if it is a fit for your environment.

    Ok, nuff said (for now…).

    Cheers
    Gs

    Greg Schulz – Multi-year Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book "Software-Defined Data Infrastructure Essentials" (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Posted on

    May 2017 Server StorageIO Data Infrastructures Update Newsletter

    Volume 17, Issue V

    Hello and welcome to the May 2017 issue of the Server StorageIO update newsletter.

    Summer officially here in the northern hemisphere is still a few weeks away, however for all practical purposes it has arrived. What this means is that in addition to normal workplace activities and projects, there are plenty of outdoor things (as well as distractions) to attend to.

    Over the past several months I have mentioned a new book that is due out this summer and which means it’s getting close to announcement time. The new book title is Software Defined Data Infrastructure Essentials – Cloud, Converged, and Virtual Fundamental Server Storage I/O Tradecraft (CRC PRess/Taylor Francis/Auerbach) that you can learn more about here (with more details being added soon). A common question is will there be electronic versions of the book and the answer is yes (more on this in future newsletter).

    Data Infrastructures

    Another common question is what is it about, what is a data infrastructure (see this post) and what is tradecraft (see this post). Software-Defined Data Infrastructures Essentials provides fundamental coverage of physical, cloud, converged, and virtual server storage I/O networking technologies, trends, tools, techniques, and tradecraft skills.

    Software-Defined Data Infrastructures Essentials provides fundamental coverage of physical, cloud, converged, and virtual server storage I/O networking technologies, trends, tools, techniques, and tradecraft skills. From webscale, software-defined, containers, database, key-value store, cloud, and enterprise to small or medium-size business, the book is filled with techniques, and tips to help develop or refine your server storage I/O hardware, software, and services skills. Whether you are new to data infrastructures or a seasoned pro, you will find this comprehensive reference indispensable for gaining as well as expanding experience with technologies, tools, techniques, and trends.

    Software-Defined Data Infrastructure Essentials SDDI SDDC
    ISBN-13: 978-1498738156
    ISBN-10: 149873815X
    Hardcover: 672 pages
    Publisher: Auerbach Publications; 1 edition (June 2017)
    Language: English

    Watch for more news and insight about my new book Software-Defined Data Infrastructure Essentials soon. In the meantime, check out the various items below in this edition of the Server StorageIO Update.

    In This Issue

    Enjoy this edition of the Server StorageIO update newsletter.

    Cheers GS

    Data Infrastructure and IT Industry Activity Trends

    Some recent Industry Activities, Trends, News and Announcements include:

    Flackbox.com has some new independent (non NetApp produced) learning resources including NetApp simulator eBook and MetroCluster tutorial. Over in the Microsoft world, Thomas Maurer has a good piece about Windows Server build 2017 and all about containers. Microsoft also announced SQL Server 2017 CTP 2.1 is now available. Meanwhile here are some my experiences and thoughts from test driving Microsoft Azure Stack.

    Speaking of NetApp among other announcements they released a new version of their StorageGrid object storage software. NVMe activity in the industry (and at customer sites) continues to increase with Cavium Qlogic NVMe over Fabric news, along with Broadcom recent NVMe RAID announcements. Keep in mind that if the answer is NVMe, than what are the questions.

    Here is a good summary of the recent OpenStack Boston Summit. Storpool did a momentum announcement which for those of you into software defined storage, add Storpool to your watch list. On the VMware front, check out this vSAN 6.6 demo (video) of stretched cluster via Yellow Bricks.

    Check out other industry news, comments, trends perspectives here.

    Server StorageIOblog Posts

    Recent and popular Server StorageIOblog posts include:

    View other recent as well as past StorageIOblog posts here

    Server StorageIO Commentary in the news

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via EnterpriseStorageForum: What to Do with Legacy Assets in a Flash Storage World
    There is still a place for hybrid arrays. A hybrid array is the home run when it comes to leveraging your existing non-flash, non-SSD based assets today.

    Via EnterpriseStorageForum: Where All-Flash Storage Makes No Sense
    A bit of flash in the right place can go a long way, and everybody can benefit from at least a some of flash somewhere. Some might say the more, the better. But where you have budget constraints that simply prevent you from having more flash for things such as cold, inactive, or seldom access data, you should explore other options.

    Via Bitpipe: Changing With the Times – Protecting VMs(PDF)

    Via FedTech: Storage Strategies: Agencies Optimize Data Centers by Focusing on Storage

    Via SearchCloudStorage: Dell EMC cloud storage strategy needs to cut through fog

    Via SearchStorage: Microsemi upgrades controllers based on HPE technology

    Via EnterpriseStorageForum: 8 Data Machine Learning and AI Storage Tips

    Via SiliconAngle: Dell EMC announces hybrid cloud platform for Azure Stack

    View more Server, Storage and I/O trends and perspectives comments here

    Events and Activities

    Recent and upcoming event activities.

    Sep. 13-15, 2017 – Fujifilm IT Executive Summit – Seattle WA

    August 28-30, 2017 – VMworld – Las Vegas

    Jully 22, 2017 – TBA

    June 22, 2017 – Webinar – GDPR and Microsoft Environments

    May 11, 2017 – Webinar – Email Archiving, Compliance and Ransomware

    See more webinars and activities on the Server StorageIO Events page here.

    Server StorageIO Industry Resources and Links

    Useful links and pages:
    Microsoft TechNet – Various Microsoft related from Azure to Docker to Windows
    storageio.com/links – Various industry links (over 1,000 with more to be added soon)
    objectstoragecenter.com – Cloud and object storage topics, tips and news items
    OpenStack.org – Various OpenStack related items
    storageio.com/protect – Various data protection items and topics
    thenvmeplace.com – Focus on NVMe trends and technologies
    thessdplace.com – NVM and Solid State Disk topics, tips and techniques
    storageio.com/converge – Various CI, HCI and related SDS topics
    storageio.com/performance – Various server, storage and I/O benchmark and tools
    VMware Technical Network – Various VMware related items

    Ok, nuff said, for now.

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book “Software-Defined Data Infrastructure Essentials” (CRC Press).

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.