Data Protection Diaries Fundamentals Walking The Data Protection Talk

Data Protection Diaries Walking The Data Protection Talk

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is Part 8 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends, and click here to view the next post who’s Doing What ( Toolbox Technology Tools).

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around what I (and Server StorageIO) does for Data Protection besides just talking the talk and is a work in progress that is being updated over time with additional insights.

Walking The Data Protection Talk What I Do

A couple of years back I did the first post as part of the Data Protection Diaries series ( view here), that included the following image showing some data protection needs and requirements, as well as what being done, along with areas for improvement. Part of what I and Server StorageIO does involves consulting (strategy, design, assessment), advising and other influencers activities (e.g. blog, write articles, create reports, webinars, seminars, videos, podcasts) pertaining to data Infrastructure topics as well as data protection.

What this means is knowing about the trends, tools, technologies, what’s old and new, who’s doing what, what should be in the data protection toolbox, as well as how to use those for different scenarios. Its one thing to talk the talk, however I also prefer to walk the talk including eating my own dog food applying various techniques, approaches, tools and technologies discussed.

The following are from a previous Data Protection Diaries post where I discuss my data protection needs (and wants) some of which have evolved since then. Note the image on the left is my Livescribe Echo digital pen and paper tablet. On the right is an example of the digital image created and imported into my computer from the Livescribe. In other words, Im able to protect my hand written notes, diagrams and figures.

Data Protection Diaries Data Protection Diaries Walking The Talk
Via my Livescribe Echo digital pen ( get your Livescribe here at Amazon.com)

My Environment and data protection is always evolving, some based on changing projects, others that are more stable. Likewise the applications along with data are varied after all, everything is not the same. My data protection includes snapshots, replication, mirror, sync, versions, backup, archive, RAID, erasure code among others technologies, tools, and techniques.

Applications range from desktop, office, email, documents, spreadsheets, presentations, video, audio and related items in support of day-to-day activities. Then there are items part of various projects that range from physical to virtual, cloud and container leveraging various tools. This means having protection copies (sync, backup, snapshots, consistency points) of virtual machines, physical machine instances, applications and databases such as SQL Server among many others. Other application workloads include web, word press blog and email among others.

The Server StorageIO environment consists of a mix of legacy on-premises technologies from servers, storage, hardware, software, networks, tools as well as software defined virtual (e.g. VMware, Hyper-V, Docker among others), as well as cloud. The StorageIO data Infrastructure environment consists of dedicated private server (DPS) that I have had for several years now that supports this blog as well as other sites and activity. I also have a passive standby site used for testing of the WordPress based blog on an AWS Lightsail server. I use tools such as Updraft Plus Premium to routinely create a complete data protection view (database, plugins, templates, settings, configuration, core) of my WordPress site (runs on DPS) that is stored in various locations, including at AWS.

Data Protection Diaries Walking The Talk
Some of my past data protection requirements (they have evolved)

Currently the Lightsail Virtual Private Server (VPS) is in passive mode, however plans are to enable it as a warm or active standby fail over site for some of the DPS functions. One of the tools I have for monitoring and insight besides those in WordPress and the DPS are AWS Route 53 alerts that I have set up to monitor endpoints. AWS Route 53 is a handy resource for monitoring your endpoints such as a website, blog among other things and have it notify you, or take action including facilitating DNS fail over if needed. For now, Im simply using Route 53 besides as a secondary DNS as a notification tool.

Speaking of AWS, I have compute instances in Elastic Cloud Compute (EC2) along with associated Elastic Block Storage (EBS) volumes as well as their snapshots. I also have AWS S3 buckets in different regions that are on various tiers from standard to infrequent access (IA), as well as some data on Glacier. Data from my DPS at Bluehost gets protected to a AWS S3 bucket that I can access from AWS EC2, as well as via other locations including Microsoft Azure as needed.

Some on-premises data also gets protected to AWS S3 (as well as to elsewhere) using various tools, for different granularity, frequency, access and retention. After all, everything is not the same, why treat it the same. Some of the data protected to AWS S3 buckets is in native format (e.g. they appear as objects to S3 or object enabled applications), as well as file to file based applications with appropriate tools.

Other data that is also protected to AWS S3 from different data protection or backup tools are stored in vendor neutral or vendor specific save set, zip, tar ball or other formats. In other words, I need the tool or compatible tool that knows the format of the saved data to retrieve individual data files, items or objects. Note that this is similar to storing data on tape, HDDs, SSD or other media in native format vs. in some type of encapsulate save set or other format.

In addition to protecting data to AWS, I also have data at Microsoft Azure among other locations. Other locations include non-cloud based off-site where encrypted removable media is periodically taken to a safe secure place as a master, gold in case of major emergency, ransomeware copy.

Why not just rely on cloud copies?

Simple, I can pull individual files or relatively small amounts of data back from the cloud sometimes faster (or easier) than from on-site copies, let alone my off-site, off-line, air gap copies. On the other hand, if I need to restore large amounts of data, without a fast network, it can be quicker to get the air gap off-line, off-site copy, do the large restore, then apply incremental or changed data via cloud. In other a hybrid approach.

Now a common question I get is why not just do one or the other and save some money. Good point, I would save some money, however by doing the above among other things, they are part of being able to test, try new and different things, gain insight, experience not to mention walk the talk vs. simply talking the talk.

Of course Im always looking for ways to streamline to make my data protection more efficient, as well as effective (along with remove complexity and costs).

  • Everything is not the same, so why treat it all the same with common SLO, RTO, RPO and retention?
  • Likewise why treat and store all data the same way, on the same tiers of technology
  • Gain insight and awareness into environment, applications, workloads, PACE needs
  • Applications, data, systems or devices are protected with different granularity and frequency
  • Apply applicable technology and tools to the task at hand
  • Any data I have in cloud has a copy elsewhere, likewise, any data on-premises has a copy in the cloud or elsewhere
  • I implement the 4 3 2 1 rule by having multiple copies, versions, data in different locations, on and off-line including cloud
  • From a security standpoint, many different things are implemented on a logical as well as physical basis including encryption
  • Ability to restore data as well as applications or image instances locally as well as into cloud environments
  • Leverage different insight and awareness, reporting, analytics and monitoring tools
  • Mix of local storage configured with different RAID and other protection
  • Test, find, fix, remediate improve the environment including leveraging lessons learned

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

Everything is not the same, thats why in my environment I use different technologies, tools and techniques to protect my data. This also means having different RTO, RPO across various applications, data and systems as well as devices. Data that is more important has more copies, versions in different locations as well as occurring more frequently as part of 4 3 2 1 data protection. Other data that does not change as frequently, or time sensitive have alternate RTO and RPO along with corresponding frequency of protection.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series Part 9 who’s Doing What (Toolbox Technology Tools).

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Data Protection Diaries Fundamentals Who Is Doing What Toolbox Technology Tools

Data Protection Toolbox Whos Doing What Technology Tools

Updated 1/17/2018

Data protection toolbox is a companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is Part 9 of a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Software Defined Data Infrastructure Essentials Book SDDC

Click here to view the previous post Part 8 Walking The Data Protection Talk, and click here to view the next post Part 10 Data Protection Resources Where to Learn More.

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection who’s Doing What ( Toolbox Technology Tools).

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

who’s Doing What (Toolbox Technology Tools)

SDDC SDDI data center data protection toolbox
Data Protection Toolbox

Note that this post is evolving with additional tools, technologies, techniques, hardware, software, services being added over time along with applicable industry links.

The following are a sampling of some hardware, software, solution and component vendors along with service providers involved with data protection from RAID, Erasure Codes (EC) to snapshots, backup, BC, BR, DR, archive, security, cloud, bulk object storage, HDDs, SSD, tape among others including buzzword (and buzz term trends) bingo. Acronis, Actifio, Arcserve, ATTO, AWS, Backblaze, Barracuda, Broadcom, Caringo, Chelsio (offload), Code42/Crashplan, Cray, Ceph, Cisco, Cloudian, Cohesity, Compuverde, Commvault, Datadog, Datrium, Datos IO, DDN, Dell EMC, Druva, E8, Elastifile, Exagrid, Excelero, Fujifilm, Fujutsu, Google, HPE, Huawei, Hedvig, IBM, Intel, Iomega, Iron Mountain, IBM, Jungledisk, Kinetic key value drives (Seagate), Lenovo, LTO organization, Mangstor, Maxta, Mellanox (offload), Micron, Microsoft (Azure, Windows, Storage Spaces), Microsemi, Nakivo, NetApp, NooBaa, Nexsan, Nutanix, OpenIO, OpenStack (Swift), Oracle, Panasas, Panzura, Promise, Pure, Quantum, Quest, Qumulo, Retrospect, Riverbed, Rozo, Rubrik, Samsung, Scale, Scality, Seagate (DotHill), Sony, Solarwinds, Spectralogic, Starwind, Storpool, Strongbox, Sureline, Swiftstack, Synology, Toshiba, Tintri, Turbonomics, Unitrends, Unix and Linux platforms, Vantara, Veeam, VMware, Western Digital (Amplidata, Tegile and others), WekaIO, X-IO, Zadara and Zmanda among many others.

Note if you dont see yours, or your favorite, preferred or clients listed above or in the data Infrastructure industry related links send us a note for consideration to be included in future updates, or having a link, or sponsor spot pointing to your site added. Feel free to add a non sales marketing pitch to courteous comments to the comment section below.

View additional IT, data center and data Infrastructure along with data protection related vendors, services, tools, technologies links here.

Where To Learn More

Continue reading additional posts in this series of Data Infrastructure Data Protection fundamentals and companion to Software Defined Data Infrastructure Essentials (CRC Press 2017) book, as well as the following links covering technology, trends, tools, techniques, tradecraft and tips.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

Part of modernizing data protection for various data center and data infrastructure environments is to know the tools, technologies and trends that are part of your data protection toolbox. The other part of modernizing data is protection is knowing the techniques of how to use different tools, technologies to meet various application workload performance, availability, capacity economic (PACE) needs.

Also keep in mind that information services requires applications (e.g. programs) and that programs are a combination of algorithms (code, rules, policies) and data structures (e.g. data and how it is organized including unstructured). What this means is that data protection needs to address not only data, also the applications, configuration settings, metadata as well as protecting the protection tools and its data.

Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here. Meanwhile, continue reading with the next post in this series, Part 10 Data Protection Fundamental Resources Where to Learn More.

Ok, nuff said, for now.

Gs

Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Data Protection Diaries Fundamental Resources Where to Learn More

Data Protection Diaries Fundamental Resources Where to Learn More

Companion to Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Fundamental Server Storage I/O Tradecraft ( CRC Press 2017)

server storage I/O data infrastructure trends

By Greg Schulzwww.storageioblog.com November 26, 2017

This is the last in a multi-part series on Data Protection fundamental tools topics techniques terms technologies trends tradecraft tips as a follow-up to my Data Protection Diaries series, as well as a companion to my new book Software Defined Data Infrastructure Essentials – Cloud, Converged, Virtual Server Storage I/O Fundamental tradecraft (CRC Press 2017).

Click here to view the previous post Part 9 – who’s Doing What ( Toolbox Technology Tools).

Software Defined Data Infrastructure Essentials Book SDDC

Post in the series includes excerpts from Software Defined Data Infrastructure (SDDI) pertaining to data protection for legacy along with software defined data centers ( SDDC), data infrastructures in general along with related topics. In addition to excerpts, the posts also contain links to articles, tips, posts, videos, webinars, events and other companion material. Note that figure numbers in this series are those from the SDDI book and not in the order that they appear in the posts.

In this post the focus is around Data Protection Resources Where to Learn More.

SDDC, SDI, SDDI data infrastructure
Figure 1.5 Data Infrastructures and other IT Infrastructure Layers

Software Defined Data Infrastructure Essentials Table of Contents (TOC)

Here is a link (PDF) to the table of contents (TOC) for Software Defined Data Infrastructure Essentials.

The following is a Software Defined Data Infrastructure Essentials book TOC summary:

Chapter 1: Server Storage I/O and Data Infrastructure Fundamentals
Chapter 2: Application and IT Environments
Chapter 3: Bits, Bytes, Blobs, and Software-Defined Building Blocks
Chapter 4: Servers: Physical, Virtual, Cloud, and Containers
Chapter 5: Server I/O and Networking
Chapter 6: Servers and Storage-Defined Networking
Chapter 7: Storage Mediums and Component Devices
Chapter 8: Data Infrastructure Services: Access and Performance
Chapter 9: Data Infrastructure Services: Availability, RAS, and RAID
Chapter 10: Data Infrastructure Services: Availability, Recovery-Point Objective, and Security
Chapter 11: Data Infrastructure Services: Capacity and Data Reduction
Chapter 12: Storage Systems and Solutions (Products and Cloud)
Chapter 13: Data Infrastructure and Software-Defined Management
Chapter 14: Data Infrastructure Deployment Considerations
Chapter 15: Software-Defined Data Infrastructure Futures, Wrap-up, and Summary
Appendix A: Learning Experiences
Appendix B: Additional Learning, Tools, and tradecraft Tricks
Appendix C: Frequently Asked Questions
Appendix D: Book Shelf and Recommended Reading
Appendix E: Tools and Technologies Used in Support of This Book
Appendix F: How to Use This Book for Various Audiences
Appendix G: Companion Website and Where to Learn More
Glossary
Index

Click here to view (PDF) table of contents (TOC).

Data Protection Resources Where To Learn More

Learn more about Data Infrastructure and Data Protection related technology, trends, tools, techniques, tradecraft and tips with the following links.

The following are the various posts that are part of this data protection series:

  • Part 1Data Infrastructure Data Protection Fundamentals
  • Part 2 – Reliability, Availability, Serviceability ( RAS) Data Protection Fundamentals
  • Part 3 – Data Protection Access Availability RAID Erasure Codes ( EC) including LRC
  • Part 4 – Data Protection Recovery Points (Archive, Backup, Snapshots, Versions)
  • Part 5 – Point In Time Data Protection Granularity Points of Interest
  • Part 6 – Data Protection Security Logical Physical Software Defined
  • Part 7 – Data Protection Tools, Technologies, Toolbox, Buzzword Bingo Trends
  • Part 8 – Data Protection Diaries Walking Data Protection Talk
  • Part 9 – who’s Doing What ( Toolbox Technology Tools)
  • Part 10Data Protection Resources Where to Learn More

  • The following are various data protection blog posts:

  • Welcome to the Data Protection Diaries
  • Until the focus expands to data protection, backup is staying alive!
  • The blame game, Does cloud storage result in data loss?
  • Loss of data access vs. data loss
  • Revisiting RAID storage remains relevant and resources
  • Only you can prevent cloud (or other) data loss
  • Data protection is a shared responsibility
  • Time for CDP (Commonsense Data Protection)?
  • Data Infrastructure Server Storage I/O Tradecraft Trends (skills, experiences, knowledge)
  • My copies were corrupted: The [4] 3-2-1 rule and more about 4 3 2 1 as well as 3 2 1 here and here
  • The following are various data protection tips and articles:

  • Via Infostor Cloud Storage Concerns, Considerations and Trends
  • Via Network World What’s a data infrastructure?
  • Via Infostor Data Protection Gaps, Some Good, Some Not So Good
  • Via Infostor Object Storage is in your future
  • Via Iron Mountain Preventing Unexpected Disasters
  • Via InfoStor – The Many Variations of RAID Storage
  • Via InfoStor – RAID Remains Relevant, Really!
  • Via WservNews Cloud Storage Considerations (Microsoft Azure)
  • Via ComputerWeekly Time to restore from backup: Do you know where your data is?
  • Via Network World Ensure your data infrastructure remains available and resilient
  • The following are various data protection related webinars and events:

  • BrightTalk Webinar Data Protection Modernization – Protect, Preserve and Serve you Information
  • BrightTalk Webinar BCDR and Cloud Backup Protect Preserve and Secure Your Data Infrastructure
  • TechAdvisor Webinar (Free with registration) All You Need To Know about ROBO data protection
  • TechAdvisor Webinar (Free with registration) Tips for Moving from Backup to Full Disaster Recovery
  • The following are various data protection tools, technologies, services, vendor and industry resource links:

  • Various Data Infrastructure related news commentary, events, tips and articles
  • Data Center and Data Infrastructure industry links (vendors, services, tools, technologies, hardware, software)
  • Data Infrastructure server storage I/O network Recommended Reading List Book Shelf
  • Software Defined Data Infrastructure Essentials (CRC 2017) Book
  • Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    What This All Means

    Everything is not the same across environments, data centers, data infrastructures including SDDC, SDX and SDDI as well as applications along with their data.

    Likewise everything is and does not have to be the same when it comes to Data Protection.

    Since everything is not the same, various data protection approaches are needed to address various application performance, availability, capacity economic (PACE) needs, as well as SLO and SLAs.

    Data protection encompasses many different hardware, software, services including cloud technologies, tools, techniques, best practices, policies and tradecraft experience skills (e.g. knowing what to use when, where, why and how).

    Software Defined Data Infrastructure Essentials Book SDDC

    Context is important as different terms have various meanings depending on what they are being discussed with. Likewise different technologies and topics such as object, blob, backup, replication, RAID, erasure code (EC), mirroring, gaps (good, bad, ugly), snapshot, checkpoint, availability, durability among others have various meanings depending on context, as well as implementation approach.

    In most cases there is no bad technology or tool, granted there are some poor or bad (even ugly) implementations, as well as deployment or configuration decisions. What this means is the best technology or approach for your needs may be different from somebody else’s and vice versa.

    Some other points include there is no such thing as an information recession with more data generated every day, granted, how that data is transformed or stored can be in a smaller footprint. Likewise there is an increase in the size of data including unstructured big data, as well as the volume (how much data), as well as velocity (speed at which it is created, moved, processed, stored). This also means there is an increased dependency on data being available, accessible and intact with consistency. Thus the fundamental role of data Infrastructures (e.g. what’s inside the data center or cloud) is to combine resources, technologies, tools, techniques, best practices, policies, people skill set, experiences (e.g. tradecraft) to protect, preserve, secure and serve information (applications and data).

    modernizing data protection including backup, availability and related topics means more than swapping out one hardware, software, service or cloud for whatever is new, and then using it in old ways.

    What this means is to start using new (and old) things in new ways, for example move beyond using SSD or HDDs like tape as targets for backup or other data protection approaches. Instead use SSD, HDDs or cloud as a tier, yet also to enable faster protection and recovery by stepping back and rethinking what to protect, when, where, why, how and apply applicable techniques, tools and technologies. Find a balance between knowing all about the tools and trends while not understanding how to use those toolbox items, as well as knowing all about the techniques of how to use the tools, yet not knowing what the tools are.

    Want to learn more, have questions about specific tools, technologies, trends, vendors, products, services or techniques discussed in this series, send a note (info at storageio dot com) or via our contact page. We can set up a time to discuss your questions or needs pertaining to Data Protection as well as data infrastructures related topics from legacy to software defined virtual, cloud, container among others. For example consulting, advisory services, architecture strategy design, technology selection and acquisition coaching, education knowledge transfer sessions, seminars, webinars, special projects, test drive lab reviews or audits, content generation, videos, podcasts, custom content, chapter excerpts, demand generation among many other things.

    Get your copy of Software Defined Data Infrastructure Essentials here at Amazon.com, at CRC Press among other locations and learn more here.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    HPE Announces AMD Powered Gen 10 ProLiant DL385 For Software Defined Workloads

    HPE Announces AMD Powered Gen 10 ProLiant DL385 For Software Defined Workloads

    server storage I/O data infrastructure trends

    By Greg Schulzwww.storageioblog.com November 20, 2017

    HPE Announced today a new AMD EPYC 7000 Powered Gen 10 ProLiant DL385 for Software Defined Workloads including server virtualization, software-defined data center (SDDC), software-defined data infrastructure (SDDI), software-defined storage among others. These new servers are part of a broader Gen10 HPE portfolio of ProLiant DL systems.

    HPE AMD EPYC Gen10 DL385
    24 Small Form Factor Drive front view DL385 Gen 10 Via HPE

    The value proposition being promoted by HPE of these new AMD powered Gen 10 DL385 servers besides supporting software-defined, SDDI, SDDC, and related workloads are security, density and lower price than others. HPE is claiming with the new AMD EPYC system on a chip (SoC) processor powered Gen 10 DL385 that it is offering up to 50 percent lower cost per virtual machine (VM) than traditional server solutions.

    About HPE AMD Powered Gen 10 DL385

    HPE AMD EPYC 7000 Gen 10 DL385 features:

    • 2U (height) form factor
    • HPE OneView and iLO management
    • Flexible HPE finance options
    • Data Infrastructure Security
    • AMD EPYC 7000 System on Chip (SoC) processors
    • NVMe storage (Embedded M.2 and U.2/8639 Small Form Factor (SFF) e.g. drive form factor)
    • Address server I/O and memory bottlenecks

    These new HPE servers are positioned for:

    • Software Defined, Server Virtualization
    • Virtual Desktop Infrastructure (VDI) workspaces
    • HPC, Cloud and other general high-density workloads
    • General Data Infrastructure workloads that benefit from memory-centric or GPUs

    Different AMD Powered DL385 ProLiant Gen 10 Packaging Options

    Common across AMD EPYC 7000 powered Gen 10 DL385 servers are 2U high form factor, iLO management software and interfaces, flexible LAN on Motherboard (LOM) options, MicroSD (optional dual MicroSD), NVMe (embedded M.2 and SFF U.2) server storage I/O interface and drives, health and status LEDs, GPU support, single or dual socket processors.

    HPE AMD EPYC Gen10 DL385 Look Inside
    HPE DL385 Gen10 Inside View Via HPE

    HPE AMD EPYC Gen10 DL385 Rear View
    HPE DL385 Gen10 Rear View Via HPE

    Other up to three storage drive bays, support for Large Form Factor (LFF) and Small Form Factor (SFF) devices (HDD and SSD) including SFF NVMe (e.g., U.2) SSD. Up to 4 x Gbe NICs, PCIe riser for GPU (optional second riser requires the second processor). Other features and options include HPE SmartArray (RAID), up to 6 cooling fans, internal and external USB 3. Optional universal media bay that can also add a front display, optional Optical Disc Drive (ODD), optional 2 x U.2 NVMe SFF SSD. Note media bay occupies one of three storage drive bays.

    HPE AMD EPYC Gen10 DL385 Form Factor
    HPE DL385 Form Factor Via HPE

    Up to 3 x Drive Bays
    Up to 12 LFF drives (2 per bay)
    Up to 24 SFF drives ( 3 x 8 drive bays, 6 SFF + 2 NVMe U.2 or 8 x NVMe)

    AMD EPYC 7000 Series

    The AMD EPYC 7000 series is available in the single and dual socket. View additional AMD EPYC speeds and feeds in this data sheet (PDF), along with AMD server benchmarks here.

    HPE AMD EPYC Specifications
    HPE DL385 Gen 10 AMD EPYC Specifications Via HPE

    AMD EPYC 7000 General Features

    • Single and dual socket
    • Up to 32 cores, 64 threads per socket
    • Up to 16 DDR4 DIMMS over eight channels per socket (e.g., up to 2TB RAM)
    • Up to 128 PCIe Gen 3 lanes (e.g. combination of x4, x8, x16 etc)
    • Future 128GB DIMM support

    AMD EPYC 7000 Security Features

    • Secure processor and secure boot for malware rootkit protection
    • System memory encryption (SME)
    • Secure Encrypted Virtualization (SEV) hypervisors and guest virtual machine memory protection
    • Secure move (e.g., encrypted) between enabled servers

    Where To Learn More

    Learn more about Data Infrastructure and related server technology, trends, tools, techniques, tradecraft and tips with the following links.

  • AMD EPYC 7000 System on Chip (SoC) processors
  • Gen10 HPE portfolio and ProLiant DL systems.
  • Various Data Infrastructure related news commentary, events, tips and articles
  • Data Center and Data Infrastructure industry links
  • Data Infrastructure server storage I/O network Recommended Reading List Book Shelf
  • Software Defined Data Infrastructure Essentials (CRC 2017) Book
  • What This All Means

    With the flexible options including HDD, SSD as well as NVMe accessible SSDs, large memory capacity along with computing cores, these new solutions provide good data infrastructure server density (e.g., CPU, memory, I/O, storage) per cubic foot or meter per cost.

    I look forward to trying one of these systems out for software-defined scenarios including virtualization, software-defined storage (SDS) among others workload scenarios. Overall the HPE announcement of the new AMD EPYC 7000 Powered Gen 10 ProLiant DL385 looks to be a good option for many environments.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    AWS Announces New S3 Cloud Storage Security Encryption Features

    AWS Announces New S3 Cloud Storage Security Encryption Features

    server storage I/O data infrastructure trends

    Updated 1/17/2018

    Amazon Web Services (AWS) recently announced new Simple Storage Service (S3) e.g. AWS S3 encryption and security enhancements including Default Encryption, Permission Checks, Cross-Region Replication ACL Overwrite, Cross-Region Replication with KMS and Detailed Inventory Report. Another recent announcement by AWS is for PrivateLinks endpoints within a Virtual Private Cloud (VPC).

    AWS Dashboard
    AWS Service Dashboard

    Default Encryption

    Extending previous security features, now you can mandate all objects stored in a given S3 bucket be encrypted without specifying a bucket policy that rejects non-encrypted objects. There are three server-side encryption (SSE) options for S3 objects including keys managed by S3, AWS KMS and SSE Customer ( SSE-C) managed keys. These options provide more flexibility as well as control for different environments along with increased granularity. Note that encryption can be forced on all objects in a bucket by specifying a bucket encryption configuration. When an unencrypted object is stored in an encrypted bucket, it will inherit the same encryption as the bucket, or, alternately specified by a PUT required.

    AWS S3 Bucket Encryption
    AWS S3 Buckets

    Permission Checks

    There is now an indicator on the S3 console dashboard prominently indicating which S3 buckets are publicly accessible. In the above image, some of my AWS S3 buckets are shown including one that is public facing. Note in the image above how there is a notion next to buckets that are open to public.

    Cross-Region Replication ACL Overwrite and KMS

    AWS Key Management Service (KMS) keys can be used for encrypting objects. Building on previous cross-region replication capabilities, now when you replicate objects across AWS accounts, a new ACL providing full access to the destination account can be specified.

    Detailed Inventory Report

    The S3 Inventory report ( which can also be encrypted) now includes the encryption status of each object.

    PrivateLink for AWS Services

    PrivateLinks enable AWS customers to access services from a VPC without using a public IP as well as traffic not having to go across the internet (e.g. keeps traffic within the AWS network. PrivateLink endpoints appear in Elastic Network Interface (ENI) with private IPs in your VPC and are highly available, resiliency and scalable. Besides scaling and resiliency, PrivateLink eliminates the need for white listing of public IPs as well as managing internet gateway, NAT and firewall proxies to connect to AWS services (Elastic Cloud Compute (EC2), Elastic Load Balancer (ELB), Kinesis Streams, Service Catalog, EC2 Systems Manager). Learn more about AWS PrivateLink for services here including  VPC Endpoint Pricing here

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    What This All Means

    Common cloud concern considerations include privacy and security. AWS S3 among other industry cloud service and storage providers have had their share of not so pleasant news coverage involving security.

    Keep in mind that data protection including security is a shared responsibility (and only you can prevent data loss). This means that the vendor or service provider has to take care of their responsibility making sure their solutions have proper data protection and security features by default, as well as extensions, and making those capabilities known to consumers.

    The other part of shared responsibility is that consumers and users of cloud services need to know what the capabilities are, defaults and options as well as when to use various approaches. Ultimately it is up to the user of a cloud service to implement best practices to leverage cloud as well as their own on-premises technologies so that they can support data infrastructure that in turn protect, preserve, secure and serve information (along with their applications and data).

    These are good enhancements by AWS to make their S3 cloud storage security encryption features available as well as provide options and awareness for users on how to use those capabilities.

     

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    October 2017 Server StorageIO Data Infrastructure Update Newsletter



    Server StorageIO October 2017 Data Infrastructure Update Newsletter

    Volume 17, Issue 10 (October 2017)

    Hello and welcome to the October 2017 issue of the Server StorageIO data infrastructure update newsletter.

    Software-Defined Data Infrastructure Essentials SDDI SDDC

    October has been a busy month pertaining data infrastructure including server storage I/O related trends, activities, news, perspectives and related topics, so let’s have a look at them.

    In This Issue

    Enjoy this edition of the Server StorageIO data infrastructure update newsletter.

    Cheers GS

    Data Infrastructure and IT Industry Activity Trends

    Some recent Industry Activities, Trends, News and Announcements include:

    Startup Aparavi launched with a SaaS platform for managing long-term data retention. As part of a move to streamline the acquisition of Brocade by Broadcom (formerly known as Avago), the Brocade data center Ethernet networking business is being sold to Extreme networks. Datacore also updated their software defined storage solutions in October.

    Cisco announced new storage networking products and acquisition of Brodsoft (cloud calling and contact center solutions). As part of continued support for Fibre Channel based data infrastructure environments, Cisco has announced a 1U MDS 9132T 32 port 32 Gbps Fibre Channel Switch with FCP (SCSI Fibre Channel Protocol) now, and emerging FC-NVMe future support. Also announced are SAN telemetry activity monitoring, insight and event streaming for analysis in MDS 9700 32Gbps module.

    Cisco also announced interoperability for data center and data infrastructure insight, activity monitoring and telemetry with Virtual Instruments Virtual Wisdom technology eliminating the reliance on hardware based probes, along with Fibre Channel N-Port virtualization on Nexus 9300-FX DC switch.

    Commvault announced scale-out data protection with ScaleProtect for Cisco UCS platforms, along with their HyperScale appliance and HyperScale software.

    IBM had several October announcements include LTO 8 related, FlashSystem V9000 updates (e.g. All Flash Array) enclosure as well as hardware based compression, FlashSystem A9000 leveraging 3D TLC NAND flash (lower cost, higher capacity) among others.

    There is plenty of content (blogs, articles, podcasts, webinars, videos, white papers, presentations) on when to do containers, microservices and serverless compute including mesos, kubernetes and docker among others. What about when not to use those approaches or caveats to be aware of, here is such a piece (via Redhat) to have a look at.

    Granted if you are part of the micro services cheerleading bandwagon crowd you might not agree with the authors points, after all, everything is not the same in data centers and data infrastructures. Speaking of serverless, containers, here is a good post about Docker Swarm vs. Kubernetes management over at Upcloud.

    In Microsoft and Azure related activity, despite some early speculation in some venues that Storage Spaces Direct (S2D) was being discontinued as it was not part of Server release 1709, the reality is S2D is very much alive.

    Microsoft LTSC and SAC release cycles
    Image via Microsoft.com

    However some clarification is needed that might have lead to some initial speculation due to lack of understanding the new Microsoft release cycle.

    Microsoft has gone to Semi Annual Channel (SAC) releases that introduce new features in advance of the Long Term Support Channel (LTSC). LTSC are what you might be familiar with Windows and Windows Server releases that are updates spread out over time for a given major version (e.g. going from Server 2012 to Server 2012 R2 and so forth). The current Windows Server LTSC is the base introduced fall of 2016 along with incremental updates.

    By comparison, think of SAC as a branch channel for early adopters to get new features and with 1709 (e.g. September 2017), the focus is on containers. A mistake that has been made is to assume that a SAC release is actually a new major LTSC release, thus probably why some thought S2D was dead as it is not in SAC 1709. Indications from Microsoft are that there will be S2D enhancements in the next SAC, as well as future LTSC.

    For those interested in IoT, check out this Microsoft Azure IoT Hub and device twin document. Here is a post by Thomas Mauer looking at 10 hidden Hyper-V features to know about.

    In other activity, Minio announced experimental AWS S3 API support for Backblaze storage service. Software Defined Serverless Storage startup OpenIO gets $5M USD in additional funding. Quantum and other LTO Organization vendors have announced support for the new LTO version 8 tape drives and media. In addition to LTO 8, new roadmaps including out to LTO 12 are outlined here, and VMware vCloud Air is hosted by OVH. Western Digital Corporation (WDC) announced Microwave Assisted Magnetic Recording (MAMR) enabled Hard Disk Drives (HDD) that will enable future, larger capacity devices to be brought to market.

    Check out other industry news, comments, trends perspectives here.

    Server StorageIO Commentary in the news

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via HPE Insights: Comments on Public cloud versus on-prem storage
    Via arsTechnica: Comments on cloud backup disaster recovery
    Via Gizmodo: Comments on WDC 40TB HDD
    Via CDW: Comments on Is Your Network About To Fail?
    Via EnterpriseStorageForum: Comments on Trends for Data Storage with Big Data Analytics
    Via EnterpriseStorageForum: Comments on 8 ways to save on cloud storage
    Via EnterpriseStorageForum: Comments on Google Cloud Platform and Storage

    View more Server, Storage and I/O trends and perspectives comments here

    Server StorageIOblog Posts

    Recent and popular Server StorageIOblog posts include:

    In Case You Missed It #ICYMI

    View other recent as well as past StorageIOblog posts here

    Server StorageIO Data Infrastructure Tips and Articles

    Recent Server StorageIO industry trends perspectives commentary in the news.

    Via EnterpriseStorageForum: Comments on Who Will Rule the Storage World?
    Via InfoGoto: Comments on Google Cloud Platform Gaining Data Storage Momentum
    Via InfoGoto: Comments on Singapore High Rise Data Centers
    Via InfoGoto: Comments on New Tape Storage Capacity
    Via EnterpriseStorageForum: Comments on 8 ways to save on cloud storage
    Via EnterpriseStorageForum: Comments on Google Cloud Platform and Storage

    View more Server, Storage and I/O trends and perspectives comments here

    Server StorageIO Recommended Reading (Watching and Listening) List

    In addition to my own books including Software Defined Data Infrastructure Essentials (CRC Press 2017), the following are Server StorageIO recommended reading, watching and listening list items. The list includes various IT, Data Infrastructure and related topics.

    Intel Recommended Reading List (IRRL) for developers is a good resource to check out.

    Its October which means that it is also Blogtober, check out some of the blogs and posts occurring during October here.

    For those involved with VMware, check out Frank Denneman VMware vSphere 6.5 host resource guide-book here at Amazon.com.

    Docker: Up & Running: Shipping Reliable Containers in Production by Karl Matthias & Sean P. Kane via Amazon.com here.

    Essential Virtual SAN (VSAN): Administrator’s Guide to VMware Virtual SAN,2nd ed. by Cormac Hogan & Duncan Epping via Amazon.com here.

    Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale by Tom White via Amazon.com here.

    Cisco IOS Cookbook: Field tested solutions to Cisco Router Problems by Kevin Dooley and Ian Brown Via Amazon.com here.

    Watch for more items to be added to the recommended reading list book shelf soon.

    Events and Activities

    Recent and upcoming event activities.

    Nov. 9, 2017 – Webinar – All You Need To Know about ROBO Data Protection Backup
    Nov. 2, 2017 – Webinar – Modern Data Protection for Hyper-Convergence
    Sep. 21, 2017 – MSP CMG – Minneapolis MN
    Sep. 20, 2017 – Webinar – BC, DR and Business Resiliency (BR) tips
    Sep. 14, 2017 – Fujifilm IT Executive Summit – Seattle WA
    Sep. 12, 2017 – SNIA Software Developers Conference (SDC) – Santa Clara CA
    Sep. 7, 2017 – Wipro SDX – Enabling, Planning Your Software Defined Journey

    See more webinars and activities on the Server StorageIO Events page here.

    Server StorageIO Industry Resources and Links

    Useful links and pages:
    Microsoft TechNet – Various Microsoft related from Azure to Docker to Windows
    storageio.com/links – Various industry links (over 1,000 with more to be added soon)
    objectstoragecenter.com – Cloud and object storage topics, tips and news items
    OpenStack.org – Various OpenStack related items
    storageio.com/downloads – Various presentations and other download material
    storageio.com/protect – Various data protection items and topics
    thenvmeplace.com – Focus on NVMe trends and technologies
    thessdplace.com – NVM and Solid State Disk topics, tips and techniques
    storageio.com/converge – Various CI, HCI and related SDS topics
    storageio.com/performance – Various server, storage and I/O benchmark and tools
    VMware Technical Network – Various VMware related items

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Data Infrastructure server storage I/O network Recommended Reading #blogtober

    server storage I/O data infrastructure trends recommended reading list

    Updated 7/30/2018

    The following is an evolving recommended reading list of data infrastructure topics including, server, storage I/O, networking, cloud, virtual, container, data protection and related topics that includes books, blogs, podcast’s, events and industry links among other resources.

    Various Data Infrastructure including hardware, software, services related links:

    Links A-E
    Links F-J
    Links K-O
    Links P-T
    Links U-Z
    Other Links

    In addition to my own books including Software Defined Data Infrastructure Essentials (CRC Press 2017), the following are Server StorageIO recommended reading list items . The recommended reading list includes various IT, Data Infrastructure and related topics.

    Intel Recommended Reading List (IRRL) for developers is a good resource to check out.

    Duncan Epping (@DuncanYB), Frank Denneman (@FrankDenneman) and Neils Hagoort (@NHagoort) have released their VMware vSphere 6.7 Clustering Deep Dive book available at venues including Amazon.com. This is the latest in a series of Cluster and deep dive books from Frank and Duncan which if you are involved with VMware, SDDC and related software defined data infrastructures these should be on your bookshelf.

    Check out the Blogtober list of check out some of the blogs and posts occurring during October 2017 here.

    Preston De Guise aka @backupbear is Author of several books has an interesting new site Foolsrushin.info that looks at topics including Ethics in IT among others. Check out his new book Data Protection: Ensuring Data Availability (CRC Press 2017) and available via Amazon.com here.

    Brendan Gregg has a great site for Linux performance related topics here.

    Greg Knieriemen has a must read weekly blog, post, column collection of whats going on in and around the IT and data infrastructure related industries, Check it out here.

    Interested in file systems, CIFS, SMB, SAMBA and related topics then check out Chris Hertels book on implementing CIFS here at Amazon.com

    For those involved with VMware, check out Frank Denneman VMware vSphere 6.5 host resource guide-book here at Amazon.com.

    Docker: Up & Running: Shipping Reliable Containers in Production by Karl Matthias & Sean P. Kane via Amazon.com here.

    Essential Virtual SAN (VSAN): Administrator’s Guide to VMware Virtual SAN,2nd ed. by Cormac Hogan & Duncan Epping via Amazon.com here.

    Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale by Tom White via Amazon.com here.

    Systems Performance: Enterprise and the Cloud by Brendan Gregg Via Amazon.com here.

    Implementing Cloud Storage with OpenStack Swift by Amar Kapadia, Sreedhar Varma, & Kris Rajana Via Amazon.com here.

    The Human Face of Big Data by Rick Smolan & Jennifer Erwitt Via Amazon.com here.

    VMware vSphere 5.1 Clustering Deepdive (Vol. 1) by Duncan Epping & Frank Denneman Via Amazon.com here. Note: This is an older title, but there are still good fundamentals in it.

    Linux Administration: A Beginners Guide by Wale Soyinka Via Amazon.com here.

    TCP/IP Network Administration by Craig Hunt Via Amazon.com here.

    Cisco IOS Cookbook: Field tested solutions to Cisco Router Problems by Kevin Dooley and Ian Brown Via Amazon.com here.

    I often mention in presentations a must have for anybody involved with software defined anything, or programming for that matter which is the Niklaus Wirth classic Algorithms + Data Structures = Programs that you can get on Amazon.com here.

    Seven Databases in Seven Weeks including NoSQL

    Another great book to have is Seven Databases in Seven Weeks (here is a book review) which not only provides an overview of popular NoSQL databases such as Cassandra, Mongo, HBASE among others, lots of good examples and hands on guides. Get your copy here at Amazon.com.

    Additional Data Infrastructure and related topic sites

    In addition to those mentioned above, other sites, venues and data infrastructure related resources include:

    aiim.com – Archiving and records management trade group

    apache.org – Various open-source software

    blog.scottlowe.org – Scott Lowe VMware Networking and topics

    blogs.msdn.microsoft.com/virtual_pc_guy – Ben Armstrong Hyper-V blog

    brendangregg.com – Linux performance-related topics

    cablemap.info – Global network maps

    CMG.org – Computer Measurement Group (CMG)

    communities.vmware.com – VMware technical community and resources

    comptia.org – Various IT, cloud, and data infrastructure certifications

    cormachogan.com – Cormac Hogan VMware and vSAN related topics

    csrc.nist.gov – U.S. government cloud specifications

    dmtf.org – Distributed Management Task Force (DMTF)

    ethernetalliance.org – Ethernet industry trade group

    fibrechannel.org – Fibre Channel trade group

    github.com – Various open-source solutions and projects

    Intel Reading List – recommended reading list for developers

    ieee.org – Institute of Electrical and Electronics Engineers

    ietf.org – Internet Engineering Task Force

    iso.org – International Standards Organizations

    it.toolbox.com – Various IT and data infrastructure topics forums

    labs.vmware.com/flings – VMware Fling additional tools and software

    nist.gov – National Institute of Standards and Technology

    nvmexpress.org – NVM Express (NVMe) industry trade group

    objectstoragecenter.com – Various object and cloud storage items

    opencompute.org – Open Compute Project (OCP) servers and related topics

    opendatacenteralliance.org – Open Data Center Alliance (ODCA)

    openfabrics.org – Open-fabric software industry group

    opennetworking.org – Open Networking Foundation (ONF)

    openstack.org – OpenStack resources

    pcisig.com – Peripheral Component Interconnect (PCI) trade group

    reddit.com – Various IT, cloud, and data infrastructure topics

    scsita.org – SCSI trade association (SAS and others)

    SNIA.org – Storage Network Industry Association (SNIA)

    Speakingintech.com – Popular industry and data infrastructure podcast

    Storage Bibliography – Collection of Dr. J. Metz storage related content

    technet.microsoft.com – Microsoft TechNet data infrastructure–related topics

    thenvmeplace.com – various NVMe and related tools, topics and links

    thevpad.com – Collection of various virtualization and related sites

    thessdplace.com – various NVM, SSD, flash, 3D XPoint related topics, tools, links

    tpc.org – Transaction Performance Council benchmark site

    vmug.org – VMware User Groups (VMUG)

    wahlnetwork.com – Chris Whal Networking and related topics

    yellow-bricks.com – Duncan Epping VMware and related topics

    Additional Data Infrastructure Venues

    Additional useful data infrastructure related information can be found at BizTechMagazine, BrightTalk, ChannelProNetwork, ChannelproSMB, ComputerWeekly, Computerworld, CRN, CruxialCIO, Data Center Journal (DCJ), Datacenterknowledge, and DZone. Other good sourses include Edtechmagazine, Enterprise Storage Forum, EnterpriseTech, Eweek.com, FedTech, Google+, HPCwire, InfoStor, ITKE, LinkedIn, NAB, Network Computing, Networkworld, and nextplatform. Also check out Reddit, Redmond Magazine and Webinars, Spiceworks Forums, StateTech, techcrunch.com, TechPageOne, TechTarget Venues (various Search sites, e.g., SearchStorage, SearchSSD, SearchAWS, and others), theregister.co.uk, TheVarGuy, Tom’s Hardware, and zdnet.com, among many others.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    The above is an evolving collection of recommended reading including what I have on my physical and virtual bookshelves, as well as list of web sites, blogs and podcasts worth listening, reading or watching. Watch for more items to be added to the book shelf soon, and if you have a suggested recommendation, add it to the comments below.

    By the way, if you have not heard, its #Blogtober, check out some of the other blogs and posts occurring during October here as part of your recommended reading list.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    PCIe Fundamentals Server Storage I/O Network Essentials

    Updated 8/31/19

    PCIe Fundamentals Server Storage I/O Network Essentials

    PCIe fundamentals data infrastructure trends

    This piece looks at PCIe Fundamentals topics for server, storage, I/O network data infrastructure environments. Peripheral Computer Interconnect (PCI) Express aka PCIe is a Server, Storage, I/O networking fundamentals component. This post is an excerpt from chapter 4 (Chapter 4: Servers: Physical, Virtual, Cloud, and Containers) of my new book Software Defined Data Infrastructure Essentials – Cloud, Converged and Virtual Fundamental Server Storage I/O Tradecraft (CRC Press 2017) Available via Amazon.com and other global venues. In this post, we look various PCIe fundamentals to learn and expand or refresh your server, storage, and I/O and networking tradecraft skills experience.

    PCIe fundamentals Server Storage I/O Fundamentals

    PCIe fundamental common server I/O component

    Common to all servers is some form of a main system board, which can range from a few square meters in supercomputers, data center rack, tower, and micro towers converged or standalone, to small Intel NUC (Next Unit of Compute), MSI and Kepler-47 footprint, or Raspberry Pi-type desktop servers and laptops. Likewise, PCIe is commonly found in storage and networking systems, appliances among other devices.

    For example, a blade server will have multiple server blades or modules, each with its motherboard, which shares a common back plane for connectivity. Another variation is a large server such as an IBM “Z” mainframe, Cray, or another supercomputer that consists of many specialized boards that function similar to a smaller-sized motherboard on a larger scale.

    Some motherboards also have mezzanine or daughter boards for attachment of additional I/O networking or specialized devices. The following figure shows a generic example of a two-socket, with eight-memory-channel-type server architecture.

    PCIe fundamentals SDDC, SDI, SDDI Server fundamentals
    Generic computer server hardware architecture. Source: Software Defined Data Infrastructure Essentials (CRC Press 2017)

    The above figure shows several PCIe, USB, SAS, SATA, 10 GbE LAN, and other I/O ports. Different servers will have various combinations of processor, and Dual Inline Memory Module (DIMM) Dynamic RAM (DRAM) sockets along with other features. What will also vary are the type and some I/O and storage expansion ports, power and cooling, along with management tools or included software.

    PCIe, Including Mini-PCIe, NVMe, U.2, M.2, and GPU

    At the heart of many servers I/O and connectivity solutions are the PCIe industry-standard interface (see PCIsig.com). PCIe is used to communicate with CPUs and the outside world of I/O networking devices. The importance of a faster and more efficient PCIe bus is to support more data moving in and out of servers while accessing fast external networks and storage.

    For example, a server with a 40-GbE NIC or adapter would have to have a PCIe port capable of 5 GB per second. If multiple 40-GbE ports are attached to a server, you can see where the need for faster PCIe interfaces come into play.

    As more VM are consolidated onto PM, as applications place more performance demand either regarding bandwidth or activity (IOPS, frames, or packets) per second, more 10-GbE adapters will be needed until the price of 40-GbE (also 25, 50 or 100 Gbe) becomes affordable. It is not if, but rather when you will grow into the performance needs on either a bandwidth/throughput basis or to support more activity and lower latency per interface.

    PCIe is a serial interface specified for how servers communicate between CPUs, memory, and motherboard-mounted as well as AiC devices. This communication includes support attachment of onboard and host bus adapter (HBA) server storage I/O networking devices such as Ethernet, Fibre Channel, InfiniBand, RapidIO, NVMe (cards, drives, and fabrics), SAS, and SATA, among other interfaces.

    In addition to supporting attachment of traditional LAN, SAN, MAN, and WAN devices, PCIe is also used for attaching GPU and video cards to servers. Traditionally, PCIe has been focused on being used inside of a given server chassis. Today, however, PCIe is being deployed on servers spanning nodes in dual, quad, or CiB, CI, and HCI or Software Defined Storage (SDS) deployments. Another variation of PCIe today is that multiple servers in the same rack or proximity can attach to shared devices such as storage via PCIe switches.

    PCIe components (hardware and software) include:

    • Hardware chipsets, cabling, connectors, endpoints, and adapters
    • Root complex and switches, risers, extenders, retimers, and repeaters
    • Software drivers, BIOS, and management tools
    • HBAs, RAID, SSD, drives, GPU, and other AiC devices
    • Mezzanine, mini-PCIe, M.2, NVMe U.2 (8639 drive form factor)

    There are many different implementations of PCIe, corresponding to generations representing speed improvements as well as physical packing options. PCIe can be deployed in various topologies, including a traditional model where an AiC such as GbE or Fibre Channel HBA connects the server to a network or storage device.

    Another variation is for a server to connect to a PCIe switch, or in a shared PCIe configuration between two or more servers. In addition to different generations and topologies, there are also various PCIe form factors and physical connectors (see the following figure), ranging from AiC of various length and height, as well as M.2 small-form-factor devices and U.2 (8639) drive form-factor device for NVMe, among others.

    Note that the presence of M.2 does not guarantee PCIe NVMe, as it also supports SATA.

    Likewise, different NVMe devices run at various PCIe speeds based on the number of lanes. For example, in the following figure, the U.2 (8639) device (looks like a SAS device) shown is a PCIe x4.

    SDDC, SDI, SDDI PCIe NVMe U.2 8639 drive fundamentals
    PCIe devices NVMe U.2, M.2, and NVMe AiC. (Source: StorageIO Labs.)

    PCIe leverages multiple serial unidirectional point-to-point links, known as lanes, compared to traditional PCI, which used a parallel bus design. PCIe interfaces can have one (x1), four (x4), eight (x8), sixteen (x16), or thirty-two (x32) lanes for data movement. Those PCIe lanes can be full-duplex, meaning data is sent and received at the same time, providing improved effective performance.

    PCIe cards are upward-compatible, meaning that an x4 can work in an x8, an x8 in an x16, and so forth. Note, however, that the cards will not perform any faster than their specified speed; an x4 in an x8 slot will only run at x8. PCIe cards can also have single, dual, or multiple external ports and interfaces. Also, note that there are still some motherboards with legacy PCI slots that are not interoperable with PCIe cards and vice versa.

    Note that PCIe cards and slots can be mechanically x1, x4, x8, x16, or x32, yet electrically (or signal) wired to a slower speed, based on the type and capabilities of the processor sockets and corresponding chipsets being used. For example, you can have a PCIe x16 slot (mechanical) that is wired for x8, which means it will only run at x8 speed.

    In addition to the differences between electrical and mechanical slots, also pay attention to what generation the PCIe slots are, such as Gen 2 or Gen 3 or higher. Also, some motherboards or servers will advertise multiple PCIe slots, but those are only active with a second or additional processor socket occupied by a CPU. For example, a PCIe card that has dual x4 external PCIe ports requiring full PCIe bandwidth will need at least PCIe x8 attachment in the server slot. In other words, for full performance, the external ports on a PCIe card or device need to match the external electrical and mechanical card type and vice versa.

    Recall big “B” as in Bytes vs. little “b” as in bits; for example, a PCIe Gen 3 x4 electrical could provide up to 4 GB/s bandwidth (your mileage and performance will vary), which translates to 8 × 4 GB or 32 Gbits/s. In the following table below, there is a mix of Big “B” Bytes per second and small “b” bits per second.

    Each generation of PCIe has improved on the previous one by increasing the effective speed of the links. Some of the speed improvements have come from faster clock rates while implementing lower overhead encoding (e.g., from 8 b/10 b to 128 b/130 b).

    For example, PCIe Gen 3 raw bit or line rate is 8 GT/s or 8 Gbps or about 2 GBps by using a 128 b/130 b encoding scheme that is very efficient compared to PCIe Gen 2 or Gen 1, which used an 8 b/10 b encoding scheme. With 8 b/10 b, there is a 20% overhead vs. a 1.5% overhead with 128 b/130 b (i.e., of 130 bits sent, 128 bits contain data, and 2 bits are for overhead).

    PCIe Gen 1

    PCIe Gen 2

    PCIe Gen 3

    PCIe Gen 4

    PCIe Gen 5

    Raw bit rate

    2.5 GT/s

    5 GT/s

    8 GT/s

    16 GT/s

    32 GT/s

    Encoding

    8 b/10 b

    8 b/10 b

    128 b/130 b

    128 b/130 b

    128 b/130 b

    x1 Lane bandwidth

    2 Gb/s

    4 Gb/s

    8 Gb/s

    16 Gb/s

    32 Gb/s

    x1 Single lane (one-way)

    ~250 MB/s

    ~500 MB/s

    ~1 GB/s

    ~2 GB/s

    ~4GB/s

    x16 Full duplex (both ways)

    ~8 GB/s

    ~16 GB/s

    ~32 GB/s

    ~64 GB/s

    ~128 GB/s

    Above Table: PCIe Generation and Sample Lane Comparison

    Note that PCIe Gen 3 is the currently generally available shipping technology with PCIe Gen 4 appearing in the not so distant future, with PCIe Gen 5 in the wings appearing a few more years down the road.

    By contrast, older generations of Fibre Channel and Ethernet also used 8 b/10 b, having switched over to 64 b/66 b encoding with 10 Gb and higher. PCIe, like other serial interfaces and protocols, can support full-duplex mode, meaning that data can be sent and received concurrently.

    PCIe Bit Rate, Encoding, Giga Transfers, and Bandwidth

    Let’s clarify something about data transfer or movement both internal and external to a server. At the core of a server, there is data movement within the sockets of the processors and its cores, as well as between memory and other devices (internal and external). For example, the QPI bus is used for moving data between some Intel processors whose performance is specified in giga transfers (GT).

    PCIe is used for moving data between processors, memory, and other devices, including internal and external facing devices. Devices include host bus adapters (HBAs), host channel adapters (HCAs), converged network adapters (CNAs), network interface cards (NICs) or RAID cards, and others. PCIe performance is specified in multiple ways, given that it has a server processor focus which involves GT for raw bit rate as well as effective bandwidth per lane.

    Note to keep in perspective PCIe mechanical as well as electrical lanes in that a card or slot may be advertised as say x8 mechanical (e.g., its physical slot form factor) yet only be x4 electrical (how many of those lanes are used or enabled). Also in the case of an adapter that has two or more ports, if the device is advertised as x8 does that mean it is x8 per port or x4 per port with an x8 connection to the PCIe bus.

    Effective bandwidth per lane can be specified as half- or full-duplex (data moving in one or both directions for send and receive). Also, effective bandwidth can be specified as a single lane (x1), four lanes (x4), eight lanes (x8), sixteen lanes (x16), or 32 lanes (x32), as shown in the above table. The difference in speed or bits moved per second between the raw bit or line rate, and the effective bandwidth per lane in a single direction (i.e., half-duplex) is the encoding that is common to all serial data transmissions.

    When data gets transmitted, the serializer/deserializer, or serdes, convert the bytes into a bit stream via encoding. There are different types of encoding, ranging from 8 b/10 b to 64 b/66 b and 128 b//130 b, shown in the following table.

    Single 1542-byte frame

    64 × 1542-byte frames

    Encoding Scheme

    Overhead

    Data Bits

    Encoding Bits

    Bits Transmitted

    Data Bits

    Encoding Bits

    Bits Transferred

    8 b/10 b

    20%

    12,336

    3,084

    15,420

    789,504

    197,376

    986,880

    64 b/66 b

    3%

    12,336

    386

    12,738

    789,504

    24,672

    814,176

    128 b/130 b

    1.5%

    12,336

    194

    12,610

    789,504

    12,336

    801,840

    Above Table: Low-Level Serial Encoding Data Transmit Efficiency

    In these encoding schemes, the smaller number represents the amount of data being sent, and the difference is the overhead. Note that this is different yet related to what occurs at a higher level with the various network protocols such as TCP/IP (IP). With IP, there is a data payload plus addressing and other integrity and management features in a given packet or frame.

    The 8-b/10-b, 64-b/66-b or 128-b/130-b encoding is at the lower physical layer. Thus, a small change there has a big impact and benefit when optimized. Table 4.2 shows comparisons of various encoding schemes using the example of moving a single 1542-byte packet or frame, as well as sending (or receiving) 64 packets or frames that are 1542 bytes in size.

    Why 1542? That is a standard IP packet including data and protocol framing without using jumbo frames (MTU or maximum transmission units).

    What does this have to do with PCIe? GbE, 10-GbE, 40-GbE, and other physical interfaces that are used for moving TCP/IP packets and frames interface with servers via PCIe.

    This encoding is important as part of server storage I/O tradecraft regarding understanding the impact of performance and network or resource usage. It also means understanding why there are fewer bits per second of effective bandwidth (independent of compression or deduplication) vs. line rate in either half- or full-duplex mode.

    Another item to note is that looking at encoding such as the example given in the above table shows how a relatively small change at a large scale can have a big effective impact benefit. If the bits and bytes encoding efficiency and effectiveness scenario in Table 4.2 do not make sense, then try imagining 13 MINI Cooper automobiles each with eight people in it (yes, that would be a tight fit) end to end on the same road.

    Now imagine a large bus that takes up much less length on the road than the 13 MINI Coopers. The bus holds 128 people, who would still be crowded but nowhere near as cramped as eight people in a MINI, plus 24 additional people can be carried on the bus. That is an example of applying basic 8-b/10-b encoding (the MINI) vs. applying 128-b/130-b encoding (the bus) and is also similar to PCIe G3 and G4, which use 128-b/130-b encoding for data movement.

    PCIe Topologies

    The basic PCIe topology configuration has one or more devices attached to the root complex shown in the following figure via an AiC or onboard device connector. Examples of AiC and motherboard-mounted devices that attach to PCIe root include LAN or SAN HBA, networking, RAID, GPU, NVM or SSD, among others. At system start-up, the server initializes the PCIe bus and enumerates the devices found with their addresses.

    PCIe devices attach (shown in the following figure) to a bus that communicates with the root complex that connects with processor CPUs and memory. At the other end of a PCIe device is an end-point target, a PCIe switch that in turn has end-point targets attached. From a software standpoint, hypervisor or operating system device drivers communicate with the PCI devices that in turn send or receive data or perform other functions.

    SDDC, SDI, SDDI PCIe fundamentals
    Basic PCIe root complex with a PCIe switch or expander.

    Note that in addition to PCIe AiC such as HBAs, GPU, and NVM SSD, among others that install into PCIe slots, servers also have converged storage or disk drive enclosures that support a mix of SAS, SATA, and PCIe. These enclosure backplanes have a connector that attaches to a SAS or SATA onboard port, or a RAID card, as well as to a PCIe riser card or motherboard connector. Depending on what type of drive is installed in the connector, either the SAS, SATA, or NVMe (AiC, U.2, and M2) using PCIe communication paths are used.

    In addition to traditional and switched PCIe, using PCIe switches as well as nontransparent bridging (NTB), various other configurations can be deployed. These include server to server for clustering, failover, or device sharing as well as fabrics. Note that this also means that while traditionally found inside a server, PCIe can today use an extender, retimer, and repeaters extended across servers within a rack or cabinet.

    A nontransparent bridge (NTB) is a point-to-point connection between two PCIe-based systems that provide electrical isolation yet functions as a transport bridge between two different address domains. Hosts on either side of the NTB see their respective memory or I/O address space. The NTB presents an endpoint exposed to the local system where writes are mirrored to memory on the remote system to allow the systems to communicate and share devices using associated device drivers. For example, in the following figure, two servers, each with a unique PCIe root complex, address, and memory map, are shown using NTB to any communication between the systems while maintaining data integrity.

    SDDC, SDI, SDDI PCIe two server fundamentals
    PCIe dual server example using NTB along with switches.

    General PCIe considerations (slots and devices) include:

    • Power consumption (and heat dissipation)
    • Physical and software plug-and-play (good interoperability)
    • Drivers (in-the-box, built into the OS, or add-in)
    • BIOS, UEFI, and firmware being current versions
    • Power draw per card or adapters
    • Type of processor, socket, and support chip (if not an onboard processor)
    • Electrical signal (lanes) and mechanical form factor per slot
    • Nontransparent bridge and root port (RP)
    • PCI multi-root (MR), single-root (SR), and hot plug
    • PCIe expansion chassis (internal or external)
    • External PCIe shared storage

    Various operating system and hypervisor commands are available for viewing and managing PCIe devices. For example, on Linux, the “lspci” and “lshw–c pci” commands displays PCIe devices and associated information. On a VMware ESXi host, the “esxcli hardware pci list” command will show various PCIe devices and information, while on Microsoft Windows systems, “device manager” (GUI) or “devcon” (command line) will show similar information.

    Who Are Some PCIe Fundamentals Vendors and Service Providers

    While not an exhaustive list, here is a sampling of some vendors and service providers involved in various ways with PCIe from solutions to components to services to trade groups include Amphenol (connectors and cables), AWS (cloud data infrastructure services), Broadcom (PCIe components), Cisco (servers), DataOn (servers), Dell EMC (servers, storage, software), E8 (storage software), Excelero (storage software), HPE (storage, servers), Huawei (storage, servers), IBM, Intel (storage, servers, adapters), Keysight (test equipment and tools).

    Others include Lenovo (servers), Liqid (composable data infrastructure), Mellanox (server and storage adapters), Micron (storage devices), Microsemi (PCIe components), Microsoft (Cloud and Software including S2D), Molex (connectors, cables), NetApp, NVMexpress.org (NVM Express trade group organizations), Open Compute Project (server, storage, I/O network industry group), Oracle, PCISIG (PCIe industry trade group), Samsung (storage devices), ScaleMP (composable data infrastructure), Seagate (storage devices), SNIA (industry trade group), Supermicro (servers), Tidal (composable data infrastructure), Vantar (formerly known as HDS), VMware (Software including vSAN), and WD among others.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    PCIe fundamentals are resources for building legacy and software-defined data infrastructures (SDDI), software-defined infrastructures (SDI), data centers and other deployments from laptop to large scale, hyper-scale cloud service providers. Learn more about Servers: Physical, Virtual, Cloud, and Containers in chapter 4 of my new book Software Defined Data Infrastructure Essentials (CRC Press 2017) Available via Amazon.com and other global venues. Meanwhile, PCIe fundamentals continues to evolve as a Server, Storage, I/O networking fundamental component.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Introducing Windows Subsystem for Linux WSL Overview #blogtober

    Introducing Windows Subsystem for Linux WSL Overview #blogtober

    server storage I/O data infrastructure trends

    Updated 1/21/2018

    Introducing Windows Subsystem for Linux WSL and Overview. Microsoft has been increasing their support of Linux across Azure public cloud, Hyper-V and Linux Integration Services (LIS) and Windows platforms including Windows Subsystem for Linux (WSL) as well as Server along with Docker support.

    WSL installed with Ubuntu on Windows 10

    WSL with Ubuntu installed and open in a window on one of my Windows 10 systems.

    WSL is not a virtual machine (VM) running on Windows or Hyper-V, rather it is a subsystem that coexists next to win32 (read more about how it works and features, enhancements here). Once installed, WSL enables use of Linux bash shell along with familiar tools (find, grep, sed, awk, rsync among others) as well as services such as ssh, MySQL among others.

    What this all means is that if you work with both Windows and Linux, you can do so on the same desktop, laptop, server or system using your preferred commands. For example in one window you can be using Powershell or traditional Windows commands and tools, while in another window working with grep, find and other tools eliminating the need to install things such as wingrep among others.

    Installing WSL

    Depending on which release of Windows desktop or server you are running, there are a couple of different install paths. Since my Windows 10 is the most recent release (e.g. 1709) I was able to simply go to the Microsoft Windows Store via desktop, search for Windows Linux, select the distribution, install and launch. Microsoft has some useful information for installing WSL on different Windows version here, as well as for Windows Servers here.

    Get WSL from Windows Store

    Get WSL from Windows Store or more information and options here.

    Microsoft WSL install

    Click on Get the app

    Select which Linux for WSL to install

    Select desired WSL distribution

    SUSE linux for WSL

    Lests select SUSE as I already have Ubuntu installed (I have both)

    WSL installing SUSE

    SUSE WSL in the process of downloading. Note SUSE needs an access code (free) that you get from https://www.suse.com/subscriptions/sles/developer/ while waiting for the download and install is a good time to get that code.

    launching WSL on Windows 10

    Launching WSL with SUSE, you will be prompted to enter the code mentioned above, if you do not have a code, get it here from SUSE.

    completing install of WSL

    The WSL installation is very straight forward, enter the SUSE code (Ubuntu did not need a code). Note the Ubuntu and SUSE WSL task bar icons circled bottom center.

    Ubuntu and SUSE WSL on Windows 10

    Provide a username for accessing the WSL bash shell along with password, confirm how root and sudo to be applied and that is it. Serious, the install for WSL at least with Windows 10 1709 is that fast and easy. Note in the above image, I have WSL with Ubuntu open in a window on the left, WSL with SUSE on the right, and their taskbar icons bottom center.

    Windows WSL install error 0x8007007e

    Enable Windows Subsystem for Linux Feature on Windows

    If you get the above WSL error message 0x8007007e when installing WSL Ubuntu, SUSE or other shell distro, make sure to enable the Windows WSL feature if not already installed.

    Windows WSL install error fix

    One option is to install additional Windows features via settings or control panel. For example, Control panel -> Programs and features -> Turn Windows features on or off -> Check the box for Windows Subsystem for Linux

    Another option is to install Windows subsystem feature via Powershell for example.

    enable-windowsoptionalfeature -online  -featurename microsoft-windows-subsystem-linux

    Using WSL

    Once you have WSL installed, try something simple such as view your present directory:

    pwd

    Then look at the Windows C: drive location

    ls /mnt/c -al

    In case you did not notice the above, you can use Windows files and folders from the bash shell by placing /mnt in front of the device path. Note that you need to be case-sensitive such as User vs. user or Documents vs. documents.

    As a further example, I needed to change several .htm, .html, .php and .xml files on a Windows system whose contents had not yet changed from https://storageio.com to https://storageio.com. Instead of installing wingrep or some tools, using WSL such as with Ubuntu finding files can be done with grep such as:

    grep "https://storageio.com" /mnt/c/Users/*.xml
    

    And then making changes using find and sed such as:

    find /mnt/c/Users -name \*.xml -exec sed  -i "s,https://storageio.com,https://storageio.com,g" {} \;

    Note that not all Linux apps and tools can use file via /mnt in which case a solution is to create a symbolic link.

    For example:

    ln -s "/mnt/c/Users/Test1/Documents"  /home/Test1/Projects

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    If you primarily work on (or have a preference for) Linux systems and need to do some functions from development to the administration or other activity on a Windows system, Windows Subsystem for Linux (WSL) provides a bash shell to do familiar tasks. Likewise, if you are primarily a Windows person and need to brush up on your Linux skills, WSL can help. If you need to run Linux server applications or workloads, put those into a Docker container, Hyper-V instance or Azure VM.

    Overall I like WSL for what it is, a tool that eliminates the need of having to install several other tools to do common tasks, plus makes it easier to work across various Linux and Windows systems including bare metal, virtual and cloud-based. Now that you have been introduced to Windows Subsystems for Linux WSL and an overview including install as well as using, add it to your data infrastructure toolbox.

    By the way, if you have not heard, its #Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Fixing the Microsoft Windows 10 1709 post upgrade restart loop

    Fixing the Microsoft Windows 10 1709 post upgrade restart loop

    server storage I/O data infrastructure trends

    Updated 1/21/2018

    Recently I needed to upgrade one of my systems to Microsoft Windows 10 1709 (e.g. the September 2017) release that post upgrade resulted in Windows Explorer, desktop and taskbar going into an endless loop. For those not familiar with Windows 10 1709 learn more here, and here including on how to get the bits (e.g. software). Windows 10 1709 is a semi-annual channel (SAC) Microsoft is following to enable a faster cadence or pace of releases making new features available faster. Note that there is a Windows 10 1709 SAC, as well as Windows Server 2017 SAC (more on that here).

    All was well with the 1709 install on Windows 10 until post upgrade when I logged into my account on my laptop (Lenovo X1). Once logged in initially everything looked good until about 10 to 20 seconds later, the screen flickered, the desktop refreshed as did the taskbar. All was well for about another 10 to 20 seconds and again the desktop refreshed as did the taskbar. Trying to use the Windows key plus other keys was no success, likewise trying to use command prompt, Powershell or other tools was futile given how quick the refresh occurred.

    Powering off the system and rebooting seemed normal, until once logged in and again the desktop and taskbar reset in the same looping fashion. Once again did a shutdown and restart, logged in and the same result.

    The Safe Mode Fix

    Unless you can access a command prompt or Powershell with administrator privileges, boot into Windows Safe mode.

    The solution to the post Windows 10 1709 upgrade desktop and taskbar restart loop was to boot into safe mode and run the following three commands.

    sfc /scannow
    
    dism.exe /online /cleanup-image /scanhealth
    dism.exe /online /cleanup-image /restorehealth

    Before you can run the above commands, access Windows Safe Mode. Tip if your Windows 10 system presents a login screen, in the lower right corner select the Shutdown, Restart icon holding down the SHIFT key and select Restart. Your system should reboot presenting you with the following options, selecting Troubleshoot.

    Windows 10 Troubleshoot

    Next select Advanced options shown below.

    Windows 10 Advanced Repair Options

    Next select Startup Settings shown below. Note that this sequence of commands are also used for other troubleshooting scenarios including boot problems, restore image or to a previous protection point among other options.

    Windows 10 System Repair Options

    The following Startup Settings screen appears, select Restart to enter Safe Mode.

    Windows 10 Startup Settings

    Your system should then present the following options, select Safe Mode with Command Prompt (option 6).

    Windows Safe Mode Options

    Next your system should display a Command Prompt where the following three commands are run:

    sfc /scannow
    dism.exe /online /cleanup-image /scanhealth
    dism.exe /online /cleanup-image /restorehealth

    Windows Command Prompt in safe mode

    Exit, shutdown, reboot and all should be good.

    Some Tips and Recommendations

    Before any upgrade, make sure you have good backups to enable various recovery points if needed.

    If you have not done so recently, make sure you have system restore enabled, as well as underlying hypervisors or storage system snapshot.

    If you have bitlocker enabled, before you do any upgrade, make sure to have a copy of your keys handy if you need to use them.

    If you rely on PIN or fingerprint for login, make sure you have your real password handy.

    If you have not done so recently, make sure your secondary standby emergency access account is working, if you dont have one, create one.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    For those who are planning on upgrading to Windows 10 1709 or any system for that matter, being ready for the unexpected can help when the unexpected does occur. While the above is focused on Windows 10 1709 upgrade, the tips and tools including safe mode are used for many other situations. Keep in mind that if something can go wrong sometime, sometime it will go wrong, so be prepared. As for Microsoft Windows 10 1709 post upgrade, all is good and it is running well on my different systems including physical as well as virtual devices.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Cloud Conversations AWS Azure Service Maps via Microsoft

    Cloud Conversations AWS Azure Service Maps via Microsoft

    server storage I/O data infrastructure trends

    Updated 1/21/2018

    Microsoft has created an Amazon Web Service AWS Azure Service Map. The AWS Azure Service Map is a list created by Microsoft looks at corresponding services of both cloud providers.

    Azure AWS service map via Microsoft.com
    Image via Azure.Microsoft.com

    Note that this is an evolving work in progress from Microsoft and use it as a tool to help position the different services from Azure and AWS.

    Also note that not all features or services may not be available in different regions, visit Azure and AWS sites to see current availability.

    As with any comparison they are often dated the day they are posted hence this is a work in progress. If you are looking for another Microsoft created why Azure vs. AWS then check out this here. If you are looking for an AWS vs. Azure, do a simple Google (or Bing) search and watch all the various items appear, some sponsored, some not so sponsored among others.

    Whats In the Service Map

    The following AWS and Azure services are mapped:

    • Marketplace (e.g. where you select service offerings)
    • Compute (Virtual Machines instances, Containers, Virtual Private Servers, Serverless Microservices and Management)
    • Storage (Primary, Secondary, Archive, Premium SSD and HDD, Block, File, Object/Blobs, Tables, Queues, Import/Export, Bulk transfer, Backup, Data Protection, Disaster Recovery, Gateways)
    • Network & Content Delivery (Virtual networking, virtual private networks and virtual private cloud, domain name services (DNS), content delivery network (CDN), load balancing, direct connect, edge, alerts)
    • Database (Relational, SQL and NoSQL document and key value, caching, database migration)
    • Analytics and Big Data (data warehouse, data lake, data processing, real-time and batch, data orchestration, data platforms, analytics)
    • Intelligence and IoT (IoT hub and gateways, speech recognition, visualization, search, machine learning, AI)
    • Management and Monitoring (management, monitoring, advisor, DevOps)
    • Mobile Services (management, monitoring, administration)
    • Security, Identity and Access (Security, directory services, compliance, authorization, authentication, encryption, firewall
    • Developer Tools (workflow, messaging, email, API management, media trans coding, development tools, testing, DevOps)
    • Enterprise Integration (application integration, content management)

    Down load a PDF version of the service map from Microsoft here.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    On one hand this can and will likely be used as a comparison however use caution as both Azure and AWS services are rapidly evolving, adding new features, extending others. Likewise the service regions and site of data centers also continue to evolve thus use the above as a general guide or tool to help map what service offerings are similar between AWS and Azure.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Amazon Web Service AWS September 2017 Software Defined Data Infrastructure Updates

    Amazon Web Service AWS September 2017 Software Defined Data Infrastructure Updates

    server storage I/O data infrastructure trends

    Amazon Web Service AWS September 2017 Software Defined Data Infrasture Updates

    September was a busy month pertaining to software defined data infrastructure including cloud and related AWS announcements. One of the announcements included VMware partnering to deliver vSphere, vSAN and NSX data infrastructure components for creating software defined data centers (SDDC) also known as multi cloud, and hybrid cloud leveraging AWS elastic bare metal servers (read more here in a companion post). Unlike traditional partner software defined solutions that relied on AWS Elastic Cloud Compute (EC2) instances, VMware is being deployed using private bare metal AWS elastic servers.

    What this means is that VMware vSphere (e.g. ESXi) hypervisor, vCenter, software defined storage (vSAN), storage defined network (NSX) and associated vRealize tools are deployed on AWS data infrastructure that can be used for deploying hybrid software defined data centers (e.g. connecting to your existing VMware environment). Learn more about VMware on AWS here or click on the following image.

    VMware on AWS via Amazon.com

    Additional AWS Updates

    Amazon Web Services (AWS) updates include, coinciding with VMworld, the initial availability of VMware on AWS (using virtual private servers e.g. think along the lines of Lightsail, not EC2 instances) was announced. Amazon Web Services (AWS) continues its expansion into database and table services with Relational Data Services (RDS) including various engines (Amazon Auora,MariaDB, MySQL, Oracle, PostgreSQL,and SQL Server along with Database Migration Service (DMS). Note that these RDS are in addition to what you can install and run your self on Elastic Cloud Compute (EC2) virtual machine instances, Lambda serverless containers, or Lightsail Virtual Private Servers (VPS).

    AWS has published a guide to database testing on Amazon RDS for Oracle plotting latency and IOPs for OLTP workloads here using SLOB. If you are not familiar with SLOB (Silly Little Oracle Benchmark) here is a podcast with its creator Kevin Closson discussing database performance and related topics. Learn more about SLOB and step by step installation for AWS RDS Oracle here, and for those who are concerned or think that you can not run workloads to evaluate Oracle platforms, have a look at this here.

    EC2 enhancements include charging by the second (previous by the hour) for some EC2 instances (see details here including what is or is not currently available) which is a growing trend by private cloud vendors aligning with how serverless containers have been billed. New large memory EC2 instances that for example support up to 3,904GB of DDR4 RAM have been added by AWS. Other EC2 enhancements include updated network performance for some instances, OpenCL development environment to leverage AWS F1 FPGA enabled instances, along with new Elastic GPU enabled instances. Other server and network enhancements include Network Load Balancer for Elastic Load Balancer announced, as well as application load balancer now supports load balancing to IP address as targets for AWS and on premises (e.g. hybrid) resources.

    Other updates and announces include data protection backups to AWS via Commvault and AWS Storage Gateway VTL announced. IBM has announced their Spectrum Scale (e.g. formerly known as SONAS aka GPFS) Scale Out Storage solution for high performance compute (HPC) quick start on AWS. Additional AWS enhancements include new edge location in Boston and a third Seattle site, while Direct Connect sites have been added in Boston and Houston along with Canberra Australia. View more AWS announcements and enhancements here.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    What This All Means

    AWS continues to grow and expand, both in terms of number of services, also the extensiveness of them. Likewise AWS continues to add more regions and data center availability zones, enhanced connectivity, along with earlier mentioned service features. The partnership with VMware should enable enterprise organizations to move towards hybrid cloud data infrastructures, while giving AWS an additional reach into those data centers. Overall a good set of enhancements by AWS who continues to evolve their cloud and software defined data infrastructure portfolio of solution offerings.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Dell EMC VMware September 2017 Software Defined Data Infrastructure Updates

    Dell EMC VMware September 2017 Software Defined Data Infrastructure Updates

    server storage I/O data infrastructure trends

    Dell EMC VMware September 2017 Software Defined Data Infrastructure Updates

    vmworld 2017

    September was a busy month including VMworld in Las Vegas that featured many Dell EMC VMware (among other) software defined data infrastructure updates and announcements.

    A summary of September VMware (and partner) related announcements include:

    VMware on AWS via Amazon.com
    VMware and AWS via Amazon Web Services

    VMware and AWS

    Some of you might recall VMware earlier attempt at public cloud with vCloud Air service (see Server StorageIO lab test drive here) which has since been depreciated (e.g. retired). This new approach by VMware leverages the large global presence of AWS enabling customers to set up public or hybrid vSphere, vSAN and NSX based clouds, as well as software defined data centers (SDDC) and software defined data infrastructures (SDDI).

    VMware Cloud on AWS exists on a dedicated, single-tenant (unlike Elastic Cloud Compute (EC2) multi-tenant instances or VMs) that supports from 4 to 16 underlying host per cluster. Unlike EC2 virtual machine instances, VMware Cloud on AWS is delivered on elastic bare-metal (e.g. dedicated private servers aka DPS). Note AWS EC2 is more commonly known, AWS also has other options for server compute including Lambda micro services serverless containers, as well as Lightsail virtual private servers (VPS).

    Besides servers with storage optimized I/O featuring low latency NVMe accessed SSDs, and applicable underlying server I/O networking, VMware Cloud on AWS leverages the VMware software stack directly on underlying host servers (e.g. there is no virtualization nesting taking place). This means more robust performance should be expected like in your on premise VMware environment. VM workloads can move between your onsite VMware systems and VMware Cloud on AWS using various tools. The VMware Cloud on AWS is delivered and managed by VMware, including pricing. Learn more about VMware Cloud on AWS here, and here (VMware PDF) and here (VMware Hands On Lab aka HOL).

    Read more about AWS September news and related updates here in this StorageIOblog post.

    VMware PKS
    VMware and Pivotal PKS via VMware.com

    Pivotal Container Service (PKS) and Google Kubernetes Partnership

    During VMworld VMware, Pivotal and Google announced a partnership for enabling Kubernetes container management called PKS (Pivotal Container Service). Kubernetes is evolving as a popular open source container microservice serverless management orchestration platform that has roots within Google. What this means is that what is good for Google and others for managing containers, is now good for VMware and Pivotal. In related news, VMware has become a platinum sponsor of the Cloud Native Compute Foundation (CNCF). If you are not familiar with CNCF, add it to your vocabulary and learn more here at www.cncf.io.

    Other VMworld and September VMware related announcements

    Hyper converged data infrastructure provider Maxta has announced a VMware vSphere Escape Pod (parachute not included ;) ) to facilitate migration from ESXi based to Red Hat Linux hypervisor environments. IBM and VMware for cloud partnership, along with Dell EMC, IBM and VMware joint cloud solutions. White listing of VMware vSphere VMs for enhanced security combine with earlier announced capabilities.

    Note that both VMware with vSphere ESXi and Microsoft with Hyper-V (Windows and Azure based) are supporting various approaches for securing Virtual Machines (VMs) and the hosts they run on. These enhancements are moving beyond simply encrypting the VMDK or VHDX virtual disks the VMs reside in or use, as well as more than password, ssh and other security measures. For example Microsoft is adding support for host guarded fabrics (and machine hosts) as well as shielded VMs. Keep an eye on how both VMware and Microsoft extend the data protection and security capabilities for software defined data infrastructures for their solutions and services.

    Dell EMC Announcements

    At VMworld in September Dell EMC announcements included:

    • Hyper Converged Infrastructure (HCI) and Hybrid Cloud enhancements
    • Data Protection, Goverence and Management suite updates
    • XtremIO X2 all flash array (AFA) availability optimized for vSphere and VDI

    HCI and Hybrid Cloud enhancements include VxRail Appliance, VxRack SDDC (vSphere 6.5, vSAN 6.6, NSX 6.3) along with hybrid cloud platforms (Enterprise Hybrid Cloud and Native Hybrid Cloud) along with vSAN Ready Nodes (vSAN 6.6 and encryption) and VMware Ready System. Note that Dell EMC in addition to supporting VMware hybrid clouds also previously announced solutions for Microsoft Azure Stack back in May.

    Software Defined Data Infrastructure Essentials at VMworld Bookstore

    xxxx

    Software Defined Data Infrastructure Essentials (CRC Press) at VMworld bookstore

    My new book Software Defined Data Infrastructure Essentials (CRC Press) made its public debut in the VMware book store where I did a book signing event. You can get your copy of Software Defined Data Infrastructure Essentials which includes Software Defined Data Centers (SDDC) along with hybrid, multi-cloud, serverless, converged and related topics at Amazon among other venues. Learn more here.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    What This All Means

    A year ago at VMworld the initial conversations were started around what would become the VMware Cloud on AWS solution. Also a year ago besides VMware Integrated Containers (VIC) and some other pieces, the overall container and in particular related management story was a bit cloudy (pun intended). However, now the fog and cloud seem to be clearing with the PKS solution, along with details of VMware Cloud on AWS. Likewise vSphere, vSAN and NSX along with associated vRealize tools continue to evolve as well as customer deployment growing. All in all, VMware continues to evolve, let’s see how things progress now over the year until the next VMworld.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.

    Microsoft Azure September 2017 Software Defined Data Infrastructure Updates

    Microsoft Azure September 2017 Software Defined Data Infrastructure Updates

    server storage I/O data infrastructure trends

    Microsoft and Azure September 2017 Software Defined Data infrastructure Updates

    September was a busy month for data infrastructure topics as well as Microsoft in terms of new and enhanced technologies. Wrapping up September was Microsoft Ignite where Azure, Azure Stack, Windows, O365, AI, IoT, development tools announcements occurred, along with others from earlier in the month. As part of the September announcements, Microsoft released a new version of Windows server (e.g. 1709) that has a focus for enhanced container support. Note that if you have deployed Storage Spaces Direct (S2D) and are looking to upgrade to 1709, do your homework as there are some caveats that will cause you to wait for the next release. Note that there had been new storage related enhancements slated for the September update, however those were announced at Ignite to being pushed to the next semi-annual release. Learn more here and also here.

    Azure Files and NFS

    Microsoft made several Azure file storage related announcements and public previews during September including Native NFS based file sharing as companion to existing Azure Files, along with public preview of new Azure File Sync Service. Native NFS based file sharing (public preview announced, service is slated to be available in 2018) is a software defined storage deployment of NetApp OnTAP running on top of Azure data infrastructure including virtual machines and leverage Azure underlying storage.

    Note that the new native NFS is in addition to the earlier native Azure Files accessed via HTTP REST and SMB3 enabling sharing of files inside Azure public cloud, as well as accessible externally from Windows based and Linux platforms including on premises. Learn more about Azure Storage and Azure Files here.

    Azure File Sync (AFS)

    Azure File Sync AFS

    Azure File Sync (AFS) has now entered public preview. While users of Windows-based systems have been able to access and share Azure Files in the past, AFS is something different. I have used AFS for some time now during several private preview iterations having seen how it has evolved, along with how Microsoft listens incorporating feedback into the solution.

    Lets take a look at what is AFS, what it does, how it works, where and when to use it among other considerations. With AFS, different and independent systems can now synchronize file shares through Azure. Currently in the AFS preview Windows Server 2012 and 2016 are supported including bare metal, virtual, and cloud based. For example I have had bare metal, virtual (VMware), cloud (Azure and AWS) as part of participating in a file sync activities using AFS.

    Not to be confused with some other storage related AFS including Andrew File System among others, the new Microsoft Azure File Sync service enables files to be synchronized across different servers via Azure. This is different then the previous available Azure File Share service that enables files stored in Azure cloud storage to be accessed via Windows and Linux systems within Azure, as well as natively by Windows platforms outside of Azure. Likewise this is different from the recently announced Microsoft Azure native NFS file sharing serving service in partnership with NetApp (e.g. powered by OnTAP cloud).

    AFS can be used to synchronize across different on premise as well as cloud servers that can also function as cache. What this means is that for Windows work folders served via different on premise servers, those files can be synchronized across Azure to other locations. Besides providing a cache, cloud tiering and enterprise file sync share (EFSS) capabilities, AFS also has robust optimization for data movement to and from the cloud and across sites, along with management tools. Management tools including diagnostics, performance and activity monitoring among others.

    Check out the AFS preview including planning for an Azure File Sync (preview) deployment (Docs Microsoft), and for those who have Yammer accounts, here is the AFS preview group link.

    Microsoft Azure Blob Events via Microsoft

    Azure Blob Storage Tiering and Event Triggers

    Two other Azure storage features that are in public preview include blob tiering (for cold archiving) and event triggers for events. As their names imply, blob tiering enables automatic migration from active to cold inactive storage of dormant date. Event triggers are policies rules (code) that get executed when a blob is stored to do various functions or tasks. Here is an overview of blob events and a quick start from Microsoft here.

    Keep in mind that not all blob and object storage are the same, a good example is Microsoft Azure that has page, block and append blobs. Append blobs are similar to what you might be familiar with other services objects. Here is a Microsoft overview of various Azure blobs including what to use when.

    Project Honolulu and Windows Server Enhancements

    Microsoft has evolved from command prompt (e.g. early MSDOS) to GUI with Windows to command line extending into PowerShell that left some thinking there is no longer need for GUI. Even though Microsoft has extended its CLI with PowerShell spanning WIndows platforms and Azure, along with adding Linux command shell, there are those who still want or need a GUI. Project Honolulu is the effort to bring GUI based management back to Windows in a simplified way for what had been headless, and desktop less deployments (e.g. Nano, Server Core). Microsoft had Server Management Tools (SMT) accessible via the Azure Portal which has been discontinued.


    Project Honolulu Image via Microsoft.com

    This is where project Honolulu comes into play for managing Windows Server platforms. What this means is that for those who dont want to rely on or have a PowerShell dependency have an alternative option. Learn more about Project Honolulu here and here, including download the public preview here.

    Storage Spaces Direct (S2D) Kepler Appliance

    Data Infrastructure provider DataOn has announced a new turnkey Windows Server 2016 Storage Spaces Direct (S2D) powered Hyper-Converged Infrastructure (e.g. productization of project Kepler-47) solution with two node small form factor servers (partner with MSI). How small? Think suitcase or airplane roller board carry on luggage size.

    What this means is that you can get into the converged, hyper-converged software defined storage game with Windows-based servers supporting Hyper-V virtual machines (Windows and Linux) including hardware for around $10,000 USD (varies by configuration and other options).

    Azure and Microsoft Networking News

    Speaking of Microsoft Azure public cloud, ever wonder what the network that enables the service looks like and some of the software defined networking (SDN) along with network virtualization function (NFV) objectives are, have a look at this piece from over at Data Center Knowledge.

    In related Windows, Azure and other focus areas, Microsoft, Facebook and Telxius have completed the installation of a high-capacity subsea cable (network) to cross the atlantic ocean. Whats so interesting from a data infrastructure, cloud or legacy server storage I/O and data center focus perspective? The new network was built by the combined companies vs. in the past by a Telco provider consortium with the subsequent bandwidth sold or leased to others.

    This new network is also 4,000 miles long including in depths of 11,000 feet, supports with current optics 160 terabits (e.g. 20 TeraBytes) per second capable of supporting 71 million HD videos streamed simultaneous. To put things into perspective, some residential Fiber Optic services can operate best case up to 1 gigabit per second (line speed) and in an asymmetrical fashion (faster download than uploads). Granted there are some 10 Gbit based services out there more common with commercial than residential. Simply put, there is a large amount of bandwidth increased across the atlantic for Microsoft and Facebook to support growing demands.

    Where To Learn More

    Learn more about related technology, trends, tools, techniques, and tips with the following links.

    What This All Means

    Microsoft announced a new release of Windows Server at Ignite as part of its new semi-annual release cycle. This latest version of Windows server is optimized for containers. In addition to Windows server enhancements, Microsoft continues to extend Azure and related technologies for public, private and hybrid cloud as well as software defined data infrastructures.

    By the way, if you have not heard, its Blogtober, check out some of the other blogs and posts occurring during October here.

    Ok, nuff said, for now.
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO. All Rights Reserved.