Software Defined, Bulk, Cloud, Scale Out, Object Storage Fundamentals

March 7, 2017 – 10:15 pm

Cloud, Bulk, Scale-Out, Object Storage Fundamentals

data infrastructure sddc object storage fundamentals

Updated 1/21/2018

Welcome to the Cloud, Big Data, Software Defined, scale-out, Bulk and Object Storage Fundamentals page.

This page contains various resources, tips, essential topics pertaining to Software Defined, scale-out, Cloud, Bulk and Object Storage Fundamentals. Other resources pertaining to Software Defined, scale-out, Cloud, Bulk and Object Storage include:

  • Software Defined Data Infrastructure Essentials book (CRC Press)
  • Cloud, Software Defined, Scale-Out, Object Storage News Trends
  • There are various types of cloud, bulk and object storage including public services such as Amazon Web Services (AWS) Simple Storage Service (S3), Google, Microsoft Microsoft Azure, IBM Softlayer, Rackspace among many others. There are also solutions for hybrid and private deployment from Cisco, Cloudian, Fujifilm, DDN, Dell EMC, Fujitsu, HDS, HPE, IBM, NetApp, Noobaa, OpenStack, Quantum, Rackspace, Scality, Seagate, Spectra, Storpool, Suse, Swift and WD among others.

    Cloud products and services among others, along with associated data infrastructures including object storage, file systems, repositories and access methods are at the center of bulk, big data, big bandwidth and little data initiatives on a public, private, hybrid and community basis. After all, not everything is the same in cloud, virtual and traditional data centers or information factories from active data to in-active deep digital archiving.

    Cloud Object Storage Fundamentals Access and Architectures

    There are many facets to object storage including technology implementation, products, services, access and architectures for various applications and use scenarios.

    • Project or Account – Top of the hierarchy that can represent the owner or billing information for a service that where buckets are also attached.
    • Region – Location where data is stored that can include one or more data centers also known as Availability Zones.
    • AWS S3 Cross region replication
      Moving and Replicating Buckets/Containers, Subfolders and Objects

    • Availability Zone (AZ) or data center or server that implement durability and accessibility for availability within a region.
    • AWS Regions and Availability Zones AZs
      Example of Regions and Availability Zones (AZs)

    • Bucket or Container – Where objects or sub-folders containing objects are attached and accessed.
    • Object storage fundamentals sddc and cloud software defined

    • Sub-folder – While object storage can be located in a flat namespace for commonality and organization some solutions and service support the notion of sub-folder that resemble traditional directory hierarchy.
    • Object – Byte (or bit) stream that can be as small as one byte to as large as several Tbytes (some solutions and services support up to 5TByte sized objects). The object contains whatever data in any organization along with metadata. Different solutions and services support from a couple hundred KBytes of meta-data to Mbytes worth of meta-data. Regarding what can be stored in an object, anything from files, videos, images, virtual disks (VMDKs, VHDX), ZIP or tar files, backup and archive save sets, executable images or ISO’s, anything you want.
    • End-point – Where or what your software, application or tool and utilities along with gateways attach to for accessing buckets and objects.
    • object storage fundamentals, sddc and cloud storage example

      A common theme for object storage is flexibility, along with scaling (performance, availability, capacity, economics) along with extensibility without compromise or complexity. From those basics, there are many themes and variations from how data is protected (RAID or no RAID, hardware or software), deployed as a service or as tin wrapped software (an appliance), optimized for archiving or video serving or other applications.

      Many facets of cloud and object storage access

      One aspect of object and cloud storage is accessing or using object methods including application programming interfaces (API’s) vs. traditional block (LUN) or NAS (file) based approaches. Keep in mind that many object storage systems, software, and services support NAS file-based access including NFS, CIFS, HDFS  among others for compatibility and ease of use.

      Likewise various API’s can be found across different object solutions, software or services including Amazon Web Services (AWS) Simple Storage Service (S3) HTTP REST based, among others. Other API’s will vary by specific vendor or product however can include IOS (e.g. Apple iPhone and iPad), WebDav, FTP, JSON, XML, XAM, CDMI, SOAP, and DICOM among others. Another aspect of object and cloud storage are expanded  and dynamic metadata.

      While traditional file systems and NAS have simple or fixed metadata, object and cloud storage systems, services and solutions along with some scale-out file systems have ability to support user defined metadata. Specific systems, solutions, software, and services will vary on the amount of metadata that could range on the low-end from 100s of KBytes  to tens or more Mbytes.

      cloud object storage

      Where to learn more

      The following resources provide additional information about big data, bulk, software defined, cloud and object storage.

      Click here to view software defined, bulk, cloud and object storage trend news.

      StorageIO Founder Greg Schulz: File Services on Object Storage with HyperFile

      Via InfoStor: Object Storage Is In Your Future
      Via FujiFilm IT Summit: Software Defined Data Infrastructures (SDDI) and Hybrid Clouds
      Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
      Via InfoStor: Cloud Storage Concerns, Considerations and Trends
      Via InfoStor: Object Storage Is In Your Future
      Via Server StorageIO: April 2015 Newsletter Focus on Cloud and Object storage
      Via StorageIOblog: AWS S3 Cross Region Replication storage enhancements
      Cloud conversations: AWS EBS, Glacier and S3 overview
      AWS (Amazon) storage gateway, first, second and third impressions
      Cloud and Virtual Data Storage Networking (CRC Book)
      Via ChannelPartnersOnline: Selling Software-Defined Storage: Not All File Systems Are the Same
      Via ITProPortal: IBM kills off its first cloud storage platform
      Via ITBusinessEdge: Time to Rein in Cloud Storage
      Via SerchCloudStorge: Ctera Networks’ file-sharing services gain intelligent cache
      Via StorageIOblog: Who Will Be At Top Of Storage World Next Decade?

      Videos and podcasts at also available via Applie iTunes. Storage I/O iTunes pod cast image

      Human Face of Big Data
      Human Face of Big Data (Book review)

      Seven Databases in Seven weeks
      Seven Databases in Seven Weeks (Book review)

      Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

      Software Defined Data Infrastructure Essentials Book SDDC

      Wrap up and summary

      Object and cloud storage are in your future, the questions are when, where, with what and how among others.

      Watch for more content and links to be added here soon to this object storage center page including posts, presentations, pod casts, polls, perspectives along with services and product solutions profiles.

      Ok, nuff said, for now.


      Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

      All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2018 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.