Check out additional Server StorageIO Tips & Articles here along with commentary in the news here

Welcome to the Object Storage Center page –

March 20, 2013 – 11:39 pm

Updated 2/23/2017

Welcome to the Cloud, Big Data, Software Defined and Object Storage page

Software defined, Bulk, Cloud and Object Storage exist to support expanding and diverse application data demands. There are various types of cloud and object storage including public provider service such as Amazon Web Services (AWS) Simple Storage Service (S3), Microsoft Azure, IBM Softlayer among many others. Likewise there are various software defined storage stacks and solutions including open source such as Ceph or OpenStack Swift, along with commercial offerings of those and many others such as Dell EMC ECS, HDS HCP, IBM Cleversafe and NetApp StorageGrid to name a few among others.

There is no such thing as an information recession with more data being generated, moved, processed, stored, preserved and served, granted there are economic realities. Likewise as a society our dependence on information being available for work or entertainment, from medical healthcare to social media and all points in between continues to increase (check out the Human Face of Big Data).

In addition, people and data are living longer, as well as getting larger (hence little data, big data and very big data). Cloud products and services along with associated object storage, file systems, repositories and access methods are at the center of big data, big bandwidth and little data initiatives on a public, private, hybrid and community basis. After all, not everything is the same in cloud, virtual and traditional data centers or information factories from active data to in-active deep digital archiving.

From openstack, s3 for active big data to deep digital archiving

Cloud Object Storage Fundamentals Access and Architectures

There are many facets to object storage including technology implementation, products, services, access and architectures for various applications and use scenarios.

  • Project or Account – Top of the hierarchy that can represent the owner or billing information for a service that where buckets are also attached.
  • Region – Location where data is stored that can include one or more data centers also known as Availability Zones.
  • Availability Zone (AZ) or data center or server that implement durability and accessibility for availability within a region.
  • Bucket or Container – Where objects or sub-folders containing objects are attached and accessed.

Object storage and cloud software defined

  • Sub-folder – While object storage can be located in a flat namespace for commonality and organization some solutions and service support the notion of sub-folder that resemble traditional directory hierarchy.
  • Object – Byte (or bit) stream that can be as small as one byte to as large as several Tbytes (some solutions and services support up to 5TByte sized objects). The object contains whatever data in any organization along with metadata. Different solutions and services support from a couple hundred KBytes of meta-data to Mbytes worth of meta-data. Regarding what can be stored in an object, anything from files, videos, images, virtual disks (VMDKs, VHDX), ZIP or tar files, backup and archive save sets, executable images or ISO’s, anything you want.
  • End-point – Where or what your software, application or tool and utilities along with gateways attach to for accessing buckets and objects.

object and cloud storage example

A common theme for object storage is flexibility, along with scaling (performance, availability, capacity, economics) along with extensibility without compromise or complexity. From those basics, there are many themes and variations from how data is protected (RAID or no RAID, hardware or software), deployed as a service or as tin wrapped software (an appliance), optimized for archiving or video serving or other applications.

Many facets of cloud and object storage access

One aspect of object and cloud storage is accessing or using object methods including application programming interfaces (API’s) vs. traditional block (LUN) or NAS (file) based approaches. Keep in mind that many object storage systems, software, and services support NAS file-based access including NFS, CIFS, HDFS  among others for compatibility and ease of use.

Likewise various API’s can be found across different object solutions, software or services including Amazon Web Services (AWS) Simple Storage Service (S3) HTTP REST based, among others. Other API’s will vary by specific vendor or product however can include IOS (e.g. Apple iPhone and iPad), WebDav, FTP, JSON, XML, XAM, CDMI, SOAP, and DICOM among others. Another aspect of object and cloud storage are expanded  and dynamic metadata.

While traditional file systems and NAS have simple or fixed metadata, object and cloud storage systems, services and solutions along with some scale-out file systems have ability to support user defined metadata. Specific systems, solutions, software, and services will vary on the amount of metadata that could range on the low-end from 100s of KBytes  to tens or more Mbytes.

Where to learn more about cloud and object storage

The following is a growing list of links and resources exploring existing, new and emerging object storage and related themes.

Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
Via InfoStor: Cloud Storage Concerns, Considerations and Trends
Via SearchCloudStorage: Open source object storage startup OpenIO adds hardware
Via InfoStor: Comments on Cloud Storage Pros and Cons
Via CRN: StorageCraft Acquires Exablox
Via TVtechglobal: Signiant CEO: The media industry isnt just one monolithic thing
Via Computerweekly: Using the cloud as a storage tier
Via SearchCloudStorage: OpenIO joins object storage cloud scrum
Via CRN: Dell Partners With Scality On High-Density Object Storage
Via BroadwayWorld:
The World’s Smallest Object Storage Solution, ‘Ceph on Olive’, is Released

Via AppDeveloperMagazine:
Via ZDnet: Micron Accelerated Solutions Launched with VMware, Supermicro and Nexenta
Via SmartDataCollective: Why You Should Be Securing Big Data In The Cloud?
Via Geekwire: Feds certify Amazon and Microsoft clouds to handle sensitive government data

Via NewsFactor: SoftNAS Introduces File Services for NetApp StorageGRID
Via Redmondmag: AWS Cloud Storage Service Turns 10
Via DropBox Blog: Scaling to exabytes and beyond
Via Computerworld: Dropbox quits Amazon cloud, takes back 500 PB of data
Via HPCwire: Seagate Sets Sights on Broader HPC Market
Via Why Samsung’s Joyent buyout could frame its future
Via GeekyGadgets: Google Drive Cloud Storage Is Now A Little Easier To Organise Thanks To New Tools
Via ZDnet: Google doubles Cloud Compute local SSD capacity: Now it’s 3TB per VM
Via AndroidPolice: Barracuda Is Shutting Down The Copy Cloud Storage Service On May 1st
Via StorageSoup: EMC expands Elastic Cloud (object) Storage
Via Cloudo Review: Access Multiple Cloud Apps from a Unified Interface
Via CyberSecurity: Cybersecurity and Accessible Storage Trends
Via eWeek: Cloud Storage Spending Poised for Increase in 2016
Via InfoStor: Object Storage Is In Your Future
Via BioScienceTechnology: Six Reasons to Add Object Storage to Your Genomics Lexicon
Via TechCrunch: Backblaze’s Low-Cost Cloud Storage Service Comes Out Of Beta
Via Emergency Management Magazine: Cloud Storage for Camera Data?
Via CloudComputingAdmin: Cloud Storage Decision Making
COSbench for cloud and object storage (Various): White Paper (CMU), PDF overview presentation, Get the bits (software) from Github
Via AWS Blogs: AWS Storage Update – New Lower Cost S3 Storage Option & Glacier Price Reduction
Via Techcrunch: IBM Acquires Cleversafe To Accelerate Hybrid Cloud Storage Strategy
Via Techcrunch: Developers Can Now Import Data To Google Cloud Storage By Mailing In Their Hard Drives, Tapes And USB Flash Drives
Via vCloud: VMware vCloud Air Object Storage
Via SearchStorage: SwiftStack adds new features to object storage software solution
Via TheVarGuy: Vendor consortium create Ceph advisory board
Via OpenStack: Whats new in OpenStack Liberty
Via Microsoft Azure: File access cloud storage in addition to blobs and object storage
Via AWS: AWS Storage Update – New Lower Cost S3 Storage Option & Glacier Price Reduction
Via VMware Blogs: Showcasing VMware vCloud Air Object Storage powered by EMC at EMC World!
Via Venturebeat: Backblaze launches B2 object storage service at 1/4 of the cost of Amazon S3
Via Techcrunch: Wuala, The Secure P2P Cloud Storage Service Acquired By LaCie, To Shut Down
Via ChannelworldIN: 13 firms join to open up Ethernet, object-storage to everyone
Via ITportal: The elephant in the room: The trust deficit issue with public clouds
Via VCpost: Dropbox latest update: Cloud storage service provider now lets you drag and drop URLs with files
Via NetworkAsia: HP’s new converged appliance offers a path to hybrid cloud
Via Techtimes: Google Files For Patent To Search Your Desktop Files, Apps And Cloud Storage
Via CloudTweaks: Questions To Ask Every Cloud Storage Provider
Via MarketWatch: Linux Foundation Brings Together Industry Leaders to Advance Cloud Object Storage Technologies
Via CBR: New Kinetic platform targets open source object storage
Via Techrunch: Seagate, Toshiba, Scality And Others Launch Kinetic Open Storage Platform For Smart, Low-Overhead Cloud Object Storage
Via StorageIOblog: July 2015 Newsletter Focus on Content Solution Platforms
Via ITnewsline:
Hitachi Data Systems Announces New Enhancements to the Hitachi Content Platform Portfolio

Via ITbusinessnet: Caringo Provides Only Software-Based Web-Scale Object Storage for Symantec Enterprise Vault
Via ITbusinessnet: Caringo Provides Only Software-Based Web-Scale Object Storage for Symantec Enterprise Vault
Via TopTechNews: Dropbox Adds USB Keys To Boost Security
Via ZDnet: SUSE brings its management software to the cloud
Via Forbes: Jumptuit Roadmap Video: A New Search Bar For Your Data At Facebook, Google and Dropbo
Via Forbes: Panzura Global File System — Disrupting the Storage Industry
Via TalkinCloud: Barracuda Launches Cloud Storage and Share Service CudaDrive
Via ITnews: Researchers debut rival to AWS cloud storage
Via HP launches new scale-out and scale-up Compute portfolios
Via Server StorageIO: May / June 2015 Newsletter Focus on server storage packaging
Via Server StorageIO: April 2015 Newsletter Focus on Cloud and Object storage
Via At NAB, Vendors and Users Both Got Serious About 4K Workflow
Via The unofficial guide to OpenStack Summit Vancouver
Via InsideHPC: Deploying Hadoop on Lustre Storage: Lessons Learned and Best Practices
Via Yahoo Engineering Blog: Yahoo Cloud Object Store – Object Storage at Exabyte Scale
Via the Platform: Inside The Ceph Exascale Storage At Yahoo
Va Swift Summit: Taking the Mystery out of Erasure Codes: A Swift Implementation
Via GovTech: Storage Costs Cloud Police Cam Issue
Via BostonHerald: Booting Up: Storage costs cloud police cam issue

Industry trends

openstack storage


Cloud and object storage book

Via StorageIOblog: S3motion Buckets Containers Objects AWS S3 Cloud and EMCcode (with podcast)

S3motion example

Via StorageIOblog: If focused on cost you might miss other cloud storage benefits
Via ComputerWorld: Amazon offers network file storage in the cloud
Via ComputerWeekly: HGST marries helium HDDs and Himalaya in object storage Active Archive
Via GoogleCloudPlatform Blog: GCS Nearline Online storage at Offline price
Via MarketWatch: Global Data Center Provider CyrusOne Announces Direct Connectivity to Google Cloud Platform
Via PRNewsWire: Quantum Announces New Archive Solutions Designed To Reduce Unstructured Data Storage Costs
Via Enterprise Storage Forum: Lustre buying guide

AWS S3 Cross region replication
Moving and Replicating Buckets/Containers, Subfolders and Objects

Via StorageIOblog: AWS S3 Cross Region Replication storage enhancements
Via StreetInsider: Western Digital (WDC) to Acquire Object Storage Software Amplidata
Via Enterprise Storage Forum: Dell Invests in Object Storage Startup Exablox
Via Enterprise Storage Forum: Introducing s3motion (S3 and object access docker based appliance)
Via Computerworld: Quantum enhances their cloud and object storage management with new StorNext software versin
2014 IDC MarketScape perspectives and companion post here
OpenStack Juno Offers Automated Hadoop Provisioning
ScaleOut Software Releases Version 5.2 of Its In-Memory Computing Platform
HP Inks Global Reseller Agreement With Object Storage Startup Scality
NetApp Introduces Software-Defined Object Storage for the Hybrid Cloud
NetApp intros software-defined object storage for hybrid cloud adoption via ZD
Are more than five nines of availability really possible? via SearchCloudStorage
How do primary storage clouds and cloud for backup differ? via SearchCloudStorage
What should I consider when using SSD cloud? via SearchCloudStorage
Given outages, are you concerned with the security of the cloud? via SearchCloudStorage
Is the cost of cloud storage really cheaper than traditional storage? via SearchCloudStorage

AWS Regions and Availability Zones AZs
Example of Regions and Availability Zones (AZs)

Panzura Collaborates With Microsoft Azure To Transform Cloud Storage Into A Globally Distributed NAS via HostReview
Caringo Awarded Patent for Erasure Coding, Replication in Storage Cluster via ITBusinessNet
Western Digital Capital Makes Strategic Investment in Amplidata via PRWeb
Dovecot in for the long haul via ITwire
Top Tips for Object Storage via InfoStor
Object Storage Buying Guide via Enterprise Storage Forum
Cloud Gateways 101via Enterprise Networking Planet
HP Unveils Smaller, Less Costly Helion Cloud Platform via eWeek
CA Technologies, AWS and Riverbed bring mainframe Cloud to big four banks via PCadvisor
The Enterprise Cloud is Finally Moving Beyond Storage via CMS wire
Microsoft 365’s Massive Cloud Storage Is ‘Icing on Cake’ to University’s Upgrade via Edtech
Quantum Acquires Symform, Plans To Bring Cloud-Based Backups To Enterprise via CRN
OpenStack Object Storage (Swift) Adds Storage Policies for Enterprise Users and Cloud Operators via BusinessWire
EMC ViPR and Data Services (including Object storage) via SearchStorage
Cloud Appliance Buying Guide via Enterprise Storage Forum
Verizon, Amplidata and Cloud Object Storage via Datacenter Dynamics
IBM drops SoftLayer cloud storage costs closer to its rivals – Via ZDNet
How long will VSAN rebuilding take with large drives? via Duncan Epping (@DuncanYB) and his
Yellow-Bricks site. Note that VMware VSAN uses one of many different forms of object storage (e.g. its not S3 or Swift or CDMI) rather a means for storing VMware vSphere entities and items (e.g. various types of objects)
Object storage: An architecture for the petabyte era? via Simon Robinson (@simonrob451) appearing at Computerweekly
Western Digital/HGST announce alternative to Seagate Kinetic HDD for object storage via MarketWatch
EMC Announces Elastic Cloud Storage Appliance via EMC
OpenStack Spring 2014 (Atlanta) Summit Day 1 via
RedHat acquires InkTank provider of Ceph services via TechCrunch
DataDirect Networks (DDN) WOS enhancements via ComputerWeekly
Storage Decision Making for Fast, Big and Very Big Data Environments (Recorded for Playback: BrightTalk Webinar)
Open Source and Cloud Storage – Enabling business, or a technology enabler? (Recorded for Playback: BrightTalk Webinar)
Hybrid Clouds: Bridging the Gap Between Public and Private Environments (Recorded for Playback: BrightTalk Webinar)
Book review: Rethinking Enterprise Storage and Hybrid Clouds by Microsoft/Storsimple and Marc Farley aka (@Gofarley (or what ever name he is currently using ;))
Goodbye 2013, hello 2014 predictions present and future
Enterprises should just ignore OpenStack via Virtualizedgeek aka Keith Townsend (@virtualizedgeek)
OpenStack Swift Object Storage for the Cloud Grows via The Var Guy (@thevarguy) aka Christopher Tozzi
For Seagate Kinetic to fly…it’s gonna take an ecosystem (Via @Seagate The Storage Effect Blog)
Object-defined storage (it’s not about object storage) (Via Enrico Signoretti @Juku)
Seagate Kinetic Cloud and Object Storage I/O platform
Why You Should Consider Object Storage (Via 21CIT)
Object Storage Is in Your Future, Even if You Use Files (Via 21CIT)
The Right Storage Option Is Important for Big Data Success (Via FedTech)
Hybrid cloud storage with an object storage back end(Via Marc Farley aka @MicroFarley at Microsoft Technet Blogs)
OpenStack Suite download (Via Rackspace)
EMC ViPR aimed at service providers, enterprise storage (Video)
EMC ViPR virtual physical object and software defined storage
EMC ViPR software defined object storage part II
EMC ViPR software defined object storage part III
The rise of cloud-integrated storage and EMC’s ViPR (Via Marc Farley of Microsoft aka @MicroFarley)
ViPR: Under the covers (Via Chad Sakacc of EMC aka @sakacc)
Introducing EMC ViPR (Via Chuck Hollis aka @Chuckhollis of EMC)
EMC Tells their SDS story (e.g. ViPR) (Via the Solutions Architect aka Michael Letschin aka @mletschin)
Object Paucity (Via Martin Glassborow aka @storagebod)
Object Storage: You can call it NAS 2.0! (Via Enrico Signoretti aka @esignoretti)
Object storage is the platform, gateways are the solution (Via Enrico Signoretti aka @esignorettii)
Files Vs. Objects: the difference is in the value (Via Enrico Signoretti aka @esignorettii)
Conversation with Justin Stottlemyer of Shutterfly and object storage discussion (Pod cast)
What-ever happened to Object Storage? (Via Martin Glassborow aka @storagebod)
Cloud conversations: AWS EBS, Glacier and S3 overview
Ceph Day Amsterdam 2012 (Object and cloud storage)
Ceph Day in Amsterdam and Sage Weil on Object Storage (Podcast)
Mr. Backup (Curtis Preston) goes back to Ceph School (Video and Podcast)
Cloud and Virtual Data Storage Networking trends (StorageExpo 2013)
Cloud and object storage primer and industry trends presentation (PDF) AWS (Amazon) storage gateway, first, second and third impressions

Wrap up and summary

Object and cloud storage are in your future, the questions are when, where, with what and how among others.

Watch for more content and links to be added here soon to this object storage center page including posts, presentations, pod casts, polls, perspectives along with services and product solutions profiles.

Ok, nuff said (for now…).


Greg Schulz – Microsoft MVP Cloud and Data Center Management, vSAN and VMware vExpert. Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier) and twitter @storageio. Watch for the spring 2017 release of his new book "Software-Defined Data Infrastructure Essentials" (CRC Press).

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2017 Server StorageIO(R) and UnlimitedIO All Rights Reserved

  1. 1 Trackback(s)

  2. Jan 8, 2014: Goodbye 2013, hello 2014, predictions past, present and future | StorageIOblog

Post a Comment

Powered by Disqus