Cloud Object Storage Fundamentals Access and Architectures
There are many facets to object storage including technology implementation, products, services, access and architectures for various applications and use scenarios.
- Project or Account – Top of the hierarchy that can represent the owner or billing information for a service that where buckets are also attached.
- Region – Location where data is stored that can include one or more data centers also known as Availability Zones.
- Availability Zone (AZ) or data center or server that implement durability and accessibility for availability within a region.
- Bucket or Container – Where objects or sub-folders containing objects are attached and accessed.
- Sub-folder – While object storage can be accessed in a flat name space for commonality and organization some solutions and service support the notion of sub-folder that resemble traditional directory hierarchy.
- Object – Byte (or bit) stream that can be as small as one byte to as large as several TBytes (some solutions and services support up to 5TByte sized objects). The object contains what ever data in any organization along with meta data. Different solutions and services support from a couple hundred KBytes of meta-data to MBytes worth of meta-data. In terms of what can be stored in an object, anything from files, videos, images, virtual disks (VMDKs, VHDX), ZIP or tar files, backup and archive save sets, executable images or ISO’s, anything you want.
- End-point – Where or what your software, application or tool and utilities along with gateways attach to for accessing buckets and objects.
A common theme for object storage is flexibility, along with scaling (performance, availability, capacity, economics) along with extensibility without compromise or complexity. From those basics, there are many themes and variations from how data is protected (RAID or no RAID, hardware or software), deployed as a service or as tin wrapped software (an appliance), optimized for archiving or video serving or other applications.
Many facets of cloud and object storage access
One aspect of object and cloud storage is accessing or using object methods including application programming interfaces (API’s) vs. traditional block (LUN) or NAS (file) based approaches.
Keep in mind that many object storage systems, software, and services support NAS file based access including NFS, CIFS, HDFS among others for compatibility and ease of use.
Likewise various API’s can be found across different object solutions, software or services including Amazon Web Services (AWS) Simple Storage Service (S3) HTTP REST based, among others.
Other API’s will vary by specific vendor or product however can include IOS (e.g. Apple iPhone and iPad), WebDav, FTP, JSON, XML, XAM, CDMI, SOAP, and DICOM among others.
Another aspect of object and cloud storage are expanded and dynamic metadata.
While traditional file systems and NAS have simple or fixed metadata, object and cloud storage systems, services and solutions along with some scale out file systems have ability to support user defined metadata.
Specific systems, solutions, software and services will vary on the amount of meta data that could range on the low-end from 100s of KBytes to tens or more MBytes.
Where to learn more about cloud and object storage
The following is a growing list of links and resources exploring existing, new and emerging object storage and related themes.
- Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
- Via InsideHPC: Deploying Hadoop on Lustre Storage: Lessons Learned and Best Practices
- Via Yahoo Engineering Blog: Yahoo Cloud Object Store – Object Storage at Exabyte Scale
- Via the Platform: Inside The Ceph Exascale Storage At Yahoo
- Va Swift Summit: Taking the Mystery out of Erasure Codes: A Swift Implementation
- Via GovTech: Storage Costs Cloud Police Cam Issue
- Via BostonHerald: Booting Up: Storage costs cloud police cam issue
- Via BostonHerald: DDN Introduces Two New Converged File Storage Platforms that Overcome Scale, Performance and Cost Hurdles of Large Scale Data Sets
- Via StorageIOblog: S3motion Buckets Containers Objects AWS S3 Cloud and EMCcode (with podcast)
- Via StorageIOblog: If focused on cost you might miss other cloud storage benefits
- Via ComputerWorld: Amazon offers network file storage in the cloud
- Via ComputerWeekly: HGST marries helium HDDs and Himalaya in object storage Active Archive
- Via GoogleCloudPlatform Blog: GCS Nearline Online storage at Offline price
- Via MarketWatch: Global Data Center Provider CyrusOne Announces Direct Connectivity to Google Cloud Platform
- Via PRNewsWire: Quantum Announces New Archive Solutions Designed To Reduce Unstructured Data Storage Costs
- Via Enterprise Storage Forum: Lustre buying guide
Moving and Replicating Buckets/Containers, Subfolders and Objects
- Via StorageIOblog: AWS S3 Cross Region Replication storage enhancements
- Via StreetInsider: Western Digital (WDC) to Acquire Object Storage Software Amplidata
- Via Enterprise Storage Forum: Dell Invests in Object Storage Startup Exablox
- Via Enterprise Storage Forum: Introducing s3motion (S3 and object access docker based appliance)
- Via Computerworld: Quantum enhances their cloud and object storage management withnew StorNext software versin
- 2014 IDC MarketScape perspectives and companion post here
- OpenStack Juno Offers Automated Hadoop Provisioning
- ScaleOut Software Releases Version 5.2 of Its In-Memory Computing Platform
- HP Inks Global Reseller Agreement With Object Storage Startup Scality
- NetApp Introduces Software-Defined Object Storage for the Hybrid Cloud
- NetApp intros software-defined object storage for hybrid cloud adoption via ZD
- Are more than five nines of availability really possible? via SearchCloudStorage
- How do primary storage clouds and cloud for backup differ? via SearchCloudStorage
- What should I consider when using SSD cloud? via SearchCloudStorage
- Given outages, are you concerned with the security of the cloud? via SearchCloudStorage
- Is the cost of cloud storage really cheaper than traditional storage? via SearchCloudStorage
Example of Regions and Availability Zones (AZs)
- Panzura Collaborates With Microsoft Azure To Transform Cloud Storage Into A Globally Distributed NAS via HostReview
- Tarmin Joins the Object Storage Alliance as a Founding Member via HeraldOnline
- Caringo Awarded Patent for Erasure Coding, Replication in Storage Cluster via ITBusinessNet
- Western Digital Capital Makes Strategic Investment in Amplidata via PRWeb
- Dovecot in for the long haul via ITwire
- Top Tips for Object Storage via InfoStor
- Object Storage Buying Guide via Enterprise Storage Forum
- Cloud Gateways 101via Enterprise Networking Planet
- HP Unveils Smaller, Less Costly Helion Cloud Platform via eWeek
- CA Technologies, AWS and Riverbed bring mainframe Cloud to big four banks via PCadvisor
- The Enterprise Cloud is Finally Moving Beyond Storage via CMS wire
- Microsoft 365’s Massive Cloud Storage Is ‘Icing on Cake’ to University’s Upgrade via Edtech
- Quantum Acquires Symform, Plans To Bring Cloud-Based Backups To Enterprise via CRN
- Basho Introduces Riak CS 1.5 via Digital Journal
- OpenStack Object Storage (Swift) Adds Storage Policies for Enterprise Users and Cloud Operators via BusinessWire
- EMC ViPR and Data Services (including Object storage) via SearchStorage
- Cloud Appliance Buying Guide via Enterprise Storage Forum
- Verizon, Amplidata and Cloud Object Storage via Datacenter Dynamics
- IBM drops SoftLayer cloud storage costs closer to its rivals – Via ZDNet
- How long will VSAN rebuilding take with large drives? via Duncan Epping (@DuncanYB) and his
Yellow-Bricks site. Note that VMware VSAN uses one of many different forms of object storage (e.g. its not S3 or Swift or CDMI) rather a means for storing VMware vSphere entities and items (e.g. various types of objects)
- Object storage: An architecture for the petabyte era? via Simon Robinson (@simonrob451) appearing at Computerweekly
- Western Digital/HGST announce alternative to Seagate Kinetic HDD for object storage via MarketWatch
- IBM stakes its claim in scale-out elastic storage for big data leveraging its GPFS scalable file system (e.g. used in SONAS and other products) CNN
- EMC Announces Elastic Cloud Storage Appliance via EMC
- OpenStack Spring 2014 (Atlanta) Summit Day 1 via OpenStack.org
- RedHat acquires InkTank provider of Ceph services via TechCrunch
- DataDirect Networks (DDN) WOS enhancements via ComputerWeekly
- Storage Decision Making for Fast, Big and Very Big Data Environments (Recorded for Playback: BrightTalk Webinar)
- Open Source and Cloud Storage – Enabling business, or a technology enabler? (Recorded for Playback: BrightTalk Webinar)
- Hybrid Clouds: Bridging the Gap Between Public and Private Environments (Recorded for Playback: BrightTalk Webinar)
- Book review: Rethinking Enterprise Storage and Hybrid Clouds by Microsoft/Storsimple and Marc Farley aka (@Gofarley (or what ever name he is currently using ;))
- Goodbye 2013, hello 2014 predictions present and future
- Enterprises should just ignore OpenStack via Virtualizedgeek aka Keith Townsend (@virtualizedgeek)
- OpenStack Swift Object Storage for the Cloud Grows via The Var Guy (@thevarguy) aka Christopher Tozzi
- For Seagate Kinetic to fly…it’s gonna take an ecosystem (Via @Seagate The Storage Effect Blog)
- Object-defined storage (it’s not about object storage) (Via Enrico Signoretti @Juku)
- Seagate Kinetic Cloud and Object Storage I/O platform
- Why You Should Consider Object Storage (Via 21CIT)
- Object Storage Is in Your Future, Even if You Use Files (Via 21CIT)
- The Right Storage Option Is Important for Big Data Success (Via FedTech)
- Hybrid cloud storage with an object storage back end(Via Marc Farley aka @MicroFarley at Microsoft Technet Blogs)
- OpenStack Suite download (Via Rackspace)
- EMC ViPR aimed at service providers, enterprise storage (Video)
- EMC ViPR virtual physical object and software defined storage
- EMC ViPR software defined object storage part II
- EMC ViPR software defined object storage part III
- The rise of cloud-integrated storage and EMC’s ViPR (Via Marc Farley of Microsoft aka @MicroFarley)
- ViPR: Under the covers (Via Chad Sakacc of EMC aka @sakacc)
- Introducing EMC ViPR (Via Chuck Hollis aka @Chuckhollis of EMC)
- EMC Tells their SDS story (e.g. ViPR) (Via the Solutions Architect aka Michael Letschin aka @mletschin)
- Object Paucity (Via Martin Glassborow aka @storagebod)
- Object Storage: You can call it NAS 2.0! (Via Enrico Signoretti aka @esignoretti)
- Object storage is the platform, gateways are the solution
(Via Enrico Signoretti aka @esignorettii)
- Files Vs. Objects: the difference is in the value (Via Enrico Signoretti aka @esignorettii)
- Conversation with Justin Stottlemyer of Shutterfly and object storage discussion (Pod cast)
- What-ever happened to Object Storage? (Via
Martin Glassborow aka @storagebod)
- Cloud conversations: AWS EBS, Glacier and S3 overview
- Ceph Day Amsterdam 2012 (Object and cloud storage)
- Ceph Day in Amsterdam and Sage Weil on Object Storage (Podcast)
- Mr. Backup (Curtis Preston) goes back to Ceph School (Video and Podcast)
- Cloud and Virtual Data Storage Networking trends (StorageExpo 2013)
- Cloud and object storage primer and industry trends presentation (PDF)
- AWS (Amazon) storage gateway, first, second and third impressions
- Getting Clear about Object Storage (Via Panasas)
- Object Storage and file system SNIA Tutorial