Data Protection Gumbo = Protect Preserve and Serve Information

Storage I/O trends

Data Protection Gumbo = Protect Preserve and Serve Information

Recently I was invited to be a guest on the podcast Data Protection Gumbo hosted by Demetrius Malbrough (@dmalbrough).

Data Protection Gumbo Podcast Description
Data Protection Gumbo is set up with the aim of expanding the awareness of anyone responsible for protecting mission critical data, by providing them with a mix of the latest news, data protection technologies, and interesting facts on topics in the Data Backup and Recovery industry.

Data Protection Gumbo Also available on

Protect Preserve and Serve Applications, Information and Data

Keep in mind that a fundamental role of Information Technology (IT) is to protect, preserve and serve business or organizations information assets including applications, configuration settings and data for use when or where needed.

Our conversation covers various aspects of data protection which has a focus of protect preserve and serve information, applications and data across different environments and customer segments. While we discuss enterprise and small medium business (SMB) data protection, we also talk about trends from Mobile to the cloud among many others tools, technologies and techniques.

Where to learn more

Learn more about data protection and related trends, tools and technologies via the following links:

Data Protection Gumbo Also available on

What this all means and wrap-up

Data protection is a broad topic that spans from logical and physical security to high availability (HA), disaster recovery (DR), business continuance (BC), business resiliency (BR), archiving (including life beyond compliance) along with various tools, technologies, techniques. Keeping with the theme of protect preserve and serve, data protection to be modernized needs to become and be seen as a business asset or enabler vs. an after thought or cost over-head topic. Also, keep in mind that only you can prevent data loss, are your restores ready for when you need them?

Check out Demetrius Data Protection Gumbo podcast, also check out his Linkedin Backup & Recovery Professionals group. Speaking of data protection, check out the www.storageioblog.com/data-protection-diaries-main/ page for more coverage of backup/restore, HA, BC, DR, archiving and restated themes.

Ok, nuff said, for now..

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

S3motion Buckets Containers Objects AWS S3 Cloud and EMCcode

Storage I/O trends

S3motion Buckets Containers Objects AWS S3 Cloud and EMCcode

It’s springtime in Kentucky and recently I had the opportunity to have a conversation with Kendrick Coleman to talk about S3motion, Buckets, Containers, Objects, AWS S3, Cloud and Object Storage, node.js, EMCcode and open source among other related topics which are available in a podcast here, or video here and available at StorageIO.tv.

In this Server StorageIO industry trends perspective podcast episode, @EMCcode (Part of EMC) developer advocate Kendrick Coleman (@KendrickColeman) joins me for a conversation. Our conversation spans spring-time in Kentucky (where Kendrick lives) which means Bourbon and horse racing as well as his blog (www.kendrickcoleman.com).

Btw, in the podcast I refer to Captain Obvious and Kendrick’s beard, for those not familiar with who or what @Captainobvious is that is made reference to, click here to learn more.


@Kendrickcoleman
& @Captainobvious

What about Clouds Object Storage Programming and other technical stuff?

Of course we also talk some tech including what is EMCcode, EMC Federation, Cloud Foundry, clouds, object storage, buckets, containers, objects, node.js, Docker, Openstack, AWS S3, micro services, and the S3motion tool that Kendrick developed.

Cloud and Object Storage Access
Click to view video

Kendrick explains the motivation behind S3motion along with trends in and around objects (including GET, PUT vs. traditional Read, Write) as well as programming among related topic themes and how context matters.

S3motion for AWS S3 Google and object storage
Click to listen to podcast

I have used S3motion for moving buckets, containers and objects around including between AWS S3, Google Cloud Storage (GCS) and Microsoft Azure as well as to/from local. S3motion is a good tool to have in your server storage I/O tool box for working with cloud and object storage along with others such as Cloudberry, S3fs, Cyberduck, S3 browser among many others.

You can get S3motion free from git hub here.

Amazon Web Services AWS

Where to learn more

Here are some links to learn more about AWS S3, Cloud and Object Storage along with related topics

Also available on

What this all means and wrap-up

Context matters when it comes to many things particular about objects as they can mean different things. Tools such as S3motion make it easy for moving your buckets or containers along with objects from one cloud storage system, solution or service to another. Also check out EMCcode to see what they are doing on different fronts from supporting new and greenfield development with Cloud Foundry and PaaS to Openstack to bridging current environments to the next generation of platforms. Also check out Kendricks blog site as he has a lot of good technical content as well as some other fun stuff to learn about. Look forward to having Kendrick on as a guest again soon to continue our conversations. In the meantime, check out S3motion to see how it can fit into your server storage I/O tool box.

Ok, nuff said, for now..

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Cloud Conversations: AWS EFS Elastic File System (Cloud NAS) First Preview Look

Storage I/O trends

Cloud Conversations: AWS EFS Elastic File System (Cloud NAS) First Preview Look

Amazon Web Services (AWS) recently announced (preview) new Elastic File System (EFS) providing Network File System (NFS) NAS (Network Attached Storage) capabilities for AWS Elastic Cloud Compute (EC2) instances. EFS AWS compliments other AWS storage offerings including Simple Storage Service (S3) along with Elastic Block Storage (EBS), Glacier and Relational Data Services (RDS) among others.

Ok, that’s a lot of buzzwords and acronyms so lets break this down a bit.

Amazon Web Services AWS

AWS EFS and Cloud Storage, Beyond Buzzword Bingo

  • EC2 – Instances exist in various Availability Zones (AZ’s) in different AWS Regions. Compute instance with various operating systems including Windows and Ubuntu among others that also can be pre-configured with applications such as SQL Server or web services among others. EC2 instances vary from low-cost to high-performance compute, memory, GPU, storage or general purposed optimized. For example, some EC2 instances rely solely on EBS, S3, RDS or other AWS storage offerings while others include on-board Solid State Disk (SSD) like DAS SSD found on traditional servers. EC2 instances on EBS volumes can be snapshot to S3 storage which in turn can be replicated to another region.
  • EBS – Scalable block accessible storage for EC2 instances that can be configured for performance or bulk storage, as well as for persistent images for EC2 instances (if you choose to configure your instance to be persistent)
  • EFS – New file (aka NAS) accessible storage service accessible from EC2 instances in various AZ’s in a given AWS region
  • Glacier – Cloud based near-line (or by some comparisons off-line) cold-storage archives.
  • RDS – Relational Database Services for SQL and other data repositories
  • S3 – Provides durable, scalable low-cost bulk (aka object) storage accessible from inside AWS as well as via externally. S3 can be used by EC2 instances for bulk durable storage as well as being used as a target for EBS snapshots.
  • Learn more about EC2, EBS, S3, Glacier, Regions, AZ’s and other AWS topics in this primer here

aws regions architecture

What is EFS

Implements NFS V4 (SNIA NFS V4 primer) providing network attached storage (NAS) meaning data sharing. AWS is indicating initial pricing for EFS at $0.30 per GByte per month. EFS is designed for storage and data sharing from multiple EC2 instances in different AZ’s in the same AWS region with scalability into the PBs.

What EFS is not

Currently it seems that EFS has an end-point inside AWS accessible via an EC2 instance like EBS. This appears to be like EBS where the storage service is accessible only to AWS EC2 instances unlike S3 which can be accessible from the out-side world as well as via EC2 instances.

Note however, that depending on how you configure your EC2 instance with different software, as well as configure a Virtual Private Cloud (VPC) and other settings, it is possible to have an application, software tool or operating system running on EC2 accessible from the outside world. For example, NAS software such as those from SoftNAS and NetApp among many others can be installed on an EC2 instance and with proper configuration, as well as being accessible to other EC2 instances, they can also be accessed from outside of AWS (with proper settings and security).

AWS EFS at this time is NFS version 4 based however does not support Windows SMB/CIFS, HDFS or other NAS access protocols. In addition AWS EFS is accessible from multiple AZ’s within a region. To share NAS data across regions some other software would be required.

EFS is not yet as of this writing released and AWS is currently accepting requests to join the EFS preview here.

Amazon Web Services AWS

Where to learn more

Here are some links to learn more about AWS S3 and related topics

What this all means and wrap-up

AWS continues to extend its cloud platform include both compute and storage offerings. EFS compliments EBS along with S3, Glacier and RDS. For many environments NFS support will be welcome while for others CIFS/SMB would be appreciated and others are starting to find that value in HDFS accessible NAS.

Overall I like this announcement and look forward to moving beyond the preview.

Ok, nuff said, for now..

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

March 2015 Server StorageIO Update Newsletter

 

 

Volume 15, Issue III

Hello and welcome to this March 2015 Server and StorageIO update newsletter. Here in the northern hemisphere at least by the calendar spring is here, weather wise winter continues to linger in some areas. March also means in the US college university sports tournaments with many focused on their NCAA men’s basketball championship brackets.

Besides various college championships, March also has a connection to back up and data protection. Thus this months newsletter has a focus on data protection, after all March 31 is World Backup Day which means it should also be World Restore test day!

Focus on Data Protection

Data protection including backup/restore, business continuance (BC), disaster recovery (DR), business resiliency (BR) and archiving across physical, virtual and cloud environments.

Data Protection Fundamentals

A reminder on the importance of data protection including backup, BC, DR and related technologies is to make sure they are occuring as planned. Also test your copies and remember the 4 3 2 1 rule or guide.

4 – Versions (different time intervals)
3 – Copies of critical data (including versions)
2 – Different media, devices or systems
1 – Off-site (cloud or elsewhere)

The above means having at least four (4) different versions from various points in time of your data. Having three (3) copies including various versions protects against one or more copies being corrupt or damaged. Placing those versions and copies on at least two (2) different storage systems, devices or media if something happens.

While it might be common sense, a bad April Fools recovery joke would be finding out all of your copies were on the same device which is damaged. That might seem obvious however sometimes the obvious needs to be stated. Also make sure that at least one (1) of your copies is off-site either on off-line media (tape, disk, ssd, optical) or cloud.

Take a few moments and to verify that your data protection strategy is being implemented and practiced as intended. Also test what is being copied including not only restore the data from cloud, disk, ssd or tape, also make sure you can actually read or use the data being protected. This means make sure that your security credentials including access certificates and decryption occur as expected.

Watch for more news, updates industry trends perspectives commentary, tips, articles and other information at Storageio.com, StorageIOblog.com, various partner venues as well as in future newsletters.

StorageIOblog posts

Data Protection Diaries
Are restores ready for World Backup Day?
In case you forgot or did not know, World Backup Day is March 31 2015 (@worldbackupday) so now is a good time to be ready. The only challenge that I have with the World Backup Day (view their site here) that has gone on for a few years know is that it is a good way to call out the importance of backing up or protecting data.
world backup day test your restore

However it’s also time to put more emphasis and focus on being able to make sure those backups or protection copies actually work.

By this I mean doing more than making sure that your data can be read from tape, disk, SSD or cloud service actually going a step further and verifying that restored data can actually be used (read, written, etc).

The problem, issue and challenges are simple, are your applications, systems and data protected as well as can you use those protection copies (e.g. backups, snapshots, replicas or archives) when as well as were needed? Read more here about World Backup Day and what I’m doing as well as various tips to be ready for successful recovery and avoid being an April 1st fool ;).

Cloud Conversations
AWS S3 Cross Region Replication
Amazon Web Services (AWS) announced several enhancements including a new Simple Storage Service (S3) cross-region replication of objects from a bucket (e.g. container) in one region to a bucket in another region.

AWS also recently enhanced Elastic Block Storage (EBS) increasing maximum performance and size of Provisioned IOPS (SSD) and General Purpose (SSD) volumes. EBS enhancements included ability to store up to 16 TBytes of data in a single volume and do 20,000 input/output operations per second (IOPS). Read more about EBS and other AWS server, storage I/O  enhancements here.
AWS regions and availability zones (AZ)
Example of some AWS Regions and AZs

AWS S3 buckets and objects are stored in a specific region designated by the customer or user (AWS S3, EBS, EC2, Glacier, Regions and Availability Zone primer can be found here). The challenge being addressed by AWS with S3 replication is being able to move data (e.g. objects) stored in AWS buckets in one region to another in a safe, secure, timely, automated, cost-effective way.

Continue reading more here about AWS S3 bucket and object replication feature along with related material.

Additional March StorageIOblog posts include:

Server Storage I/O performance (Image licensed from Shutterstock by StorageIO)

 

 

View other recent as well as past blog posts here

In This Issue

  • Industry Trends Perspectives News
  • Commentary in the news
  • Tips and Articles
  • StorageIOblog posts
  • Events and Webinars
  • Recommended Reading List
  • StorageIOblog posts
  • Server StorageIO Lab reports
  • Resources and Links

 

Industry News and Activity

Recent Industry news and activity

EMC sets up cloudfoundry Dojo
AWS S3, EBS IOPs and other updates
New backup/data protection vendor Rubrik
Google adds nearline Cloud Storage
AWS and Microsoft Cloud Price battle

View other recent and upcoming events here

StorageIO Commentary in the news

StorageIO news (image licensed for use from Shutterstock by StorageIO)
Recent Server StorageIO commentary and industry trends perspectives about news, activities and announcements.

Processor: Enterprise Backup Solution Tips
Processor: Failed & Old Drives
EnterpriseStorageForum: Disk Buying Guide
ChannelProNetwork: 2015 Tech and SSD
Processor: Detect & Avoid Drive Failures

View more trends comments here

StorageIO Tips and Articles

So you have a new storage device or system. How will you test or find its performance? Check out this quick-read tip on storage benchmark and testing fundamentals over at BizTech.

Keeping with this months theme of data protection including backup/restore, BC, DR, BR and archiving, here are some more tips. These tips span server storage I/O networking hardware, software, cloud, virtual, performance, data protection applications and related themes including:

  • Test your data restores, can you read and actually use the data? Is you data decrypted, proper security certificates applied?
  • Remember to back up or protect your security encryption keys, certificates and application settings!
  • Revisit what format your data is being saved in including how will you be able to use data saved to the cloud. Will you be able to do a restore to a cloud server or do you need to make sure a copy of your backup tools are on your cloud server instances?

Check out these resources and links on server storage I/O performance and benchmarking tools. View more tips and articles here

Various Industry Events

EMCworld – May 4-6 2015

Interop – April 29 2015 (Las Vegas)

Presenting Smart Shopping for Your Storage Strategy

NAB – April 14-15 2015

SNIA DSI Event – April 7-9

View other recent and upcoming events here

Webinars

December 11, 2014 – BrightTalk
Server & Storage I/O Performance

December 10, 2014 – BrightTalk
Server & Storage I/O Decision Making

December 9, 2014 – BrightTalk
Virtual Server and Storage Decision Making

December 3, 2014 – BrightTalk
Data Protection Modernization

Videos and Podcasts

StorageIO podcasts are also available via and at StorageIO.tv

From StorageIO Labs

Research, Reviews and Reports

Datadynamics StorageX
Datadynamics StorageX

More than a data mover migration tool, StorageX is a tool for adding management and automation around unstructured local and distributed NAS (NFS, CIFS, DFS) file data. Read more here.

View other StorageIO lab review reports here

Recommended Reading List

This is a new section being introduced in this edition of the Server StorageIO update mentioning various books, websites, blogs, articles, tips, tools, videos, podcasts along with other things I have found interesting and want to share with you.

    • Introducing s3motion (via EMCcode e.g. opensource) a tool for copying buckets and objects between public, private and hybrid clouds (e.g. AWS S3, GCS, Microsoft Azure and others) as well as object storage systems. This is a great tool which I have added to my server storage I/O cloud, virtual and physical toolbox. If you are not familiar with EMCcode check it out to learn more…
  • Running Hadoop on Ubuntu Linux (Series of tutorials) for those who want to get their hands dirty vs. using one of the All In One (AIO) appliances.
    • Yellow-bricks (Good blog focused on virtualization, VMware and other related themes) by Duncan Epping @duncanyb

Resources and Links

Check out these useful links and pages:
storageio.com/links
objectstoragecenter.com
storageioblog.com/data-protection-diaries-main/

storageperformance.us
thessdplace.com
storageio.com/raid
storageio.com/ssd

Enjoy this edition of the Server and StorageIO update newsletter and watch for new tips, articles, StorageIO lab report reviews, blog posts, videos and podcasts along with in the news commentary appearing soon.

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Cloud Conversations: AWS S3 Cross Region Replication storage enhancements

Storage I/O trends

Cloud Conversations: AWS S3 Cross Region Replication storage enhancements

Amazon Web Services (AWS) recently among other enhancements announced new Simple Storage Service (S3) cross-region replication of objects from a bucket (e.g. container) in one region to a bucket in another region. AWS also recently enhanced Elastic Block Storage (EBS) increasing maximum performance and size of Provisioned IOPS (SSD) and General Purpose (SSD) volumes. EBS enhancements included ability to store up to 16 TBytes of data in a single volume and do 20,000 input/output operations per second (IOPS). Read more about EBS and other recent AWS server, storage I/O and application enhancements here.

Amazon Web Services AWS

The Problem, Issue, Challenge, Opportunity and Need

The challenge is being able to move data (e.g. objects) stored in AWS buckets in one region to another in a safe, secure, timely, automated, cost-effective way.

Even though AWS has a global name-space, buckets and their objects (e.g. files, data, videos, images, bit and byte streams) are stored in a specific region designated by the customer or user (AWS S3, EBS, EC2, Glacier, Regions and Availability Zone primer can be found here).

aws regions architecture

Understanding the challenge and designing a strategy

The following diagram shows the challenge and how to copy or replicate objects in an S3 bucket in one region to a destination bucket in a different region. While objects can be copied or replicated without S3 cross-region replication, that involves essentially reading your objects pulling that data out via the internet and then writing to another place. The catch is that this can add extra costs, take time, consume network bandwidth and need extra tools (Cloudberry, Cyberduck, S3fuse, S3motion, S3browser, S3 tools (not AWS) and a long list of others).
aws cross region replication

What is AWS S3 Cross-region replication

Highlights of AWS S3 Cross-region replication include:

  • AWS S3 Cross region replication is as its name implies, replication of S3 objects from a bucket in one region to a destination bucket in another region.
  • S3 replication of new objects added to an existing or new bucket (note new objects get replicated)
  • Policy based replication tied into S3 versioning and life-cycle rules
  • Quick and easy to set up for use in a matter of minutes via S3 dashboard or other interfaces
  • Keeps region to region data replication and movement within AWS networks (potential cost advantage)

To activate, you simply enable versioning on a bucket, enable cross-region replication, indicate source bucket (or prefix of objects in bucket), specify destination region and target bucket name (or create one), then create or select an IAM (Identify Access Management) role and objects should be replicated.

  • Some AWS S3 cross-region replication things to keep in mind (e.g. considerations):
  • As with other forms of mirroring and replication if you add something on one side it gets replicated to other side
  • As with other forms of mirroring and replication if you deleted something from the other side it can be deleted on both (be careful and do some testing)
  • Keep costs in perspective as you still need to pay for your S3 storage at both locations as well as applicable internal data transfer and GET fees
  • Click here to see current AWS S3 fees for various regions

S3 Cross-region replication and alternative approaches

There are several regions around the world and up until today AWS customers could copy, sync or replicate S3 bucket contents between AWS regions manually (or via automation) using various tools such as Cloudberry, Cyberduck, S3browser and S3motion to name just a few as well as via various gateways and other technologies. Some of those tools and technologies are open-source or free, some are freemium and some are premium for a few that also vary by interface (some with GUI, others with CLI or APIs) including ability to mount an S3 bucket as a local network drive and use tools to sync or copy.

However a catch with the above mentioned tools (among others) and approaches is that to replicate your data (e.g. objects in a bucket) can involve other AWS S3 fees. For example reading data (e.g. a GET which has a fee) from one AWS region and then copying out to the internet has fees. Likewise when copying data into another AWS S3 region (e.g. a PUT which are free) there is also the cost of storage at the destination.

Storage I/O trends

AWS S3 cross-region hands on experience (first look)

For my first hands on (first look) experience with AWS cross-region replication today I enabled a bucket in the US Standard region (e.g. Northern Virginia) and created a new target destination bucket in the EU Ireland. Setup and configuration was very quick, literally just a few minutes with most of the time spent reading the text on the new AWS S3 dashboard properties configuration displays.

I selected an existing test bucket to replicate and noticed that nothing had replicated over to the other bucket until I realized that new objects would be replicated. Once some new objects were added to the source bucket within a matter of moments (e.g. few minutes) they appeared across the pond in my EU Ireland bucket. When I deleted those replicated objects from my EU Ireland bucket and switched back to my view of the source bucket in the US, those new objects were already deleted from the source. Yes, just like regular mirroring or replication, pay attention to how you have things configured (e.g. synchronized vs. contribute vs. echo of changes etc.).

While I was not able to do a solid quantifiable performance test, simply based on some quick copies and my network speed moving via S3 cross-region replication was faster than using something like s3motion with my server in the middle.

It also appears from some initial testing today that a benefit of AWS S3 cross-region replication (besides being bundled and part of AWS) is that some fees to pull data out of AWS and transfer out via the internet can be avoided.

Amazon Web Services AWS

Where to learn more

Here are some links to learn more about AWS S3 and related topics

What this all means and wrap-up

For those who are looking for a way to streamline replicating data (e.g. objects) from an AWS bucket in one region with a bucket in a different region you now have a new option. There are potential cost savings if that is your goal along with performance benefits in addition to using what ever might be working in your environment. Replicating objects provides a way of expanding your business continuance (BC), business resiliency (BR) and disaster recovery (DR) involving S3 across regions as well as a means for content cache or distribution among other possible uses.

Overall, I like this ability for moving S3 objects within AWS, however I will continue to use other tools such as S3motion and s3sfs for moving data in and out of AWS as well as among other public cloud serves and local resources.

Ok, nuff said, for now..

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Data Protection Diaries: Are your restores ready for World Backup Day 2015?

Data Protection Diaries: Are your restores ready for World Backup Day 2015?

This is part of an ongoing data protection diaries series of post about, well, cloud and data protection and what I’m doing pertaining to World Backup Day 2015 along with related topics.

In case you forgot or did not know, World Backup Day is March 31 2015 (@worldbackupday) so now is a good time to be ready. The only challenge that I have with the World Backup Day (view their site here) that has gone on for a few years know is that it is a good way to call out the importance of backing up or protecting data. However its time to also put more emphasis and focus on being able to make sure those backups or protection copies actually work.

By this I mean doing more than making sure that your data can be read from tape, disk, SSD or cloud service actually going a step further and verifying that restored data can actually be used (read, written, etc).

The Problem, Issue, Challenge, Opportunity and Need

The problem, issue and challenges are simple, are your applications, systems and data protected as well as can you use those protection copies (e.g. backups, snapshots, replicas or archives) when as well as were needed?

storage I/O data protection

The opportunity is simple, avoiding downtime or impact to your business or organization by being proactive.

Understanding the challenge and designing a strategy

The following is my preparation checklist for World Backup Data 2015 (e.g. March 31 2015) which includes what I need or want to protect, as well as some other things to be done including testing, verification, address (remediate or fix) known issues while identifying other areas for future enhancements. Thus perhaps like yours, data protection for my environment which includes physical, virtual along with cloud spanning servers to mobile devices is constantly evolving.

collect TPM metrics from SQL Server with hammerdb
My data protection preparation, checklist and to do list

Finding a solution

While I already have a strategy, plan and solution that encompasses different tools, technologies and techniques, they are also evolving. Part of the evolving is to improve while also exploring options to use new and old things in new ways as well as eat my down dog food or walk the talk vs. talk the talk. The following figure provides a representation of my environment that spans physical, virtual and clouds (more than one) and how different applications along with systems are protected against various threats or risks. Key is that not all applications and data are the same thus enabling them to be protected in different ways as well as over various intervals. Needless to say there is more to how, when, where and with what different applications and systems are protected in my environment than show, perhaps more on that in the future.

server storageio and unlimitedio data protection
Some of what my data protection involves for Server StorageIO

Taking action

What I’m doing is going through my checklist to verify and confirm the various items on the checklist as well as find areas for improvement which is actually an ongoing process.

Do I find things that need to be corrected?

Yup, in fact found something that while it was not a problem, identified a way to improve on a process that will once fully implemented enabler more flexibility both if a restoration is needed, as well as for general everyday use not to mention remove some complexity and cost.

Speaking of lessons learned, check this out that ties into why you want 4 3 2 1 based data protection strategies.

Storage I/O trends

Where to learn more

Here are some extra links to have a look at:

Data Protection Diaries
Cloud conversations: If focused on cost you might miss other cloud storage benefits
5 Tips for Factoring Software into Disaster Recovery Plans
Remote office backup, archiving and disaster recovery for networking pros
Cloud conversations: Gaining cloud confidence from insights into AWS outages (Part II)
Given outages, are you concerned with the security of the cloud?
Data Archiving: Life Beyond Compliance
My copies were corrupted: The 3-2-1 rule
Take a 4-3-2-1 approach to backing up data
Cloud and Virtual Data Storage Networks – Chapter 8 (CRC/Taylor and Francis)

What this all means and wrap-up

Be prepared, be proactive when it comes to data protection and business resiliency vs. simply relying reacting and recovering hoping that all will be ok (or works).

Take a few minutes (or longer) and test your data protection including backup to make sure that you can:

a) Verify that in fact they are working protecting applications and data in the way expected

b) Restore data to an alternate place (verify functionality as well as prevent a problem)

c) Actually use the data meaning it is decrypted, inflated (un-compressed, un-de duped) and security certificates along with ownership properties properly applied

d) Look at different versions or generations of protection copies if you need to go back further in time

e) Identify area of improvement or find and isolate problem issues in advance vs. finding out after the fact

Time to get back to work checking and verifying things as well as attending to some other items.

Ok, nuff said, for now…

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Cloud conversations: If focused on cost you might miss other cloud storage benefits

Storage I/O trends

Cloud conversations: If focused on cost you might miss other cloud storage benefits

Drew Robb (@robbdrew) has a good piece (e.g. article) over at InfoStor titled Eight Ways to Avoid Cloud Storage Pricing Surprises that you can read here.

Drew start’s his piece out with this nice analogy or story:

Let’s begin with a cautionary tale about pricing: a friend hired a moving company as they quoted a very attractive price for a complex move. They lured her in with a low-ball price then added more and more “extras” to the point where their price ended up higher than many of the other bids she passed up. And to make matters worse, they are already two weeks late with delivery of the furniture and are saying it might take another two weeks.

Drew extends his example in his piece to compare how some cloud providers may start with pricing as low as some amount only for the customer to be surprised when they did not do their homework to learn about the various fees.

Note that most reputable cloud providers do not hide their fees even though there are myths that all cloud vendors have hidden fees, instead they list what those costs are on their sites. However that means the smart shopper or person procuring cloud services needs to go look for those fee’s and what they mean to avoid surprises. On the other hand if you can not find what extra fee’s would be along with what is or is not included in a cloud service price, to quote Jenny’s line in the movie Forest Gump, "…Run, Forest! Run!…".

In Drew’s piece he mentions five general areas to keep an eye on pertaining cloud storage costs including:

  • Be Duly Diligent
  • Trace Out Application Interaction
  • Avoid Fixed Usage Rates
  • Beware Lowballing
  • Demand Enterprise Visibility

Beware Lowballing

In Drew’s piece, he includes a comment from myself shown below.

Just as in the moving business, lowballing is alive and well in cloud pricing. Greg Schulz, an analyst with StorageIO Group, warned users to pay attention to services that have very low-cost per GByte/TByte yet have extra fees and charges for use, activity or place service caps. Compare those with other services that have higher base fees and attempt to price it based on your real storage and usage patterns.

“Watch out for usage and activity fees with lower cost services where you may get charged for looking at or visiting your data, not to mention for when you actually need to use it,” said Schulz. “Also be aware of limits or caps on performance that may apply to a particular class of service.”

As a follow-up to Drew’s good article, I put together the following thoughts that appeared earlier this year over at InfoStor titled Cloud storage: Is It All About Cost? that you can read here. In that article I start out with the basic question of:

So what is your take on cloud storage, and in what context?

Is cloud storage all about removing cost, cost cutting, free storage?

Or perhaps even getting something else in addition to free storage?

I routinely talk with different people from various backgrounds, environments from around the world, and the one consistency I hear when it comes to cloud services including storage is that there is no consistency.

What I mean by this is that there are the cloud crowd cheerleaders who view or cheer for anything cloud related, some of them actually use the cloud vs. simply cheering.

What does this have to do with cloud costs

Simple, how do you know if cloud is cheaper or more expensive if you do not know your own costs?

How do you know if cloud storage is available, reliable, durable if you do not have a handle on your environment?

Are you making apples to oranges comparisons or simple trading or leveraging hype and fud for or against?

Similar to regular storage, how you choose to use and configure on-site traditional storage for high-availability, performance, security among other best practices should be applied to cloud solutions. After all, only you can prevent cloud (or on premise) data loss, granted it is a shared responsibility. Shared responsibility means your service provider or system vendor needs to deliver quality robust solution that you can then take responsibility for configure to use with resiliency.

For some of you perhaps cloud might be about lowering, reducing or cutting storage costs, perhaps even getting some other service(s) in addition to free storage.

On the other hand, some of you might be

Yet another class of cloud storage (e.g. AWS EBS) are those intended or optimized to be accessed from within a cloud via cloud servers or compute instances (e.g. AWS EC2 among others) vs. those that are optimized for both inside the cloud as well as outside the cloud access (e.g. AWS S3 or Glacier with costs shown here). I am using AWS examples; however, you could use Microsoft Azure (pricing shown here), Google (including their new Nearline service with costs shown here), Rackspace, (calculator here or other cloud files pricing here), HP Cloud (costs shown here), IBM Softlayer (object storage costs here) and many others.

Not all types of cloud storage are the same, which is similar to traditional storage you may be using or have used in your environment in the past. For example, there is high-capacity low-cost storage, including magnetic tape for data protection, archiving of in-active data along with near-line hard disk drives (HDD). There are different types of HDDs, as well as fast solid-state devices (SSD) along with hybrid or SSHD storage used for different purposes. This is where some would say the topic of cloud storage is highly complex.

Where to learn more

Data Protection Diaries
Cloud Conversations: AWS overview and primer)
Only you can prevent cloud data loss
Is Computer Data Storage Complex? It Depends
Eight Ways to Avoid Cloud Storage Pricing Surprises
Cloud and Object Storage Center
Cloud Storage: Is It All About Cost?
Cloud conversations: Gaining cloud confidence from insights into AWS outages (Part II)
Given outages, are you concerned with the security of the cloud?
Is the cost of cloud storage really cheaper than traditional storage?
Are more than five nines of availability really possible?
What should I look for in an enterprise file sync-and-share app?
How do primary storage clouds and cloud for backup differ?
What should I consider when using SSD cloud?
What’s most important to know about my cloud privacy policy?
Data Archiving: Life Beyond Compliance
My copies were corrupted: The 3-2-1 rule
Take a 4-3-2-1 approach to backing up data

What this means

In my opinion there are cheap clouds (products, services, solutions) and there are low-cost options as well as there are value and premium offerings. Avoid confusing value with cheap or low-cost as something might have a higher cost, however including more capabilities or fees included that if useful can be more value. Look beyond the up-front cost aspects of clouds also considering ongoing recurring fees for actually using a server or solution.

If you can find low-cost storage at or below a penny per GByte per month that could be a good value if it also includes many free access, retrieval GETS head and lists for management or reporting. On the other hand, if you find a service that is at or below a penny per GByte per month however charges for any access including retrieval, as well as network bandwidth fees along with reporting, that might not be as good of a value.

Look beyond the basic price and watch out for statements like "…as low as…" to understand what is required to get that "..as low as.." price. Also understand what the extra fee’s are which most of the reputable providers list these on their sites, granted you have to look for them. If you are already using cloud services, pay attention to your monthly invoices and track what you are paying for to avoid surprises.

From my InfoStor piece:

For cloud storage, instead of simply focusing on lowest cost of storage per capacity, look for value, along with ability to configure or use with as much resiliency as you need. Value will mean different things depending on your needs and cloud storage servers, yet the solution should be cost-effective with availability including durability, secure and applicable performance.

Shopping for cloud servers and storage is similar to acquiring regular servers and storage in that you need to understand what you are acquiring along with up-front and recurring fee’s to understand the total cost of ownership and cost of operations not to mention making apples to apples vs. apples to oranges comparisons.

Btw, instead of simply using lower cost cloud services to cut cost, why not also use those capabilities to create or park another copy of your important data somewhere else just to be safe…

What say you about cloud costs?

Ok, nuff said, for now…

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Top vblog voting V2.015 (Its IT award season, cast your votes)

Top vblog voting V2.015 (Its IT award season, cast your votes)

Storage I/O trends

It’s that time of the year again for award season:

  • The motion picture association Academy awards (e.g. the Oscars)
  • The Grammys and other entertainment awards
  • As well as Eric Siebert (aka @ericsiebert) vsphere-land.com top vblog

Vsphere-land.com top vblog

Eric has run for several years now an annual top VMware, Virtualization, Storage and related blogs voting now taking place until March 16th 2015 (click on the image below). You will find a nice mix of new school, old school and a few current or future school theme blogs represented with some being more VMware specific. However there are also many blogs at the vpad site that have a cloud, virtual, server, storage, networking, software defined, development and other related themes.

top vblog voting
Click on the above image to cast your vote for favorite:

  • Ten blogs (e.g. select up to ten and then rank 1 through 10)
  • Storage blog
  • Scripting blog
  • VDI blog
  • New Blogger
  • Independent Blogger (e.g. non-vendor)
  • News/Information Web site
  • Podcast

Call to action, take a moment to cast your vote

My StorageIOblog.com has been on the vLaunchPad site for several years now as well as having syndicated content that also appears via some of the other venues listed there.

Six time VMware vExpert

In addition to my StorageIOblog and podcast, you will also find many of my fellow VMware vExperts among others at the vLaunchpad site so check them out as well.

What this means

This is a people’s choice process (yes it is a popularity process of sorts as well) however also a way of rewarding or thanking those who take time to create and share content with you and others. If you take time to read various blogs, listen to podcasts as well as consume other content, please take a few moments and cast your vote here (thank you in advance) which I hope includes StorageIOblog.com as part of the top ten, as well as being nominated in the Storage, Podcast and Independent blogger categories.

Ok, nuff said, for now…

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Microsoft Diskspd (Part II): Server Storage I/O Benchmark Tools

Microsoft Diskspd (Part II): Server Storage I/O Benchmark Tools

server storage I/O trends

This is part-two of a two-part post pertaining Microsoft Diskspd.that is also part of a broader series focused on server storage I/O benchmarking, performance, capacity planning, tools and related technologies. You can view part-one of this post here, along with companion links here.

Microsoft Diskspd StorageIO lab test drive

Server and StorageIO lab

Talking about tools and technologies is one thing, installing as well as trying them is the next step for gaining experience so how about some quick hands-on time with Microsoft Diskspd (download your copy here).

The following commands all specify an I/O size of 8Kbytes doing I/O to a 45GByte file called diskspd.dat located on the F: drive. Note that a 45GByte file is on the small size for general performance testing, however it was used for simplicity in this example. Ideally a larger target storage area (file, partition, device) would be used, otoh, if your application uses a small storage device or volume, then tune accordingly.

In this test, the F: drive is an iSCSI RAID protected volume, however you could use other storage interfaces supported by Windows including other block DAS or SAN (e.g. SATA, SAS, USB, iSCSI, FC, FCoE, etc) as well as NAS. Also common to the following commands is using 16 threads and 32 outstanding I/Os to simulate concurrent activity of many users, or application processing threads.
server storage I/O performance
Another common parameter used in the following was -r for random, 7200 seconds (e.g. two hour) test duration time, display latency ( -L ) disable hardware and software cache ( -h), forcing cpu affinity (-a0,1,2,3). Since the test ran on a server with four cores I wanted to see if I could use those for helping to keep the threads and storage busy. What varies in the commands below is the percentage of reads vs. writes, as well as the results output file. Some of the workload below also had the -S option specified to disable OS I/O buffering (to view how buffering helps when enabled or disabled). Depending on the goal, or type of test, validation, or workload being run, I would choose to set some of these parameters differently.

diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -w0 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noh_write000.txt

diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -w50 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noh_write050.txt

diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -w100 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noh_write100.txt

diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -S -w0 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noSh_test_write000.txt

diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -S -w50 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noSh_write050.txt

diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -S -w100 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noSh_write100.txt

The following is the output from the above workload command.
Microsoft Diskspd sample output
Microsoft Diskspd sample output part 2
Microsoft Diskspd sample output part 3

Note that as with any benchmark, workload test or simulation your results will vary. In the above the server, storage and I/O system were not tuned as the focus was on working with the tool, determining its capabilities. Thus do not focus on the performance results per say, rather what you can do with Diskspd as a tool to try different things. Btw, fwiw, in the above example in addition to using an iSCSI target, the Windows 2012 R2 server was a guest on a VMware ESXi 5.5 system.

Where to learn more

The following are related links to read more about server (cloud, virtual and physical) storage I/O benchmarking tools, technologies and techniques.

Drew Robb’s benchmarking quick reference guide
Server storage I/O benchmarking tools, technologies and techniques resource page
Server and Storage I/O Benchmarking 101 for Smarties.
Microsoft Diskspd download and Microsoft Diskspd overview (via Technet)
I/O, I/O how well do you know about good or bad server and storage I/Os?
Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I and Part II)

Comments and wrap-up

What I like about Diskspd (Pros)

Reporting including CPU usage (you can’t do server and storage I/O without CPU) along with IOP’s (activity), bandwidth (throughout or amount of data being moved), per thread and total results along with optional reporting. While a GUI would be nice particular for beginners, I’m used to setting up scripts for different workloads so having an extensive options for setting up different workloads is welcome. Being associated with a specific OS (e.g. Windows) the CPU affinity and buffer management controls will be handy for some projects.

Diskspd has the flexibility to use different storage interfaces and types of storage including files or partitions should be taken for granted, however with some tools don’t take things for granted. I like the flexibility to easily specify various IO sizes including large 1MByte, 10MByte, 20MByte, 100MByte and 500MByte to simulate application workloads that do large sequential (or random) activity. I tried some IO sizes (e.g. specified by -b parameter larger than 500MB however, I received various errors including "Could not allocate a buffer bytes for target" which means that Diskspd can do IO sizes smaller than that. While not able to do IO sizes larger than 500MB, this is actually impressive. Several other tools I have used or with have IO size limits down around 10MByte which makes it difficult for creating workloads that do large IOP’s (note this is the IOP size, not the number of IOP’s).

Oh, something else that should be obvious however will state it, Diskspd is free unlike some industry de-facto standard tools or workload generators that need a fee to get and use.

Where Diskspd could be improved (Cons)

For some users a GUI or configuration wizard would make the tool easier to get started with, on the other hand (oth), I tend to use the command capabilities of tools. Would also be nice to specify ranges as part of a single command such as stepping through an IO size range (e.g. 4K, 8K, 16K, 1MB, 10MB) as well as read write percentages along with varying random sequential mixes. Granted this can easily be done by having a series of commands, however I have become spoiled by using other tools such as vdbench.

Summary

Server and storage I/O performance toolbox

Overall I like Diskspd and have added it to my Server Storage I/O workload and benchmark tool-box

Keep in mind that the best benchmark or workload generation technology tool will be your own application(s) configured to run as close as possible to production activity levels.

However when that is not possible, the an alternative is to use tools that have the flexibility to be configured as close as possible to your application(s) workload characteristics. This means that the focus should not be as much on the tool, as opposed to how flexible is a tool to work for you, granted the tool needs to be robust.

Having said that, Microsoft Diskspd is a good and extensible tool for benchmarking, simulation, validation and comparisons, however it will only be as good as the parameters and configuration you set it up to use.

Check out Microsoft Diskspd and add it to your benchmark and server storage I/O tool-box like I have done.

Ok, nuff said (for now)

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Green and Virtual Data Center: Productive Economical Efficient Effective Flexible

Green and Virtual Data Center

A Green and Virtual IT Data Center (e.g. an information factory) means an environment comprising:

  • Habitat for technology or physical infrastructure (e.g. physical data center, yours, co-lo, managed service or cloud)
  • Power, cooling, communication networks, HVAC, smoke and fire suppression, physical security
  • IT data information infrastructure (e.g. hardware, software, valueware, cloud, virtual, physical, servers, storage, network)
  • Data Center Infrastructure Management (DCIM) along with IT Service Management (ITSM) software defined management tools
  • Tools for monitoring, resource tracking and usage, reporting, diagnostics, provisioning and resource orchestration
  • Portals and service catalogs for automated, user initiated and assisted operation or access to IT resources
  • Processes, procedures, best-practices, work-flows and templates (including data protection with HA, BC, BR, DR, backup/restore, logical and physical security)
  • Metrics that matter for management insight and awareness
    People and skill sets among other items

Green and Virtual Data Center Resources

Click here to learn about "The Green and Virtual Data Center" book (CRC Press) for enabling efficient, productive IT data centers. This book covers cloud, virtualization, servers, storage, networks, software, facilities and associated management topics, technologies and techniques including metrics that matter. This book by industry veteran IT advisor and author Greg Schulz is the definitive guide for enabling economic efficiency and productive next generation data center strategies.

Intel recommended reading
Publisher: CRC Press – Taylor & Francis Group
By Greg P. Schulz of StorageIO www.storageio.com
 ISBN-10: 1439851739 and ISBN-13: 978-1439851739
 Hardcover * 370 pages * Over 100 illustrations figures and tables

Read more here and order your copy here. Also check out Cloud and Virtual Data Storage Networking (CRC Press) a new book by Greg Schulz.

Productive Efficient Effective Economical Flexible Agile and Sustainable

Green hype and green washing may be on the endangered species list and going away, however, green IT for servers, storage, networks, facilities as well as related software and management techniques that address energy efficiency including power and cooling along with e-waste, environmental health and safety related issues are topics that wont be going away anytime soon. There is a growing green gap between green hype messaging or green washing and IT pain point issues including limits on availability or rising costs of power, cooling, floor-space as well as e-waste and environmental health and safety (PCFE). To close the gap will involve addressing green messaging and rhetoric closer to where IT organizations pain points are and where budget dollars exists that can address PCFE and other green related issues as a by-product.

The green gap will also be narrowed as awareness of broader green related topics coincide with IT data center pain points, in other words, alignment of messaging with IT issues that have or will have budget dollars allocated towards them to sustain business and economic growth via IT resource usage efficiency. Read more here.

Where to learn more

The following are useful links to related efficient, effective, productive, flexible, scalable and resilient IT data center along with server storage I/O networking hardware and software that supports cloud and virtual green data centers.

Various IT industry vendor and service provider links
Green and Virtual Data Center Primer
Green and Virtual Data Center links
Are large storage arrays dead at the hands of SSD?
Closing the Green Gap
Energy efficient technology sales depend on the pitch
EPA Energy Star for Data Center Storage Update
EPA Energy Star for data center storage draft 3 specification
Green IT Confusion Continues, Opportunities Missed! 
Green IT deferral blamed on economic recession might be result of green gap
How much SSD do you need vs. want?
How to reduce your Data Footprint impact (Podcast) 
Industry trend: People plus data are aging and living longer
In the data center or information factory, not everything is the same
More storage and IO metrics that matter
Optimizing storage capacity and performance to reduce your data footprint 
Performance metrics: Evaluating your data storage efficiency
PUE, Are you Managing Power, Energy or Productivity?
Saving Money with Green Data Storage Technology
Saving Money with Green IT: Time To Invest In Information Factories 
Shifting from energy avoidance to energy efficiency
SNIA Green Storage Knowledge Center
Speaking of speeding up business with SSD storage
SSD and Green IT moving beyond green washing
Storage Efficiency and Optimization: The Other Green
Supporting IT growth demand during economic uncertain times
The Green and Virtual Data Center Book (CRC Press, Intel Recommended Reading)
The new Green IT: Efficient, Effective, Smart and Productive 
The other Green Storage: Efficiency and Optimization 
What is the best kind of IO? The one you do not have to do

Watch for more links and resources to be added soon.

What this all means

The result of a green and virtual data center is that of a flexible, agile, resilient, scalable information factory that is also economical, productive, efficient, productive as well as sustainable.

Ok, nuff said (for now)

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

Green and Virtual Data Center Links

Updated 10/25/2017

Green and Virtual IT Data Center Links

Moving beyond Green Hype and Green washing

Green hype and green washing may be on the endangered species list and going away, however, green IT for servers, storage, networks, facilities as well as related software and management techniques that address energy efficiency including power and cooling along with e-waste, environmental health and safety related issues are topics that wont be going away anytime soon.

There is a growing green gap between green hype messaging or green washing and IT pain point issues including limits on availability or rising costs of power, cooling, floor-space as well as e-waste and environmental health and safety (PCFE).

To close the gap will involve addressing green messaging and rhetoric closer to where IT organizations pain points are and where budget dollars exists that can address PCFE and other green related issues as a by-product. The green gap will also be narrowed as awareness of broader green related topics coincide with IT data center pain points, in other words, alignment of messaging with IT issues that have or will have budget dollars allocated towards them to sustain business and economic growth via IT resource usage efficiency. Read more here.

Enabling Effective Produtive Efficient Economical Flexible Scalable Resilient Information Infrastrctures

The following are useful links to related efficient, effective, productive, flexible, scalable and resilient IT data center along with server storage I/O networking hardware and software that supports cloud and virtual green data centers.

Various IT industry vendors and other links

Via StorageIOblog – Happy Earth Day 2016 Eliminating Digital and Data e-Waste

Green and Virtual Data Center Primer
Green and Virtual Data Center: Productive Economical Efficient Effective Flexible
Are large storage arrays dead at the hands of SSD?
Closing the Green Gap
Energy efficient technology sales depend on the pitch
EPA Energy Star for Data Center Storage Update
EPA Energy Star for data center storage draft 3 specification
Green IT Confusion Continues, Opportunities Missed! 
Green IT deferral blamed on economic recession might be result of green gap
How much SSD do you need vs. want?
How to reduce your Data Footprint impact (Podcast) 
Industry trend: People plus data are aging and living longer
In the data center or information factory, not everything is the same
More storage and IO metrics that matter
Optimizing storage capacity and performance to reduce your data footprint 
Performance metrics: Evaluating your data storage efficiency
PUE, Are you Managing Power, Energy or Productivity?
Saving Money with Green Data Storage Technology
Saving Money with Green IT: Time To Invest In Information Factories 
Shifting from energy avoidance to energy efficiency
SNIA Green Storage Knowledge Center
Speaking of speeding up business with SSD storage
SSD and Green IT moving beyond green washing
Storage Efficiency and Optimization: The Other Green
Supporting IT growth demand during economic uncertain times
The Green and Virtual Data Center Book (CRC Press, Intel Recommended Reading)
The new Green IT: Efficient, Effective, Smart and Productive 
The other Green Storage: Efficiency and Optimization 
What is the best kind of IO? The one you do not have to do

Intel recommended reading
Click here to learn about "The Green and Virtual Data Center" book (CRC Press) for enabling efficient , productive IT data centers. This book covers cloud, virtualization, servers, storage, networks, software, facilities and associated management topics, technologies and techniques including metrics that matter. This book by industry veteran IT advisor and author Greg Schulz is the definitive guide for enabling economic efficiency and productive next generation data center strategies. Read more here and order your copyhere. Also check out Cloud and Virtual Data Storage Networking (CRC Press) a new book by Greg Schulz.

White papers, analyst reports and perspectives

Business benefits of data footprint reduction (archiving, compression, de-dupe)
Data center I/O and performance issues – Server I/O and storage capacity gap
Analysis of EPA Report to Congress (Law 109-431)
The Many Faces of MAID Storage Technology
Achieving Energy Efficiency with FLASH based SSD
MAID 2.0: Energy Savings without Performance Compromises

Articles, Tips, Blogs, Webcasts and Podcasts

AP – SNIA Green Emerald Program and measurements
AP – Southern California heat wave strains electrical system
Ars Technica – EPA: Power usage in data centers could double by 2011
Ars Technica – Meet the climate savers: Major tech firms launch war on energy-inefficient PCs – Article
Askageek.com – Buying an environmental friendly laptop – November 2008
Baseline – Examining Energy Consumption in the Data Center
Baseline – Burts Bees: What IT Means When You Go Green
Bizcovering – Green architecture for the masses
Broadstuff – Are Green 2.0 and Enterprise 2.0 Incompatible?
Business Week – CEO Guide to Technology
Business Week – Computers’ elusive eco factor
Business Week – Clean Energy – Its Getting Affordable
Byte & Switch – Keeping it Green This Summer – Don’t be "Green washed"
Byte & Switch – IBM Sees Green in Energy Certificates
Byte & Switch – Users Search for power solutions
Byte & Switch – DoE issues Green Storage Warning
CBR – The Green Light for Green IT
CBR – Big boxes make greener data centers
CFO – Power Scourge
Channel Insider – A 12 Step Program to Dispose of IT Equipment
China.org.cn – China publishes Energy paper
CIO – Green Storage Means Money Saved on Power
CIO – Data center designers share secrets for going green
CIO – Best Place to Build a Data Center in North America
CIO Insight – Clever Marketing or the Real Thing?
Cleantechnica – Cooling Data Centers Could Prevent Massive Electrical Waste – June 2008
Climatebiz – Carbon Calculators Yield Spectrum of Results: Study
CNET News – Linux coders tackle power efficiency
CNET News – Research: Old data centers can be nearly as ‘green’ as new ones
CNET News – Congress, Greenpeace move on e-wast
CNN Money – A Green Collar Recession
CNN Money – IBM creates alliance with industry leaders supporting new data center standards
Communication News – Utility bills key to greener IT
Computerweekly – Business case for green storage
Computerweekly – Optimising data centre operations
Computerweekly – Green still good for IT, if it saves money
Computerweekly – Meeting the Demands for storage
Computerworld – Wells Fargo Free Data Center Cooling System
Computerworld – Seven ways to get green and save money
Computerworld – Build your data center here: The most energy-efficient locations
Computerworld – EPA: U.S. needs more power plants to support data centers
Computerworld – GreenIT: A marketing ploy or new technology?
Computerworld – Gartner Criticizes Green Grid
Computerworld – IT Skills no longer sufficient for data center execs.
Computerworld – Meet MAID 2.0 and Intelligent Power Management
Computerworld – Feds to offer energy ratings on servers and storage
Computerworld – Greenpeace still hunting for truly green electronics
Computerworld – How to benchmark data center energy costs
ComputerworldUK – Datacenters at risk from poor governance
ComputerworldUK – Top IT Leaders Back Green Survey
ComputerworldMH – Lean and Green
CTR – Strategies for enhancing energy efficiency
CTR – Economies of Scale – Green Data Warehouse Appliances
Datacenterknowledge – Microsoft to build Illinois datacenter
Data Center Strategies – Storage The Next Hot Topic
Earthtimes – Fujitsu installs hydrogen fuel cell power
eChannelline – IBM Goes Green(er)
Ecoearth.info – California Moves To Speed Solar, Wind Power Grid Connections
Ecogeek – Solar power company figures they can power 90% of America
Economist – Cool IT
Electronic Design – How many watts in that Gigabyte
eMazzanti – Desktop virtualization movement creeping into customer sites
ens-Newswire – Western Governors Ask Obama for National Green Energy Plan
Environmental Leader – Best Place to Build an Energy Efficient Data Center
Environmental Leader – New Guide Helps Advertisers Avoid Greenwash Complaints
Enterprise Storage Forum – Power Struggles Take Center Stage at SNW
Enterprise Storage Forum – Pace Yourself for Storage Power & Cooling Needs
Enterprise Storage Forum – Storage Power and Cooling Issues Heat Up – StorageIO Article
Enterprise Storage Forum – Score Savings With A Storage Power Play
Enterprise Storage Forum – I/O, I/O, Its off to Virtual Work I Go
Enterprise Storage Forum – Not Just a Flash in the Pan – Various SSD options
Enterprise Storage Forum – Closing the Green Gap – Article August 2008
EPA Report to Congress and Public Law 109-431 – Reports & links
eWeek – Saving Green by being Green
eWeek – ‘No Cooling Necessary’ Data Centers Coming?
eWeek – How the ‘Down’ Macroeconomy Will Impact the Data Storage Sector
ExpressComputer – In defense of Green IT
ExpressComputer – What data center crisis
Forbes – How to Build a Quick Charging Battery
GCN – Sun launches eco data center
GreenerComputing – New Code of Conduct to Establish Best Practices in Green Data Centers
GreenerComputing – Silicon valley’s green detente
GreenerComputing – Majority of companies plan to green their data centers
GreenerComputing – Citigroup to spend $232M on Green Data Center
GreenerComputing – Chicago and Quincy, WA Top Green Data Center Locations
GreenerComputing – Using airside economizers to chill data center cooling bills
GreenerComputing – Making the most of asset disposal
GreenerComputing – Greenpeace vendor rankings
GreenerComputing – Four Steps to Improving Data Center Efficiency without Capital Expenditures
GreenerComputing – Enabling a Green and Virtual Data Center
Green-PC – Strategic Steps Down the Green Path
Greeniewatch – BBC news chiefs attack plans for climate change campaign
Greeniewatch – Warmest year predictions and data that has not yet been measured
GoverenmentExecutive – Public Private Sectors Differ on "Green" Efforts
HPC Wire – How hot is your code
Industry Standard – Why green data centers mean partner opportunities
InformationWeek – It could be 15 years before we know what is really green
InformationWeek – Beyond Server Consolidaiton
InformationWeek – Green IT Beyond Virtualization: The Case For Consolidation
InfoWorld – Sun celebrates green datacenter innovations
InfoWorld – Tech’s own datacenters are their green showrooms
InfoWorld – 2007: The Year in Green
InfoWorld – Green Grid Announces Tech Forum in Feb 2008
InfoWorld – SPEC seeds future green-server benchmarks
InfoWorld – Climate Savers green catalog proves un-ripe
InfoWorld – Forester: Eco-minded activity up among IT pros
InfoWorld – Green ventures in Silicon Valley, Mass reaped most VC cash in ’07
InfoWorld – Congress misses chance to see green-energy growth
InfoWorld – Unisys pushes green envelope with datacenter expansion
InfoWorld – No easy green strategy for storage
Internet News – Storage Technologies for a Slowing Economy
Internet News – Economy will Force IT to Transform
ITManagement – Green Computing, Green Revenue
itnews – Data centre chiefs dismiss green hype
itnews – Australian Green IT regulations could arrive this year
IT Pro – SNIA Green storage metrics released
ITtoolbox – MAID discussion
Linux Power – Saving power with Linux on Intel platforms
MSNBC – Microsoft to build data center in Ireland
National Post – Green technology at the L.A. Auto Show
Network World – Turning the datacenter green
Network World – Color Interop Green
Network World – Green not helpful word for setting environmental policies
NewScientistEnvironment – Computer servers as bad for climate as SUVs
Newser – Texas commission approves nation’s largest wind power project
New Yorker – Big Foot: In measuring carbon emissions, it’s easy to confuse morality and science
NY Times – What the Green Bubble Will Leave Behind
PRNewswire – Al Gore and Cisco CEO John Chambers to debate climate change
Processor – More than just monitoring
Processor – The new data center: What’s hot in Data Center physical infrastructure:
Processor – Liquid Cooling in the Data Center
Processor – Curbing IT Power Usage
Processor – Services To The Rescue – Services Available For Today’s Data Centers
Processor – Green Initiatives: Hire A Consultant?
Processor – Energy-Saving Initiatives
Processor – The EPA’s Low Carbon Campaig
Processor – Data Center Power Planning
SAN Jose Mercury – Making Data Centers Green
SDA-Asia – Green IT still a priority despite Credit Crunch
SearchCIO – EPA report gives data centers little guidance
SearchCIO – Green IT Strategies Could Lead to hefty ROIs
SearchCIO – Green IT In the Data Center: Plenty of Talk, not much Walk
SearchCIO – Green IT Overpitched by Vendors, CIOs beware
SearchDataCenter – Study ranks cheapest places to build a data center
SearchDataCenter – Green technology still ranks low for data center planners
SearchDataCenter – Green Data Center: Energy Effiecnty Computing in the 21st Century
SearchDataCenter – Green Data Center Advice: Is LEED Feasible
SearchDataCenter – Green Data Centers Tackle LEED Certification
SearchDataCenter – PG&E invests in data center effieicny
SearchDataCenter – A solar powered datacenter
SearchSMBStorage – Improve your storage energy efficiency
SearchSMBStorage – SMB capacity planning: Focusing on energy conservation
SearchSMBStorage – Data footprint reduction for SMBs
SearchSMBStorage – MAID & other energy-saving storage technologies for SMBs
SearchStorage – How to increase your storage energy efficiency
SearchStorage – Is storage now top energy hog in the data center
SearchStorage – Storage eZine: Turning Storage Green
SearchStorage – The Green Storage Gap
SearchStorageChannel – Green Data Storage Projects
Silicon.com – The greening of IT: Cooling costs
SNIA – SNIA Green Storage Overview
SNIA – Green Storage
SNW – Beyond Green-wash
SNW Spring 2008 Beyond Green-wash
State.org – Why Texas Has Its Own Power Grid
StorageDecisions – Different Shades of Green
Storage Magazine – Storage still lacks energy metrics
StorageIOblog – Posts pertaining to Green, power, cooling, floor-space, EHS (PCFE)
Storage Search – Various postings, news and topics pertaining to Green IT
Technology Times – Revealed: the environmental impact of Google searches
TechTarget – Data center power efficiency
TechTarget – Tip for determining power consumption
Techworld – Inside a green data center
Techworld – Box reduction – Low hanging green datacenter fruit
Techworld – Datacentere used to heat swimming pool
Theinquirer – Spansion and Virident flash server farms
Theinquirer – Storage firms worry about energy efficiency How green is the valley
TheRegister – Data Centre Efficiency, the good, the bad and the way to hot
TheRegister – Server makers snub whalesong for serious windmill abuse
TheRegister – Green data center threat level: Not Green
The Standard – Growing cynicism around going Green
ThoughtPut – Energy Central
Thoughtput – Power, Cooling, Green Storage and related industry trends
Wallstreet Journal – Utilities Amp Up Push To Slash Energy Use
Wallstreet Journal – The IT in Green Investing
Wallstreet Journal – Tech’s Energy Consumption on the Rise
Washingtonpost – Texas approves major new wind power project
WhatPC – Green IT: It doesnt have to cost the earth
WHIRnews – SingTel building green data center
Wind-watch.org – Loss of wind causes Texas power grid emergency
WyomingNews – Overcoming Greens Stereotype
Yahoo – Washington Senate Unviel Green Job Plan
ZDnet – Will supercomputer speeds hit a plateau?
Are data centers causing climate change

News and Press Releases

Business Wire – The Green and Virtual Data Center
Enterprise Storage Forum – Intel and HGST (Hitachi) partner on FLASH SSD
PCworld – Intel and HP describe Green Strategy
DoE – To Invest Approximately $1.3 Billion to Commercialize CCS Technology
Yahoo – Shell Opens Los Angeles’ First Combined Hydrogen and Gasoline Station
DuPont – DuPont Projects Save Enough Energy to Power 25,000 Homes
Gartner – Users Are Becoming Increasingly Confused About the Issues and Solutions Surrounding Green IT

Websites and Tools

Various power, cooling, emmisions and device configuration tools and calculators
Solar Action Alliance web site
SNIA Emerald program
Carbon Disclosure Project
The Chicago Climate Exchange
Climate Savers
Data Center Decisions
Electronic Industries Alliance (EIA)
EMC – Digital Life Calculator
Energy Star
Energy Star Data Center Initiatives
Greenpeace – Technology ranking website also here
GlobalActionPlan
KyotoPlanet
LBNL High Tech Data centers
Millicomputing
RoHS & WEE News
Storage Performance Council (SPC)
SNIA Green Technical Working Group
SPEC
Transaction Processing Council (TPC)
The Green Grid
The Raised Floor
Terra Pass Carbon Offset Credits – Website with CO2 calculators
Energy Information Administration – EIA (US and International Electrical Information)
U.S. Department of Energy and related information
U.S. DOE Energy Efficient Industrial Programs
U.S. EPA server and storage energy topics
Zerofootprint – Various "Green" and environmental related links and calculators

Vendor Centric and Marketing Website Links and tools

Vendors and organizations have different types of calculators some with focus on power, cooling, floor space, carbon offsets or emissions,

ROI, TCO and other IT data center infrastructure resource management. Following is an evolving list and by no means definitive even for a particular vendors as

different manufactures may have multiple different calculators for different product lines or areas of focus.

Brocade – Green website
Cisco – Green and Environmental websites here, here and here
Dell – Green website
EMC – EMC Energy, Power and Cooling Related Website
HDS – How to be green – HDS Positioning White Paper
HP – HP Green Website
IBM – Green Data Center – IBM Positioning White Paper
IBM – Green Data Center for Education – IBM Positioning White Paper
Intel – What is an Efficient Data Center and how do I measure it?
LSI – Green site and white paper
NetApp – Press Release and related information
Sun – Various articles and links
Symantec – Global 2000 Struggle to Adopt "Green" Data Centers – Announcement of Survey results
ACTON
Adinfa
APC
Australian Conservation Foundation
Avocent
BBC
Brocade
Carbon Credit Calculator UK
Carbon Footprint Site
Carbon Planet
Carbonify
CarbonZero
Cassatt
CO2 Stats Site
Copan
Dell
DirectGov UK Acton
Diesel Service & Supply Power Calculator & Converter
Eaton Powerware
Ecobusinesslinks
Ecoscale
EMC Power Calculator
EMC Web Power Calculator
EMC Digital Life Calculator
EPA Power Profiler
EPA Related Tools
EPEAT
Google UK Green Footprint
Green Grid Calculator
HP and more here
HVAC Calculator
IBM
Logicalis
Kohler Power (Business and Residential)
Micron
MSN Carbon Footprint Calculator
National Wildlife Foundation
NEF UK
NetApp
Rackwise
Platespin
Safecom
Sterling Planet
Sun and more here and here and here
Tandberg
TechRepublic
TerraPass Carbon Offset Credits
Thomas Kreen AG
Toronto Hydro Calculator
80 Plus Calculator
VMware
42u Green Grid PUE DCiE calculator
42u energy calculator

Green and Virtual Tools

What’s your power, cooling, floor space, energy, environmental or green story?

What’s your power, cooling, floor space, energy, environmental or green story? Do you have questions or want to learn more about

energy issues pertaining to IT data center and data infrastructure topics? Do you have a solution or technology or a success story that you would like to share

with us pertaining to data storage and server I/O energy optimization strategies?  Do you need assistance in developing, validating or reviewing your strategy

or story? Contact us at: info@storageio.com or 651-275-1563 to learn more about green data storage and server I/O or to

schedule a briefing to tell us about your energy efficiency and effectiveness story pertaining to IT data centers and data infrastructures.

Disclaimer and note:  URL’s submitted for inclusion on this site will be reviewed for consideration and to be

in generally accepted good taste in regards to the theme of this site.  Best effort has been made to validate and verify the URLs that appear on this page and

website however they are subject to change. The author and/or maintainer’s) of this page and web site make no endorsement to and assume no responsibility for the

URLs and their content that are listed on this page.

Green and Virtual Metrics

Chapter 5 "Measurement, Metrics, and Management of IT Resources" in the book "The Green and Virtual Data Center" (CRC Press) takes a look at the importance of being able to measure and monitor to enable effective management and utilization of IT resources across servers, storage, I/O networks, software, hardware and facilities.

There are many different points of interest for collecting metrics in an IT data center for servers, storage, networking and facilities along with various points of interest or perspectives. Data center personal have varied interest from a facilities to a resource (server, storage, networking) usage and effectiveness perspective for normal use as well as planning purposes or comparison when evaluating new technology. Vendors have different uses for metrics during R&D, Q/A testing and marketing or sales campaigns as well as on-going service and support. Industry trade groups including 80 Plus, SNIA and the green grid along with government groups including the EPA Energy Star are working to define and establish applicable metrics pertinent for Green and Virtual data centers.

Acronym

Description

Comment

DCiE

Data center Efficiency = (IT equipment / Total facility power) * 100

Shows a ratio of how well a data center is consuming power

DCPE

Data center Performance Efficiency = Effective IT workload / total facility power

Shows how effective data center is consuming power to produce a given level of service or work such as energy per transaction or energy per business function performed

PUE

Power usage effectiveness = Total facility power / IT equipment power

Inverse of DCE

Kilowatts (kw)

Watts / 1,000

One thousand watts

Annual kWh

kWh x 24 x 365

kWh used in on year

Megawatts (mw)

kW / 1,000

One thousand kW

BTU/hour

watts x 3.413

Heat generated in an hour from using energy in British Thermal Units. 12,000 BTU/hour can equate to 1 Ton of cooling.

kWh

1,000 watt hours

The number of watts used in one hour

Watts

Amps x Volts (e.g. 12 amps * 12 volts = 144 watts)

Unit of electrical energy power

Watts

BTU/hour x 0.293

Convert BTU/hr to watts

Volts

Watts / Amps (e.g. 144 watts / 12 amps = 12 volts)

The amount of force on electrons

Amps

Watts / Volts (e.g. 144 watts / 12 volts = 12 amps)

The flow rate of electricity

Volt-Amperes (VA)

Volts x Amps

Sometimes power expressed in Volt-Ampres

kVA

Volts x Amp / 1000

Number of kilovolt-ampres

kW

kVA x power-factor

Power factor is the efficiency of a piece of equipments use of power

kVA

kW / power-factor

Killovolt-Ampres

U

1U = 1.75”

EIA metric describing height of equipment in racks.

 

Activity / Watt Amount of work accomplished per unit of energy consumed. This could be IOPS, Transactions or Bandwidth per watt. Indicator how much work and how efficient energy is being used to accomplish useful work. This metric applies to active workloads or actively used and frequently accessed storage and data. Examples would be IOPS per watt, Bandwidth per watt, Transactions per watt, Users or streams per watt. Activity per watt should also be used in conjunction with another metric such as how much capacity is supported per watt and total watts consumed for a representative picture.

IOPS / Watt

Number of I/O operations (or transactions) / energy (watts)

Indicator of how effectively energy is being used to perform a given amount of work. The work could be I/Os, transactions, throughput or other indicator of application activity. For example SPC-1 / Watt, SPEC / Watt, TPC / Watt, transaction / watt,  IOP / Watt.

Bandwidth / Watt GBPS or TBPS or PBPS / Watt Amount of data transferred or moved per second and energy used. Often confused with Capacity per watt This indicates how much data is moved or accessed per second or time interval per unit of energy consumed. This is often confused with capacity per watt given that both bandwidth and capacity reference GByte, TByte, PByte.

Capacity / Watt

GB or TB or PB (storage capacity space / watt

Indicator of how much capacity (space) or bandwidth supported in a given configuration or footprint per watt of energy. For inactive data or off-line and archive data, capacity per watt can be an effective measurement gauge however for active workloads and applications activity per watt also needs to be looked at to get a representative indicator of how energy is being used

Mhz / Watt

Processor performance / energy (watts)

Indicator of how effectively energy is being used by a CPU or processor.

Carbon Credit

Carbon offset credit

Offset credits that can be bought and sold to offset your CO2 emissions

CO2 Emission

Average 1.341 lbs per kWh of electricity generated

The amount of average carbon dioxide (CO2) emissions from generating an average kWh of electricity

Various power, cooling, floor space and green storage or IT  related metrics

Metrics include Data center Efficiency (DCiE) via the greengrid which is the indicator ratio of a IT data center energy efficiency defined as IT equipment (servers, disk and tape storage, networking switches, routers, printers, etc) / Total facility power x 100 (for percentage). For example, if the sum of all IT equipment energy usage resulted in 1,500 kilowatt hours (kWh) per month yet the total facility power including UPS, energy switching, power conversation and filtering, cooling and associated infrastructure costs as well as IT equipment resulting in 3,500 kWh, the DCiE would be (1,500 / 3,500) x 100 = 43%. DCiE can be used as a ratio for example to show in the above scenario that IT equipment accounts for about 43% of energy consumed by the data center with in this scenario 57% of electrical energy being consumed by cooling, conversion and conditioning or lighting.

Power usage effectiveness (PUE) is the indicator ratio of total energy being consumed by the data center to energy being used to operate IT equipment. PUE is defined as total facility power / IT equipment energy consumption. Using the above scenario PUE = 2.333 (3,500 / 1,500) which means that a server requiring 100 watts of power would actually require (2.333 * 100) 233.3 watts of energy that includes both direct power and cooling costs. Similarly a storage system that required 1,500 kWh of energy to power would require (1,500*2.333) 3,499.5 kWh of electrical power including cooling.

Another metric that has the potential to have meaning is Data center Performance Efficiency (DCPE) that takes into consideration how much useful and effective work is performed by the IT equipment and data center per energy consumed. DCPE is defined as useful work / total facility power with an example being some number of transactions processed using servers, networks and storage divided by energy for the data center to power and cool the equipment. An relatively easy and straightforward implementation of DCPE is an IOPs per watt measurement that looks at how many IOPs can be performed (regardless of size or type such as reads or writes) per unit of energy in this case watts.

DCPE = Useful work / Total facility power, for example IOPS per watt of energy used

DCiE = IT equipment energy / Total facility power = 1 / PUE

PUE = Total facility energy / IT equipment energy

IOPS per Watt = Number of IOPs (or bandwidth) / energy used by the storage system

The importance of these numbers and metrics is to focus on the larger impact of a piece of IT equipment that includes its cost and energy consumption that factors in cooling and other hosting or site environmental costs. Naturally energy costs and CO2 (carbon offsets) will vary by geography and region along with type of electrical power being used (Coal, Natural Gas, Nuclear, Wind, Thermo, Solar, etc) and other factors that should be kept in perspective as part of the big picture. Learn more in Chapter 5 "Measurement, Metrics, and Management of IT Resources" in the book "The Green and Virtual Data Center" (CRC) and in the book Cloud and Virtual Data Storage Networking (CRC).

Disclaimer and notes

Disclaimer and note:  URL’s submitted for inclusion on this site will be reviewed for consideration and to be in generally accepted good taste in regards to the theme of this site.  Best effort has been made to validate and verify the URLs that appear on this page and web site however they are subject to change. The author and/or maintainer’s) of this page and web site make no endorsement to and assume no responsibility for the URLs and their content that are listed on this page.

What this all means

The result of a green and virtual data center is that of a flexible, agile, resilient, scalable information factory that is also economical, productive, efficient, productive as well as sustainable.

Ok, nuff said (for now)

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

December 2014 Server StorageIO Newsletter

December 2014

Hello and welcome to this December Server and StorageIO update newsletter.

Seasons Greetings

Seasons greetings

Commentary In The News

StorageIO news

Following are some StorageIO industry trends perspectives comments that have appeared in various venues. Cloud conversations continue to be popular including concerns about privacy, security and availability. Over at BizTech Magazine there are some comments about cloud and ROI. Some comments on AWS and Google SSD services can be viewed at SearchAWS. View other trends comments here

Tips and Articles

View recent as well as past tips and articles here

StorageIOblog posts

Recent StorageIOblog posts include:

View other recent as well as past blog posts here

In This Issue

  • Industry Trends Perspectives
  • Commentary in the news
  • Tips and Articles
  • StorageIOblog posts
  • Events & Activities

    View other recent and upcoming events here

    Webinars

    December 11, 2014 – BrightTalk
    Server & Storage I/O Performance

    December 10, 2014 – BrightTalk
    Server & Storage I/O Decision Making

    December 9, 2014 – BrightTalk
    Virtual Server and Storage Decision Making

    December 3, 2014 – BrightTalk
    Data Protection Modernization

    Videos and Podcasts

    StorageIO podcasts are also available via and at StorageIO.tv

    From StorageIO Labs

    Research, Reviews and Reports

    StarWind Virtual SAN for Microsoft SOFS

    May require registration
    This looks at the shared storage needs of SMB’s and ROBO’s leveraging Microsoft Scale-Out File Server (SOFS). Focus is on Microsoft Windows Server 2012, Server Message Block version (SMB) 3.0, SOFS and StarWind Virtual SAN management software

    View additional reports and lab reviews here.

    Resources and Links

    Check out these useful links and pages:
    storageio.com/links
    objectstoragecenter.com
    storageioblog.com/data-protection-diaries-main/
    storageio.com/ssd
    storageio.com/ssd

    Enjoy this edition of the Server and StorageIO update newsletter and watch for new tips, articles, StorageIO lab report reviews, blog posts, videos and podcasts along with in the news commentary appearing soon.

    Seasons greetings 2014

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Data Storage Tape Update V2014, Its Still Alive

    Data Storage Tape Update V2014, It’s Still Alive

    server storage I/O trends

    A year or so ago I did a piece tape summit resources. Despite being declared dead for decades, and will probably stay being declared dead for years to come, magnetic tape is in fact still alive being used by some organizations, granted its role is changing while the technology still evolves.

    Here is the memo I received today from the PR folks of the Tape Storage Council (e.g. tape vendors marketing consortium) and for simplicity (mine), I’m posting it here for you to read in its entirety vs. possibly in pieces elsewhere. Note that this is basically a tape status and collection of marketing and press release talking points, however you can get an idea of the current messaging, who is using tape and technology updates.

    Tape Data Storage in 2014 and looking towards 2015

    True to the nature of magnetic tape as a data storage medium, this is not a low latency small post, rather a large high-capacity bulk post or perhaps all you need to know about tape for now, or until next year. Otoh, if you are a tape fan, you can certainly take the memo from the tape folks, as well as visit their site for more info.

    From the tape storage council industry trade group:

    Today the Tape Storage Council issued its annual memo to highlight the current trends, usages and technology innovations occurring within the tape storage industry. The Tape Storage Council includes representatives of BDT, Crossroads Systems, FUJIFILM, HP, IBM, Imation, Iron Mountain, Oracle, Overland Storage, Qualstar, Quantum, REB Storage Systems, Recall, Spectra Logic, Tandberg Data and XpresspaX.  

    Data Growth and Technology Innovations Fuel Tape’s Future
    Tape Addresses New Markets as Capacity, Performance, and Functionality Reach New Levels

    Abstract
    For the past decade, the tape industry has been re-architecting itself and the renaissance is well underway. Several new and important technologies for both LTO (Linear Tape Open) and enterprise tape products have yielded unprecedented cartridge capacity increases, much longer media life, improved bit error rates, and vastly superior economics compared to any previous tape or disk technology. This progress has enabled tape to effectively address many new data intensive market opportunities in addition to its traditional role as a backup device such as archive, Big Data, compliance, entertainment and surveillance. Clearly disk technology has been advancing, but the progress in tape has been even greater over the past 10 years. Today’s modern tape technology is nothing like the tape of the past.

    The Growth in Tape  
    Demand for tape is being fueled by unrelenting data growth, significant technological advancements, tape’s highly favorable economics, the growing requirements to maintain access to data “forever” emanating from regulatory, compliance or governance requirements, and the big data demand for large amounts of data to be analyzed and monetized in the future. The Digital Universe study suggests that the world’s information is doubling every two years and much of this data is most cost-effectively stored on tape.

    Enterprise tape has reached an unprecedented 10 TB native capacity with data rates reaching 360 MB/sec. Enterprise tape libraries can scale beyond one exabyte. Enterprise tape manufacturers IBM and Oracle StorageTek have signaled future cartridge capacities far beyond 10 TBs with no limitations in sight.  Open systems users can now store more than 300 Blu-ray quality movies with the LTO-6 2.5 TB cartridge. In the future, an LTO-10 cartridge will hold over 14,400 Blu-ray movies. Nearly 250 million LTO tape cartridges have been shipped since the format’s inception. This equals over 100,000 PB of data protected and retained using LTO Technology. The innovative active archive solution combining tape with low-cost NAS storage and LTFS is gaining momentum for open systems users.

    Recent Announcements and Milestones
    Tape storage is addressing many new applications in today’s modern data centers while offering welcome relief from constant IT budget pressures. Tape is also extending its reach to the cloud as a cost-effective deep archive service. In addition, numerous analyst studies confirm the TCO for tape is much lower than disk when it comes to backup and data archiving applications. See TCO Studies section below.

    • On Sept. 16, 2013 Oracle Corp announced the StorageTek T10000D enterprise tape drive. Features of the T10000D include an 8.5 TB native capacity and data rate of 252 MB/s native. The T10000D is backward read compatible with all three previous generations of T10000 tape drives.
    • On Jan. 16, 2014 Fujifilm Recording Media USA, Inc. reported it has manufactured over 100 million LTO Ultrium data cartridges since its release of the first generation of LTO in 2000. This equates to over 53 thousand petabytes (53 exabytes) of storage and more than 41 million miles of tape, enough to wrap around the globe 1,653 times.
    • April 30, 2014, Sony Corporation independently developed a soft magnetic under layer with a smooth interface using sputter deposition, created a nano-grained magnetic layer with fine magnetic particles and uniform crystalline orientation. This layer enabled Sony to successfully demonstrate the world’s highest areal recording density for tape storage media of 148 GB/in2. This areal density would make it possible to record more than 185 TB of data per data cartridge.
    • On May 19, 2014 Fujifilm in conjunction with IBM successfully demonstrated a record areal data density of 85.9 Gb/in2 on linear magnetic particulate tape using Fujifilm’s proprietary NANOCUBIC™ and Barium Ferrite (BaFe) particle technologies. This breakthrough in recording density equates to a standard LTO cartridge capable of storing up to 154 terabytes of uncompressed data, making it 62 times greater than today’s current LTO-6 cartridge capacity and projects a long and promising future for tape growth.
    • On Sept. 9, 2014 IBM announced LTFS LE version 2.1.4 4 extending LTFS (Linear Tape File System) tape library support.
    • On Sept. 10, 2014 the LTO Program Technology Provider Companies (TPCs), HP, IBM and Quantum, announced an extended roadmap which now includes LTO generations 9 and 10. The new generation guidelines call for compressed capacities of 62.5 TB for LTO-9 and 120 TB for generation LTO-10 and include compressed transfer rates of up to 1,770 MB/second for LTO-9 and a 2,750 MB/second for LTO-10. Each new generation will include read-and-write backwards compatibility with the prior generation as well as read compatibility with cartridges from two generations prior to protect investments and ease tape conversion and implementation.
    • On Oct. 6, 2014 IBM announced the TS1150 enterprise drive. Features of the TS1150 include a native data rate of up to 360 MB/sec versus the 250 MB/sec native data rate of the predecessor TS1140 and a native cartridge capacity of 10 TB compared to 4 TB on the TS1140. LTFS support was included.
    • On Nov. 6, 2014, HP announced a new release of StoreOpen Automation that delivers a solution for using LTFS in automation environments with Windows OS, available as a free download. This version complements their already existing support for Mac and Linux versions to help simplify integration of tape libraries to archiving solutions.

    Significant Technology Innovations Fuel Tape’s Future
    Development and manufacturing investment in tape library, drive, media and management software has effectively addressed the constant demand for improved reliability, higher capacity, power efficiency, ease of use and the lowest cost per GB of any storage solution. Below is a summary of tape’s value proposition followed by key metrics for each:

    • Tape drive reliability has surpassed disk drive reliability
    • Tape cartridge capacity (native) growth is on an unprecedented trajectory
    • Tape has a faster device data rate than disk
    • Tape has a much longer media life than any other digital storage medium
    • Tape’s functionality and ease of use is now greatly enhanced with LTFS
    • Tape requires significantly less energy consumption than any other digital storage technology
    • Tape storage has  a much lower acquisition cost and TCO than disk

    Reliability. Tape reliability levels have surpassed HDDs. Reliability levels for tape exceeds that of the most reliable disk drives by one to three orders of magnitude. The BER (Bit Error Rate – bits read per hard error) for enterprise tape is rated at 1×1019 and 1×1017 for LTO tape. This compares to 1×1016 for the most reliable enterprise Fibre Channel disk drive.

    Capacity and Data Rate. LTO-6 cartridges provide 2.5 TB capacity and more than double the compressed capacity of the preceding LTO-5 drive with a 14% data rate performance boost to 160 MB/sec. Enterprise tape has reached 8.5 TB native capacity and 252 MB/sec on the Oracle StorageTek T10000D and 10 TB native capacity and 360 MB/sec on the IBM TS1150. Tape cartridge capacities are expected to grow at unprecedented rates for the foreseeable future.

    Media Life. Manufacturers specifications indicate that enterprise and LTO tape media has a life span of 30 years or more while the average tape drive will be deployed 7 to 10 years before replacement. By comparison, the average disk drive is operational 3 to 5 years before replacement.

    LTFS Changes Rules for Tape Access. Compared to previous proprietary solutions, LTFS is an open tape format that stores files in application-independent, self-describing fashion, enabling the simple interchange of content across multiple platforms and workflows. LTFS is also being deployed in several innovative “Tape as NAS” active archive solutions that combine the cost benefits of tape with the ease of use and fast access times of NAS. The SNIA LTFS Technical Working Group has been formed to broaden cross–industry collaboration and continued technical development of the LTFS specification.

    TCOStudies. Tape’s widening cost advantage compared to other storage mediums makes it the most cost-effective technology for long-term data retention. The favorable economics (TCO, low energy consumption, reduced raised floor) and massive scalability have made tape the preferred medium for managing vast volumes of data. Several tape TCO studies are publicly available and the results consistently confirm a significant TCO advantage for tape compared to disk solutions.

    According to the Brad Johns Consulting Group, a TCO study for an LTFS-based ‘Tape as NAS’ solution totaled $1.1M compared with $7.0M for a disk-based unified storage solution.  This equates to a savings of over $5.9M over a 10-year period, which is more than 84 percent less than the equivalent amount for a storage system built on a 4 TB hard disk drive unified storage system.  From a slightly different perspective, this is a TCO savings of over $2,900/TB of data. Source: Johns, B. “A New Approach to Lowering the Cost of Storing File Archive Information,”.

    Another comprehensive TCO study by ESG (Enterprise Strategies Group) comparing an LTO-5 tape library system with a low-cost SATA disk system for backup using de-duplication (best case for disk) shows that disk deduplication has a 2-4x higher TCO than the tape system for backup over a 5 year period. The study revealed that disk has a TCO of 15x higher than tape for long-term data archiving.

    Select Case Studies Highlight Tape and Active Archive Solutions
    CyArk Is a non-profit foundation focused on the digital preservation of cultural heritage sites including places such as Mt. Rushmore, and Pompeii. CyArk predicted that their data archive would grow by 30 percent each year for the foreseeable future reaching one to two petabytes in five years. They needed a storage solution that was secure, scalable, and more cost-effective to provide the longevity required for these important historical assets. To meet this challenge CyArk implemented an active archive solution featuring LTO and LTFS technologies.

    Dream Works Animation a global Computer Graphic (CG) animation studio has implemented a reliable, cost-effective and scalable active archive solution to safeguard a 2 PB portfolio of finished movies and graphics, supporting a long-term asset preservation strategy. The studio’s comprehensive, tiered and converged active archive architecture, which spans software, disk and tape, saves the company time, money and reduces risk.

    LA Kings of the NHL rely extensively on digital video assets for marketing activities with team partners and for its broadcast affiliation with Fox Sports. Today, the Kings save about 200 GB of video per game for an 82 game regular season and are on pace to generate about 32-35 TB of new data per season. The King’s chose to implement Fujifilm’s Dternity NAS active archive appliance, an open LTFS based architecture. The Kings wanted an open source archiving solution which could outlast its original hardware while maintaining data integrity. Today with Dternity and LTFS, the Kings don’t have to decide what data to keep because they are able to cost-effectively save everything they might need in the future. 

    McDonald’s primary challenge was to create a digital video workflow that streamlines the management and distribution of their global video assets for their video production and post-production environment. McDonald’s implemented the Spectra T200 tape library with LTO-6 providing 250 TB of McDonald’s video production storage. Nightly, incremental backup jobs store their media assets into separate disk and LTO- 6 storage pools for easy backup, tracking and fast retrieval. This system design allows McDonald’s to effectively separate and manage their assets through the use of customized automation and data service policies.

    NCSA employs an Active Archive solution providing 100 percent of the nearline storage for the NCSA Blue Waters supercomputer, which is one of the world’s largest active file repositories stored on high capacity, highly reliable enterprise tape media. Using an active archive system along with enterprise tape and RAIT (Redundant Arrays of Inexpensive Tape) eliminates the need to duplicate tape data, which has led to dramatic cost savings.

    Queensland Brain Institute (QBI) is a leading center for neuroscience research.  QBI’s research focuses on the cellular and molecular mechanisms that regulate brain function to help develop new treatments for neurological and mental disorders.  QBI’s storage system has to scale extensively to store, protect, and access tens of terabytes of data daily to support cutting-edge research.  QBI choose an Oracle solution consisting of Oracle’s StorageTek SL3000 modular tape libraries with StorageTek T10000 enterprise tape drives.   The Oracle solution improved QBI’s ability to grow, attract world-leading scientists and meet stringent funding conditions.

    Looking Ahead to 2015 and Beyond
    The role tape serves in today’s modern data centers is expanding as IT executives and cloud service providers address new applications for tape that leverage its significant operational and cost advantages. This recognition is driving investment in new tape technologies and innovations with extended roadmaps, and it is expanding tape’s profile from its historical role in data backup to one that includes long-term archiving requiring cost-effective access to enormous quantities of stored data. Given the current and future trajectory of tape technology, data intensive markets such as big data, broadcast and entertainment, archive, scientific research, oil and gas exploration, surveillance, cloud, and HPC are expected to become significant beneficiaries of tape’s continued progress. Clearly the tremendous innovation, compelling value proposition and development activities demonstrate tape technology is not sitting still; expect this promising trend to continue in 2015 and beyond. 

    Visit the Tape Storage Council at tapestorage.org

    What this means and summary

    Like it not tape is still alive being used along with the technology evolving with new enhancements as outlined above.

    Good to see the tape folks doing some marketing to get their story told and heard for those who are still interested.

    Does that mean I still use tape?

    Nope, I stopped using tape for local backups and archives well over a decade ago using disk to disk and disk to cloud.

    Does that mean I believe that tape is dead?

    Nope, I still believe that for some organizations and some usage scenarios it makes good sense, however like with most data storage related technologies, it’s not a one size or type of technology fits everything scenario value proposition.

    On a related note for cloud and object storage, visit www.objectstoragecenter.com

    Ok, nuff said, for now…

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Cloud Conversations: Revisiting re:Invent 2014 and other AWS updates

    server storage I/O trends

    This is part one of a two-part series about Amazon Web Services (AWS) re:Invent 2014 and other recent cloud updates, read part two here.

    Revisiting re:Invent 2014 and other AWS updates

    AWS re:Invent 2014

    A few weeks ago I attended Amazon Web Service (AWS) re:Invent 2014 in Las Vegas for a few days. For those of you who have not yet attended this event, I recommend adding it to your agenda. If you have interest in compute servers, networking, storage, development tools or management of cloud (public, private, hybrid), virtualization and related topic themes, you should check out AWS re:invent.

    AWS made several announcements at re:invent including many around development tools, compute and data storage services. One of those to keep an eye on is cloud based Aurora relational database service that complement existing RDS tools. Aurora is positioned as an alternative to traditional SQL based transactional databases commonly found in enterprise environments (e.g. SQL Server among others).

    Some recent AWS announcements prior to re:Invent include

    AWS vCenter Portal

    Using the AWS Management Portal for vCenter adds a plug-in within your VMware vCenter to manage your AWS infrastructure. The vCenter for AWS plug-in includes support for AWS EC2 and Virtual Machine (VM) import to migrate your VMware VMs to AWS EC2, create VPC (Virtual Private Clouds) along with subnet’s. There is no cost for the plug-in, you simply pay for the underlying AWS resources consumed (e.g. EC2, EBS, S3). Learn more about AWS Management Portal for vCenter here, and download the OVA plug-in for vCenter here.

    AWS re:invent content


    AWS Andy Jassy (Image via AWS)

    November 12, 2014 (Day 1) Keynote (highlight video, full keynote). This is the session where AWS SVP Andy Jassy made several announcements including Aurora relational database that complements existing RDS (Relational Data Services). In addition to Andy, the key-note sessions also included various special guests ranging from AWS customers, partners and internal people in support of the various initiatives and announcements.


    Amazon.com CTO Werner Vogels (Image via AWS)

    November 13, 2014 (Day 2) Keynote (highlight video, full keynote). In this session, Amazon.com CTO Werner Vogels appears making announcements about the new Container and Lambda services.

    AWS re:Invent announcements

    Announcements and enhancements made by AWS during re:Invent include:

    • Key Management Service (KMS)
    • Amazon RDS for Aurora
    • Amazon EC2 Container Service
    • AWS Lambda
    • Amazon EBS Enhancements
    • Application development, deployed and life-cycle management tools
    • AWS Service Catalog
    • AWS CodeDeploy
    • AWS CodeCommit
    • AWS CodePipeline

    Key Management Service (KMS)

    Hardware security module (HSM) based key managed service for creating and control of encryption keys to protect security of digital assets and their keys. Integration with AWS EBS and others services including S3 and Redshift along with CloudTrail logs for regulatory, compliance and management. Learn more about AWS KMS here

    AWS Database

    For those who are not familiar, AWS has a suite of database related services including SQL and no SQL based, simple to transactional to Petabyte (PB) scale data warehouses for big data and analytics. AWS offers the Relational Database Service (RDS) which is a suite of different database types, instances and services. RDS instance and types include SimpleDB, MySQL, Postgress, Oracle, SQL Server and the new AWS Aurora offering (read more below).  Other little data database and big data repository related offerings include DynamoDB (a non-SQL database), ElasticCache (in memory cache repository) and Redshift (large-scale data warehouse and big data repository).

    In addition to database services offered by AWS, you can also combine various AWS resources including EC2 compute, EBS and other storage offerings to create your own solution. For example there are various Amazon Machine Images (AMI’s) or pre-built operating systems and database tools available with EC2 as well as via the AWS Marketplace , such as MongoDB and Couchbase among others. For those not familiar with MongoDB, Couchbase, Cassandra, Riak along with other non SQL or alternative databases and key value repositories, check out Seven Databases in Seven Weeks in my book review of it here.

    Seven Databases book review
    Seven Databases in Seven Weeks and NoSQL movement available from Amazon.com

    Amazon RDS for Aurora

    Aurora is a new relational database offering part of the AWS RDS suite of services. Positioned as an alternative to commercial high-end database, Aurora is a cost-effective database engine compatible with MySQL. AWS is claiming 5x better performance than standard MySQL with Aurora while being resilient and durable. Learn more about Aurora which will be available in early 2015 and its current preview here.

    Amazon EC2 C4 instances

    AWS will be adding a new C4 instance as a next generation of EC2 compute instance based on Intel Xeon E5-2666 v3 (Haswell) processors. The Intel Xeon E5-2666 v3 processors run at a clock speed of 2.9 GHz providing the highest level of EC2 performance. AWS is targeting traditional High Performance Computing (HPC) along with other compute intensive workloads including analytics, gaming, and transcoding among others. Learn more AWS EC2 instances here, and view this Server and StorageIO EC2, EBS and associated AWS primer here.

    Amazon EC2 Container Service

    Containers such as those via Docker have become popular to support developers rapidly build as well as deploy scalable applications. AWS has added a new feature called EC2 Container Service that supports Docker using simple API’s. In addition to supporting Docker, EC2 Container Service is a high performance scalable container management service for distributed applications deployed on a cluster of EC2 instances. Similar to other EC2 services, EC2 Container Service leverages security groups, EBS volumes and Identity Access Management (IAM) roles along with scheduling placement of containers to meet your needs. Note that AWS is not alone in adding container and docker support with Microsoft Azure also having recently made some announcements, learn more about Azure and Docker here. Learn more about EC2 container service here and more about Docker here.

    Docker for smarties

    Continue reading about re:Invent 2014 and other recent AWS enhancements here in part two of this two-part series.

    Ok, nuff said (for now)

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved