Tag: database

Posted on

Software Defined Cloud Bulk Object Storage News Trends

Cloud, Software Defined, Scale-Out, Object Storage News Trends

server storage I/O trends

Updated 1/21/2018

Welcome to the Cloud, Big Data, Software Defined, Bulk and Object Storage News Trends page.

This page contains various links pertaining to Software Defined, scale-out, Cloud, Bulk and Object Storage News Trends and resources. Other resources pertaining to Software Defined, scale-out, Cloud, Bulk and Object Storage include:

  • www.objectstoragecenter.com
  • Software Defined, Cloud, Bulk and Object Storage Fundamentals
  • Software Defined Data Infrastructure Essentials book (CRC Press)

    • Software defined, Bulk, Cloud and Object Storage exist to support expanding and diverse application data demands. There are various types of cloud, bulk and object storage including public services such as Amazon Web Services (AWS) Simple Storage Service (S3), Google, Microsoft Azure, IBM Softlayer, Rackspace among many others. There are also solutions for hybrid and private deployment from Cisco, DDN, Dell EMC, Fujitsu, HDS, HPE, IBM, NetApp, Noobaa, OpenStack, Quantum, Rackspace, Scality, Seagate, Spectra, Storpool, Suse, Swift and WD among others.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    Cloud products and services among others, along with associated data infrastructures including object storage, file systems, repositories and access methods are at the center of bulk, big data, big bandwidth and little data initiatives on a public, private, hybrid and community basis. After all, not everything is the same in cloud, virtual and traditional data centers or information factories from active data to in-active deep digital archiving.



    Via InfoStor: Object Storage Is In Your Future
    Via FujiFilm IT Summit: Software Defined Data Infrastructures (SDDI) and Hybrid Clouds
    Via ChannelPartnersOnline: Selling Software-Defined Storage: Not All File Systems Are the Same
    Via ITProPortal: IBM kills off its first cloud storage platform
    Via MultiChannel: After ditching cloud business, Verizon inks Virtual Network Services deal with Amazon
    Via ITBusinessEdge: Time to Rein in Cloud Storage
    Via BusinessInsider: Kodiak Data Brings Enterprise Storage and Database Services to the Cloud Era with StorageCloud
    Via SerchCloudStorge: Ctera Networks’ file-sharing services gain intelligent cache
    Via StorageIOblog: Who Will Be At Top Of Storage World Next Decade?

    cloud object storage

    The following resources provide additional information about big data, bulk, software defined, cloud and object storage.

    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via CloudComputingAdmin: Cloud Storage Decision Making
    Via Wservernews: Cloud storage considerations
    Via PRNewsWire: Scality Accelerates Multi-Cloud Adoption with New Open-Source Release
    Via EnterpriseStorageForum: Top 10 Tips for Software-Defined Storage Deployment
    Via MultiChannel: Verizon Digital Media Services now offers integrated Microsoft Azure Storage
    Via BroadCstingCable: Quantum’s New StorNext 6 Delivers Unparalleled Combination of Best-in-Class 4K Performance and Advanced Data Management
    Via ShootOnLine: Morro Data Accelerates Media Collaboration With Cache and Sync Appliance
    Via TechCrunch: Backblaze drops download pricing for its B2 storage platform by 60%
    Via PRNewsWire: FileCloud Online Launches an All-In-One File Share, Sync and Backup SaaS Offering in a Challenge to Dropbox and Box

    cloud object storage

    Via HITInfrastructure: Benefits of Object Storage to Health IT Infrastructure
    Via ITBusinessedge: Growing Diversity of Cloud Storage Solutions
    Via Telecompaper: Interoute launches storage zones in Switzerland
    Via TradeCalls:
    Object Storage Market Size, Status and Forecast 2022
    Via OpenStack: Creating object storage container causes user to be logged out, OpenStack Dashboard (Horizon)
    Via RethinkResearch: Nokia claims a 40% cut in cloud DVR Capex, using new Skim Storage
    Via Cloudwards: A Beginner’s Guide on How Cloud Backup With Microsoft Azure Works
    Via ITProPortal: Lack of understanding of data inhibits move to the cloud

    cloud object storage

    Via InfoStor:Cloud Storage Concerns, Considerations and Trends
    Via StreetInsider: IBM Announces New Pricing Model for Cloud Storage and NetApp Object Storage Partnership

    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via ComputerWeekly: Storage 101: Object storage vs block vs file
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via Yahoo Finance: SwiftStack Joins Google Cloud Technology Partner Program
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look

    Via StorageIOblog: HPE Continues Buying Into Server Storage I/O Data Infrastructures
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via TheStack:Google to launch nine new data centres over three new cloud regions
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via DigitalJournal:Global Business Cloud Storage market 2017 share, Segmentation and Industrial Overview forecasts to 2022
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via NextPlatform:How Microsoft Fell Hard for FPGAs
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via NextPlatform:Google Courts Enterprise For Cloud Platform

    cloud object storage

    Via IBM RedBooks: Cloud Object Storage as a
    Service – IBM Cloud Object Storage from Theory to Practice
    Via VentureBeat: Microsoft confirms Azure storage issues around the world
    Via Microsoft Azure: Azure status history
    Via NextPlatform: Windows Server Comes To ARM Chips, But Only For Azure
    Via NextPlatform: ARM And AMD X86 Server Chips Get Mainstream Lift From Microsoft
    Via ITBusinessEdge: Google Expands Enterprise Cloud Service Footprint
    Via ChannelWorld: Google announces three new cloud regions, contract discounts
    Via AWS: AWS Service Health Status
    Via AWS (S3 Feb 28 2017 Port mortem): Summary of the Amazon S3 Service Disruption
    Via SearchCloudStorage: Open source object storage startup OpenIO adds hardware

    cloud object storage

    Via InfoStor: Comments on Cloud Storage Pros and Cons
    Via CRN: StorageCraft Acquires Exablox
    Via TVtechglobal: Signiant CEO: The media industry isnt just one monolithic thing
    Via Computerweekly: Using the cloud as a storage tier
    Via SearchCloudStorage: OpenIO joins object storage cloud scrum
    Via CRN: Dell Partners With Scality On High-Density Object Storage
    Via BroadwayWorld: The World’s Smallest Object Storage Solution, ‘Ceph on Olive’, is Released
    Via SmartDataCollective: Why You Should Be Securing Big Data In The Cloud?
    Via Geekwire: Feds certify Amazon and Microsoft clouds to handle sensitive government data
    Via NewsFactor: SoftNAS Introduces File Services for NetApp StorageGRID
    Via Redmondmag: AWS Cloud Storage Service Turns 10
    Via DropBox Blog: Scaling to exabytes and beyond
    Via Computerworld: Dropbox quits Amazon cloud, takes back 500 PB of data
    Via HPCwire: Seagate Sets Sights on Broader HPC Market

    cloud object storage

    Via Channelnomics.com: Why Samsung’s Joyent buyout could frame its future
    Via GeekyGadgets: Google Drive Cloud Storage Is Now A Little Easier To Organize Thanks To New Tools
    Via ZDnet: Google doubles Cloud Compute local SSD capacity: Now it’s 3TB per VM
    Via AndroidPolice: Barracuda Is Shutting Down The Copy Cloud Storage Service On May 1st
    Via StorageSoup: EMC expands Elastic Cloud (object) Storage
    Via Cloud Review: Access Multiple Cloud Apps from a Unified Interface
    Via CyberSecurity: Cybersecurity and Accessible Storage Trends
    Via eWeek: Cloud Storage Spending Poised for Increase in 2016
    Via InfoStor: Object Storage Is In Your Future
    Via BioScienceTechnology: Six Reasons to Add Object Storage to Your Genomics Lexicon
    Via TechCrunch: Backblaze’s Low-Cost Cloud Storage Service Comes Out Of Beta
    Via CloudComputingAdmin: Cloud Storage Decision Making
    COSbench for cloud and object storage (Various): White Paper (CMU), PDF overview presentation, Get the bits (software) from Github
    Via AWS Blogs: AWS Storage Update – New Lower Cost S3 Storage Option & Glacier Price Reduction
    Via Techcrunch: IBM Acquires Cleversafe To Accelerate Hybrid Cloud Storage Strategy

    cloud object storage

    Via Techcrunch: Developers Can Now Import Data To Google Cloud Storage By Mailing In Their Hard Drives, Tapes And USB Flash Drives
    Via vCloud: VMware vCloud Air Object Storage
    Via SearchStorage: SwiftStack adds new features to object storage software solution
    Via TheVarGuy: Vendor consortium create Ceph advisory board
    Via OpenStack: Whats new in OpenStack Liberty
    Via Microsoft Azure: File access cloud storage in addition to blobs and object storage
    Via AWS: AWS Storage Update – New Lower Cost S3 Storage Option & Glacier Price Reduction
    Via VMware Blogs: Showcasing VMware vCloud Air Object Storage powered by EMC at EMC World!
    Via Venturebeat: Backblaze launches B2 object storage service at 1/4 of the cost of Amazon S3
    Via Techcrunch: Wuala, The Secure P2P Cloud Storage Service Acquired By LaCie, To Shut Down
    Via ChannelworldIN: 13 firms join to open up Ethernet, object-storage to everyone
    Via ITportal: The elephant in the room: The trust deficit issue with public clouds
    Via VCpost: Dropbox latest update: Cloud storage service provider now lets you drag and drop URLs with files
    Via NetworkAsia: HP’s new converged appliance offers a path to hybrid cloud
    Via Techtimes: Google Files For Patent To Search Your Desktop Files, Apps And Cloud Storage

    cloud object storage

    Via CloudTweaks: Questions To Ask Every Cloud Storage Provider
    Via MarketWatch: Linux Foundation Brings Together Industry Leaders to Advance Cloud Object Storage Technologies
    Via CBR: New Kinetic platform targets open source object storage
    Via Techcrunch: Seagate, Toshiba, Scality And Others Launch Kinetic Open Storage Platform For Smart, Low-Overhead Cloud Object Storage
    Via StorageIOblog: July 2015 Newsletter Focus on Content Solution Platforms
    Via ITnewsline: Hitachi Data Systems Announces New Enhancements to the Hitachi Content Platform Portfolio
    Via ITbusinessnet: Caringo Provides Only Software-Based Web-Scale Object Storage for Symantec Enterprise Vault
    Via ITbusinessnet: Caringo Provides Only Software-Based Web-Scale Object Storage for Symantec Enterprise Vault
    Via TopTechNews: Dropbox Adds USB Keys To Boost Security
    Via ZDnet: SUSE brings its management software to the cloud
    Via Forbes: Jumptuit Roadmap Video: A New Search Bar For Your Data At Facebook, Google and Dropbox
    Via Forbes: Panzura Global File System — Disrupting the Storage Industry
    Via TalkinCloud: Barracuda Launches Cloud Storage and Share Service CudaDrive
    Via ITnews: Researchers debut rival to AWS cloud storage
    Via NetworkAsia.net: HP launches new scale-out and scale-up Compute portfolios

    cloud object storage

    Via Server StorageIO: May / June 2015 Newsletter Focus on server storage packaging
    Via Server StorageIO: April 2015 Newsletter Focus on Cloud and Object storage
    Via StudioDaily.com: At NAB, Vendors and Users Both Got Serious About 4K Workflow
    Via OpenSource.com: The unofficial guide to OpenStack Summit Vancouver
    Via InsideHPC: Deploying Hadoop on Lustre Storage: Lessons Learned and Best Practices
    Via Yahoo Engineering Blog: Yahoo Cloud Object Store – Object Storage at Exabyte Scale
    Via the Platform: Inside The Ceph Exascale Storage At Yahoo
    Via Swift Summit: Taking the Mystery out of Erasure Codes: A Swift Implementation
    Via GovTech: Storage Costs Cloud Police Cam Issue
    Via BostonHerald: Booting Up: Storage costs cloud police cam issue
    Via StorageIOblog: S3motion Buckets Containers Objects AWS S3 Cloud and EMCcode (with podcast)
    Via StorageIOblog: If focused on cost you might miss other cloud storage benefits
    Via ComputerWorld: Amazon offers network file storage in the cloud
    Via ComputerWeekly: HGST marries helium HDDs and Himalaya in object storage Active Archive
    Via GoogleCloudPlatform Blog: GCS Nearline Online storage at Offline price

    cloud object storage

    Via MarketWatch: Global Data Center Provider CyrusOne Announces Direct Connectivity to Google Cloud Platform
    Via PRNewsWire: Quantum Announces New Archive Solutions Designed To Reduce Unstructured Data Storage Costs
    Via Enterprise Storage Forum: Lustre buying guide
    Via StorageIOblog: AWS S3 Cross Region Replication storage enhancements
    Via StreetInsider: Western Digital (WDC) to Acquire Object Storage Software Amplidata
    Via Enterprise Storage Forum: Dell Invests in Object Storage Startup Exablox
    Via Enterprise Storage Forum: Introducing s3motion (S3 and object access docker based appliance)
    Via Computerworld: Quantum enhances their cloud and object storage management with new StorNext software version
    Via HealthIndustry: 2014 IDC MarketScape perspectives
    Via MarketWired: ScaleOut Software Releases Version 5.2 of Its In-Memory Computing Platform
    Via CRN: HP Inks Global Reseller Agreement With Object Storage Startup Scality
    Via NetApp: NetApp Introduces Software-Defined Object Storage for the Hybrid Cloud
    Via ZDnet: NetApp intros software-defined object storage for hybrid cloud adoption
    Via SearchCloudStorage: Are more than five nines of availability really possible?

    cloud object storage

    Via SearchCloudStorage: How do primary storage clouds and cloud for backup differ?
    Via SearchCloudStorage: What should I consider when using SSD cloud?
    Via SearchCloudStorage: Given outages, are you concerned with the security of the cloud?
    Via SearchCloudStorage: Is the cost of cloud storage really cheaper than traditional storage?
    Via HostReview: Panzura Collaborates With Microsoft Azure To Transform Cloud Storage Into A Globally Distributed NAS
    Via ITBusinessNet: Caringo Awarded Patent for Erasure Coding, Replication in Storage Cluster
    Via PRWeb: Western Digital Capital Makes Strategic Investment in Amplidata
    Via ITwire: Dovecot in for the long haul
    Via InfoStor: Top Tips for Object Storage
    Via Enterprise Storage ForumObject Storage Buying Guide
    Via Enterprise Networking Planet: Cloud Gateways 101
    Via eWeek: HP Unveils Smaller, Less Costly Helion Cloud Platform
    Via PCadvisor: CA Technologies, AWS and Riverbed bring mainframe Cloud to big four banks
    Via CMS wire: The Enterprise Cloud is Finally Moving Beyond Storage
    Via Edtech: Microsoft 365’s Massive Cloud Storage Is ‘Icing on Cake’ to University’s Upgrade

    cloud object storage

    Via CRN: Quantum Acquires Symform, Plans To Bring Cloud-Based Backups To Enterprise
    Via BusinessWire: OpenStack Object Storage (Swift) Adds Storage Policies for Enterprise Users and Cloud Operators
    Via SearchStorage: EMC ViPR and Data Services (including Object storage)
    Via Enterprise Storage Forum: Cloud Appliance Buying Guide
    Via Datacenter Dynamics: Verizon, Amplidata and Cloud Object Storage
    Via ZDnet: IBM drops SoftLayer cloud storage costs closer to its rivals
    Via YellowBricks: How long will VSAN rebuilding take with large drives? via Duncan Epping (@DuncanYB) and his Yellow-Bricks site. Note that VMware VSAN uses one of many different forms of object storage (e.g. its not S3 or Swift or CDMI) rather a means for storing VMware vSphere entities and items (e.g. various types of objects)
    Via Computerweekly: Object storage: An architecture for the petabyte era? Via Simon Robinson (@simonrob451) appearing at
    Via MarketWatch: Western Digital/HGST announce alternative to Seagate Kinetic HDD for object storage
    Via EMC: EMC Announces Elastic Cloud Storage Appliance via EMC
    Via TechCrunch: RedHat acquires InkTank provider of Ceph services
    Via CompueterWeekly: DataDirect Networks (DDN) WOS enhancements

    cloud object storage

    Hybrid Clouds: Bridging the Gap Between Public and Private Environments (Recorded for Playback: BrightTalk Webinar)
    Book review: Rethinking Enterprise Storage and Hybrid Clouds by Microsoft/Storsimple and Marc Farley aka (@Gofarley (or what ever name he is currently using ;))
    Goodbye 2013, hello 2014 predictions present and future
    Enterprises should just ignore OpenStack via Virtualizedgeek aka Keith Townsend (@virtualizedgeek)
    OpenStack Swift Object Storage for the Cloud Grows via The Var Guy (@thevarguy) aka Christopher Tozzi
    For Seagate Kinetic to fly…it’s gonna take an ecosystem (Via @Seagate The Storage Effect Blog)
    Object-defined storage (it’s not about object storage) (Via Enrico Signoretti @Juku)
    Seagate Kinetic Cloud and Object Storage I/O platform
    Why You Should Consider Object Storage (Via 21CIT)
    Object Storage Is in Your Future, Even if You Use Files (Via 21CIT)
    The Right Storage Option Is Important for Big Data Success (Via FedTech)
    Hybrid cloud storage with an object storage back end(Via Marc Farley aka @MicroFarley at Microsoft Technet Blogs)
    cloud object storage

    Via SearchStorage: EMC ViPR aimed at service providers, enterprise storage (Video)
    EMC ViPR virtual physical object and software defined storage
    EMC ViPR software defined object storage part II
    EMC ViPR software defined object storage part III
    The rise of cloud-integrated storage and EMC’s ViPR (Via Marc Farley of Microsoft aka @MicroFarley)
    ViPR: Under the covers (Via Chad Sakacc of EMC aka @sakacc)
    Introducing EMC ViPR (Via Chuck Hollis aka @Chuckhollis of EMC)
    EMC Tells their SDS story (e.g. ViPR) (Via the Solutions Architect aka Michael Letschin aka @mletschin)
    Object Paucity (Via Martin Glassborow aka @storagebod)
    Object Storage: You can call it NAS 2.0! (Via Enrico Signoretti aka @esignoretti)
    Object storage is the platform, gateways are the solution (Via Enrico Signoretti aka @esignorettii)
    Files Vs. Objects: the difference is in the value (Via Enrico Signoretti aka @esignorettii)
    Conversation with Justin Stottlemyer of Shutterfly and object storage discussion (Pod cast)

    cloud object storage

    Cloud conversations: AWS EBS, Glacier and S3 overview
    Ceph Day Amsterdam 2012 (Object and cloud storage)
    Ceph Day in Amsterdam and Sage Weil on Object Storage (Podcast)
    Mr. Backup (Curtis Preston) goes back to Ceph School (Video and Podcast)
    AWS (Amazon) storage gateway, first, second and third impressions

    cloud object storage

    Where to learn more

    The following resources provide additional information about big data, bulk, software defined, cloud and object storage.

    Click here to view (and hear) more content including cloud and object storage fundamentals.
    Via StorageIOblog: AWS EFS Elastic File System (Cloud NAS) First Preview Look
    Via InfoStor: Cloud Storage Concerns, Considerations and Trends
    Via InfoStor: Object Storage Is In Your Future
    Via Server StorageIO: April 2015 Newsletter Focus on Cloud and Object storage
    Via StorageIOblog: AWS S3 Cross Region Replication storage enhancements
    Cloud conversations: AWS EBS, Glacier and S3 overview
    AWS (Amazon) storage gateway, first, second and third impressions

    Videos and podcasts at storageio.tv also available via Applie iTunes.

    Human Face of Big Data
    Human Face of Big Data (Book review)

    Seven Databases in Seven weeks
    Seven Databases in Seven Weeks (Book review)

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    Wrap up and summary

    Object and cloud storage are in your future, the questions are when, where, with what and how among others. Watch for more content and links to be added here as well as on the main object storage center page including posts, presentations, pod casts, perspectives along with services and product solutions profiles.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Kevin Closson discusses SLOB Server CPU I/O Database Performance benchmarks

    Slilly Little Oracle Benchmark (SLOB) Database Server I/O Podcast

    server storage I/O trends

    In this Server StorageIO podcast episode, I am joined by @Kevinclosson who is an Oracle (along with other Databases) performance expert and creator of the Silly Little Oracle Benchmark (SLOB) tool. Not surprising our data infrastructure discussion involves server CPU, software, I/O, storage, performance, software, tools, best practices, fundamental tradecraft skills among other items.

    server storage I/O performance

    Kevin has been involved in database performance (and porting) optimization for decades which means CPU server, memory, I/O and storage issues, resources and tuning. Part of server, storage I/O a tuning is understanding the workloads, also the demands of software such as databases along with how they use CPU and its impact on resources. This means that somewhere in the technology stack, server CPUs are still needed, even in serverless environments.

    We also discuss metrics, gaining insight to resources uses, what they mean including how CPU wait may be costing your lost productivity with overhead, as well as benchmarks, simulations, and related themes. Check out Kevins website www.kevinclosson.net to learn more about Oracle, Databases, SLOB, tools and other content. Listen to the podcast discussion here (42 minutes) as well as on iTunes.

    Where to learn more

    Learn more about Oracle, Database Performance, Benchmarking along with other tools via the following links:

    What this all means and wrap-up

    Check out my discussion here with Kevin Closson where you may have some Dejavu, or learn something new on server, storage I/O, database performance, software, benchmark workloads as well as much more. Also available on 

    Ok, nuff said for now…

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert (and vSAN). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

    Posted on

    Cloud and Object storage are in your future, what are some questions?

    Cloud and Object storage are in your future, what are some questions?

    server storage I/O trends

    IMHO there is no doubt that cloud and object storage are in your future, what are some questions?

    Granted, what type of cloud and object storage or service along with for work or entertainment are some questions.

    Likewise, what are your cloud and object storage concerns (assuming you already have heard the benefits)?

    Some other questions include when, where for different applications workload needs, as well as how and with what among others.

    Keep in mind that there are many aspects to cloud storage and they are not all object, likewise, there are many facets to object storage.

    Recently I did a piece over at InfoStor titled Cloud Storage Concerns, Considerations and Trends that looks at the above among other items including:

    • Is cloud storage cheaper than traditional storage?
    • How do you access cloud object storage from legacy block and file applications?
    • How do you implement on-site cloud storage?
    • Is enterprise file sync and share (EFSS) safe and secure?
    • Does cloud storage need to be backed up and protected?
    • What geographic location requirements or regulations apply to you?

    When it comes to cloud computing and, in particular, cloud storage, context matters. Conversations are necessary to discuss concerns, as well as discuss various considerations, options and alternatives. People often ask me questions about the best cloud storage to use, concerns about privacy, security, performance and cost.

    Some of the most common cloud conversations topics involve context :

    • Public, private or hybrid cloud; turnkey subscription service or do it yourself (DIY)?
    • Storage, compute server, networking, applications or development tools?
    • Storage application such as file sync and share like Dropbox?
    • Storage resources such as table, queues, objects, file or block?
    • Storage for applications in the cloud, on-site or hybrid?

    Continue reading Cloud Storage Concerns, Considerations and Trends over at InfoStor.

    Where To Learn More

    Additional related content can be found at:

    What This All Means

    As I mentioned above, cloud and object storage are in your future, granted your future may not rely on just cloud or object storage. Take a few minutes to check out some of the conversation topics, tips and trends in my piece over at InfoStor Cloud Storage Concerns, Considerations and Trends along with more material at www.objectstoragecenter.com.

    Btw, what are your questions, comments, concerns, claims or caveats as part of cloud and object storage conversations?

    Ok, nuff said, for now…

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, vSAN and VMware vExpert. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

    Posted on

    Gaining Server Storage I/O Insight into Microsoft Windows Server 2016

    Server Storage I/O Insight into Microsoft Windows Server 2016

    server storage I/O trends
    Updated 12/8/16

    In case you had not heard, Microsoft announced the general availability (GA, also known as Release To Manufacturing (RTM) ) of the newest version of its Windows server operating system aka Windows Server 2016 along with System Center 2016. Note that as well as being released to traditional manufacturing distribution mediums as well as MSDN, the Windows Server 2016 bits are also available on Azure.

    Microsoft Windows Server 2016
    Windows Server 2016 Welcome Screen – Source Server StorageIOlab.com

    For some this might be new news, or a refresh of what Microsoft announced a few weeks ago (e.g. the formal announcement). Likewise, some of you may not be aware that Microsoft is celebrating WIndows Server 20th Birthday (read more here).

    Yet for others who have participated in the public beta aka public technical previews (TP) over the past year or two or simply after the information coming out of Microsoft and other venues, there should not be a lot of surprises.

    Whats New With Windows Server 2016

    Microsoft Windows Server 2016 Desktop
    Windows Server 2016 Desktop and tools – Source Server StorageIOlab.com

    Besides a new user interface including visual GUI and Powershell among others, there are many new feature functionalities summarized below:

    • Enhanced time-server with 1ms accuracy
    • Nano and Windows Containers (Linux via Hyper-V)
    • Hyper-V enhanced Linux services including shielded VMs
    • Simplified management (on-premisess and cloud)
    • Storage Spaces Direct (S2D) and Storage Replica (SR) – view more here and here


    Storage Replica (SR) Scenarios including synchronous and asynchronous – Via Microsoft.com

    • Resilient File System aka ReFS (now default file system) storage tiering (cache)
    • Hot-swap virtual networking device support
    • Reliable Change Tracking (RCT) for faster Hyper-V backups
    • RCT improves resiliency vs. VSS change tracking
    • PowerShell and other management enhancements
    • Including subordinated / delegated management roles
    • Compliment Azure AD with on premise AD
    • Resilient/HA RDS using Azure SQL DB for connection broker
    • Encrypted VMs (at rest and during live migration)
    • AD Federation Services (FS) authenticate users in LDAP dir.
    • vTPM for securing and encrypting Hyper-V VMs
    • AD Certificate Services (CS) increase support for TPM
    • Enhanced TPM support for smart card access management
    • AD Domain Services (DS) security resiliency for hybrid and mobile devices

    Here is a Microsoft TechNet post that goes into more detail of what is new in WIndows Server 2016.

    Free ebook: Introducing Windows Server 2016 Technical Preview (Via Microsoft Press)

    Check out the above free ebook, after looking through it, I recommend adding it to your bookshelf. There are lots of good intro and overview material for Windows Server 2016 to get you up to speed quickly, or as a refresh.

    Storage Spaces Direct (S2D) CI and HCI

    Storage Spaces Direct (S2D) builds on Storage Spaces that appeared in earlier Windows and Windows Server editions. Some of the major changes and enhancements include ability to leverage local direct attached storage (DAS) such as internal (or external) dedicated NVMe, SAS and SATA HDDs as well as flash SSDs that used for creating software defined storage for various scenarios.

    Scenarios include converged infrastructure (CI) disaggregated as well as aggregated hyper-converged infrastructure (HCI) for Hyper-V among other workloads. Windows Server 2016 S2D nodes communicate (from a storage perspective) via a software storage bus. Data Protection and availability is enabled between S2D nodes via Storage Replica (SR) that can do software based synchronous and asynchronous replication.


    Aggregated – Hyper-Converged Infrastructure (HCI) – Source Microsoft.com


    Desegregated – Converged Infrastructure (CI) – Source Microsoft.com

    The following is a Microsoft produced YouTube video providing a nice overview and insight into Windows Server 2016 and Microsoft Software Defined Storage aka S2D.




    YouTube Video Storage Spaces Direct (S2D) via Microsoft.com

    Server storage I/O performance

    What About Performance?

    A common question that comes up with servers, storage, I/O and software defined data infrastructure is what about performance?

    Following are some various links to different workloads showing performance for Hyper-V, S2D and Windows Server. Note as with any benchmark, workload or simulation take them for what they are, something to compare that may or might not be applicable to your own workload and environments.

    • Large scale VM performance with Hyper-V and in-memory transaction processing (Via Technet)
    • Benchmarking Microsoft Hyper-V server, VMware ESXi and Xen Hypervisors (Via cisjournal PDF)
    • Server 2016 Impact on VDI User Experience (Via LoginVSI)
    • Storage IOPS update with Storage Spaces Direct (Via TechNet)
    • SQL Server workload (benchmark) Order Processing Benchmark using In-Memory OLTP (Via Github)
    • Setting up testing Windows Server 2016 and S2D using virtual machines (Via MSDN blogs)
    • Storage throughput with Storage Spaces Direct (S2D TP5 (Via TechNet)
    • Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I)

    Where To Learn More

    For those of you not as familiar with Microsoft Windows Server and related topics, or that simply need a refresh, here are several handy links as well as resources.

    • Introducing Windows Server 2016 (Free ebook from Microsoft Press)
    • What’s New in Windows Server 2016 (Via TechNet)
    • Microsoft S2D Software Storage Bus (Via TechNet)
    • Understanding Software Defined Storage with S2D in Windows Server 2016 (Via TechNet)
    • Microsoft Storage Replica (SR) (Via TechNet)
    • Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I)
    • Microsoft Windows S2D Software Defined Storage (Via TechNet)
    • Windows Server 2016 and Active Directory (Redmond Magazine Webinar)
    • Data Protection for Modern Microsoft Environments (Redmond Magazine Webinar)
    • Resilient File System aka ReFS (Via TechNet)
    • DISKSPD now on GitHub, and the mysterious VMFLEET released (Via TechNet)
    • Hyper-converged solution using Storage Spaces Direct in Windows Server 2016 (Via TechNet)
    • NVMe, SSD and HDD storage configurations in Storage Spaces Direct TP5 (Via TechNet)
    • General information about SSD at www.thessdplace.com and NVMe at www.thenvmeplace.com
    • How to run nested Hyper-V and Windows Server 2016 (Via Altaro and via MSDN)
    • How to run Nested Windows Server and Hyper-V on VMware vSphere ESXi (Via Nokitel)
    • Get the Windows Server 2016 evaluation bits here
    • Microsoft Azure Stack overview and related material via Microsoft
    • Introducing Windows Server 2016 (Via MicrosoftPress)
    • Various WIndows Server and S2D lab scripts (Via Github)
    • Storage Spaces Direct – Lab Environment Setup (Via Argon Systems)
    • Setting up S2D with a 4 node configuration (Via StarWind blog)
    • SQL Server workload (benchmark) Order Processing Benchmark using In-Memory OLTP (Via Github)
    • Setting up testing Windows Server 2016 and S2D here using virtual machines (Via MSDN blogs)
    • Hyper-V large-scale VM performance for in-memory transaction processing (Via Technet)
    • BrightTalk Webinar – Software-Defined Data Centers (SDDC) are in your Future (if not already here)
    • Microsoft TechNet: Understand the cache in Storage Spaces Direct
    • BrightTalk Weibniar – Software-Defined Data Infrastructures Enabling Software-Defined Data Centers
    • Happy 20th Birthday Windows Server, ready for Server 2016?
    • Server StorageIO resources including added links, tools, reports, events and more.

    What This All Means

    While Microsoft Windows Server recently celebrated its 20th birthday (or anniversary), a lot has changed as well as evolved. This includes Windows Servers 2016 supporting new deployment and consumption models (e.g. lightweight Nano, full data center with desktop interface, on-premises, bare metal, virtualized (Hyper-V, VMware, etc) as well as cloud). Besides how consumed and configured, which can also be for CI and HCI modes, Windows Server 2016 along with Hyper-V extend the virtualization and container capabilities into non-Microsoft environments specifically around Linux and Docker. Not only are the support for those environments and platforms enhanced, so to are the management capabilities and interfaces from Powershell to Bash Linux shell being part of WIndows 10 and Server 2016.

    What this all means is that if you have not looked at Windows Server in some time, its time you do, even if you are not a WIndows or Microsoft fan, you will want to know what it is that has been updated (perhaps even update your fud if that is the case) to stay current. Get your hands on the bits and try Windows Server 2016 on a bare metal server, or as a VM guest, or via cloud including Azure, or simply leverage the above resources to learn more and stay informed.

    Ok, nuff said, for now…

    Cheers
    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, vSAN and VMware vExpert. Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier) and twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

    Posted on

    Get in the NVMe SSD game (if you are not already) with Intel 750 among others

    Get in the NVMe SSD game (if you are not already)

    server storage I/O trends

    Updated 2/2/2018

    Get in the NVMe SSD game. If you are or have talked the Non Volatile Memory (NVM) and NVM Express (NVMe) flash SSD talk, however have not yet done the walk, there are some good opportunities to do so. One example how to start walking the talk is get something like the Intel 750 400GB NVMe PCIe Add in Card (AiC) which you should be able to find on different venues including Amazon for $400 USD or less (recently saw the 400GB version for about $309 USD delivered). Of course the question might then be what to do with a NVMe device when you get it which besides running your applications on it, you can find some benchmark and workload generator tools, scripts and examples here.

    NVMe ssd storage
    Various NVM flash SSD including NVMe devices

    Sure there are newer, more expensive, higher capacity cards, some are general availability (GA) while others are on early preview or evaluation basis.

    However a good value, cost-effective way to get into the NVMe game, or expand your footprint are devices such as the Intel 750.

    I have some of these cards and they have been great for use with various workloads. The Intel 750 cards are plug and play in various systems including my TS140s among larger robust servers running Windows Server (2012 R2, 2016 TP5), Ubuntu (various) and VMware ESXi (various).

    There are in the box drivers for these Intel 750 cards, however to get a little more performance recommend visiting the Intel site, get the SSD tools, update the firmware on the cards if needed, as well as try the Intel vs. native drivers.

    There are many vendors with NVMe PCIe cards, as well as U.2/SFF 8639 drives and M.2 mini-cards including Intel, Micron, Seagate, WD among many others

    Where To Learn More

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Time to get in the NVMe game if you have not already done so moving from talking the talk to walking the talk. With cost of early generation NVMe devices such as the Intel 750 among others there are great opportunities to get started. On the other hand if you are already using NVMe, there are some good opportunities to add extra NVMe based SSD to other servers or systems.

    Speaking of NVMe talk, join me and others on a panel (Session 303) moderated by Dr. Jay Metz (@drjmetz) August 11th at Flash Memory Summit where we will discuss NVMe including NVMe over fabric networks (e.g. moving beyond the PCIe bus). Also keep in mind, with NVMe the question is not if, rather when, where, why, how with what and how to coexist and leverage existing technologies, hardware, software and people skill sets.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    12Gb SAS SSD Enabling Server Storage I/O Performance and Effectiveness Webinar

    12Gb SAS SSD Enabling Server Storage I/O Performance and Effectiveness Webinar

    server storage I/O trends

    Non-Volatile Memory (NVM) Solid State Devices (SSDs) including nand flash, DRAM as well as emerging PCM and 3D XPoint as part of Storage Class Memories (SCMs) are in your future. The questions are where, when, for what, how much as well as what form factor packaging, along with server storage I/O interface are applicable for your different applications and data infrastructures.

    server storage I/O SCM NVM SSD performance

    Server storage I/O physical interfaces for access NVM SSDs include PCIe Add in Cards (AiC), M.2 as well as emerging SFF 8639 (e.g. NVMe U2 drive form factor) along with mSATA (e.g. mini PCIe card) in addition to SAS, SATA, USB among others. Protocols include NVM Express (NVMe), SAS, SATA as well as general server storage I/O access of shared storage systems that leverage NVM SSD and SCM technologies.

    To help address the question of which server storage I/O interface is applicable for different environments, I invite you to a webinar on June 22, 2016 at 1PM ET hosted by and compliments of Micron.

    During the webinar myself and Rob Peglarr (@peglarr) of Micron will discuss and answer questions about how 12Gb SAS remains a viable option for attach NVM SSD storage to servers, as well as via storage systems today and into the future. Today’s 12Gb SAS SSDs enable you to leverage your existing knowledge, skill sets, as well as technology to maximize your data infrastructure investments. For servers or storage systems that are PCIe slot constrained, 12Gb SAS enables more SSD including 2.5" form factor multiple TByte capacity devices to be used to boost performance and capacity in a cost as well as energy effective way.

    server storage I/O nvm ssd options

    In addition to Rob Peglarr, we will also be joined by Doug Rollins of Micron (@GreyHairStorage) who will share some technical speeds, feeds, slots and watts information about Micron 12Gb SAS SSDs that can scale into the TBs in capacity per device.

    Here’s the synopsis from the Micron information page for this webinar.

    Don’t let old, slow SAS HDDs drag down your data center

    Modernize it by upgrading your storage from SAS HDDs to SAS SSDs. It’s an easy upgrade that provides a significant boost in performance, longer lasting endurance and nearly 4X the capacity. Flash storage changes how you do business and keeps you competitive.

    We invite you to join Rob Peglar, Greg Schulz, along with Doug Rollins, from Micron’s technical marketing team to learn:

    • Simple solutions to solving the challenges with today’s ever-growing data demands
    • Why SAS—how it continues to fuel the data center
    • HDDs versus SDDs—before and after stories from your peers, including upfront cost savings

    We will also have a live Q&A session so you can talk with the experts. Please register today! If you’re unable to attend the live webinar, we encourage you to register anyway to receive a link to the recorded session, as well as a copy of the presentation.

    Where To Learn More

    What This All Means

    Remember, everything is not the same in the data center or with data infrastructures that support different applications, like there are various NVM SSD options as well as interfaces.

    Join us for this webinar, you can view more information here, as well as register for the event.

    Ok, nuff said, for now…

    Cheers
    Gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

    Posted on

    Which Enterprise HDD for Content Server Platform

    Which Enterprise HDD to use for a Content Server Platform

    data infrastructure HDD server storage I/O trends

    Updated 1/23/2018

    Which enterprise HDD to use with a content server platform?

    Insight for effective server storage I/O decision making
    Server StorageIO Lab Review

    Which enterprise HDD to use for content servers

    This post is the first in a multi-part series based on a white paper hands-on lab report I did compliments of Equus Computer Systems and Seagate that you can read in PDF form here. The focus is looking at the Equus Computer Systems (www.equuscs.com) converged Content Solution platforms with Seagate Enterprise Hard Disk Drive (HDD’s). I was given the opportunity to do some hands-on testing running different application workloads with a 2U content solution platform along with various Seagate Enterprise 2.5” HDD’s handle different application workloads. This includes Seagate’s Enterprise Performance HDD’s with the enhanced caching feature.

    Issues And Challenges

    Even though Non-Volatile Memory (NVM) including NAND flash solid state devices (SSDs) have become popular storage for use internal as well as external to servers, there remains the need for HDD’s Like many of you who need to make informed server, storage, I/O hardware, software and configuration selection decisions, time is often in short supply.

    A common industry trend is to use SSD and HDD based storage mediums together in hybrid configurations. Another industry trend is that HDD’s continue to be enhanced with larger space capacity in the same or smaller footprint, as well as with performance improvements. Thus, a common challenge is what type of HDD to use for various content and application workloads balancing performance, availability, capacity and economics.

    Content Applications and Servers

    Fast Content Needs Fast Solutions

    An industry and customer trend are that information and data are getting larger, living longer, as well as there is more of it. This ties to the fundamental theme that applications and their underlying hardware platforms exist to process, move, protect, preserve and serve information.

    Content solutions span from video (4K, HD, SD and legacy streaming video, pre-/post-production, and editing), audio, imaging (photo, seismic, energy, healthcare, etc.) to security surveillance (including Intelligent Video Surveillance [ISV] as well as Intelligence Surveillance and Reconnaissance [ISR]). In addition to big fast data, other content solution applications include content distribution network (CDN) and caching, network function virtualization (NFV) and software-defined network (SDN), to cloud and other rich unstructured big fast media data, analytics along with little data (e.g. SQL and NoSQL database, key-value stores, repositories and meta-data) among others.

    Content Solutions And HDD Opportunities

    A common theme with content solutions is that they get defined with some amount of hardware (compute, memory and storage, I/O networking connectivity) as well as some type of content software. Fast content applications need fast software, multi-core processors (compute), large memory (DRAM, NAND flash, SSD and HDD’s) along with fast server storage I/O network connectivity. Content-based applications benefit from having frequently accessed data as close as possible to the application (e.g. locality of reference).

    Content solution and application servers need flexibility regarding compute options (number of sockets, cores, threads), main memory (DRAM DIMMs), PCIe expansion slots, storage slots and other connectivity. An industry trend is leveraging platforms with multi-socket processors, dozens of cores and threads (e.g. logical processors) to support parallel or high-concurrent content applications. These servers have large amounts of local storage space capacity (NAND flash SSD and HDD) and associated I/O performance (PCIe, NVMe, 40 GbE, 10 GbE, 12 Gbps SAS etc.) in addition to using external shared storage (local and cloud).

    Where To Learn More

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Fast content applications need fast content and flexible content solution platforms such as those from Equus Computer Systems and HDD’s from Seagate. Key to a successful content application deployment is having the flexibility to hardware define and software defined the platform to meet your needs. Just as there are many different types of content applications along with diverse environments, content solution platforms need to be flexible, scalable and robust, not to mention cost effective.

    Continue reading part two of this multi-part series here where we look at how and what to test as well as project planning.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Which Enterprise HDD for Content Applications Different File Size Impact

    Which HDD for Content Applications Different File Size Impact

    Different File Size Impact server storage I/O trends

    Updated 1/23/2018

    Which enterprise HDD to use with a content server platform different file size impact.

    Insight for effective server storage I/O decision making
    Server StorageIO Lab Review

    Which enterprise HDD to use for content servers

    This is the fifth in a multi-part series (read part four here) based on a white paper hands-on lab report I did compliments of Servers Direct and Seagate that you can read in PDF form here. The focus is looking at the Servers Direct (www.serversdirect.com) converged Content Solution platforms with Seagate Enterprise Hard Disk Drive (HDD’s). In this post the focus looks at large and small file I/O processing.

    File Performance Activity

    Tip, Content solutions use files in various ways. Use the following to gain perspective how various HDD’s handle workloads similar to your specific needs.

    Two separate file processing workloads were run (12), one with a relative small number of large files, and another with a large number of small files. For the large file processing (table-3), 5 GByte sized files were created and then accessed via 128 Kbyte (128KB) sized I/O over a 10 hour period with 90% read using 64 threads (workers). Large file workload simulates what might be seen with higher definition video, image or other content streaming.

    (Note 12) File processing workloads were run using Vdbench 5.04 and file anchors with sample script configuration below. Instead of vdbench you could also use other tools such as sysbench or fio among others.

    VdbenchFSBigTest.txt
    # Sample script for big files testing
    fsd=fsd1,anchor=H:,depth=1,width=5,files=20,size=5G
    fwd=fwd1,fsd=fsd1,rdpct=90,xfersize=128k,fileselect=random,fileio=random,threads=64
    rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=10h,interval=30

    vdbench -f VdbenchFSBigTest.txt -m 16 -o Results_FSbig_H_060615

    VdbenchFSSmallTest.txt
    # Sample script for big files testing
    fsd=fsd1,anchor=H:,depth=1,width=64,files=25600,size=16k
    fwd=fwd1,fsd=fsd1,rdpct=90,xfersize=1k,fileselect=random,fileio=random,threads=64
    rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=10h,interval=30

    vdbench -f VdbenchFSSmallTest.txt -m 16 -o Results_FSsmall_H_060615

    The 10% writes are intended to reflect some update activity for new content or other changes to content. Note that 128KB per second translates to roughly 1 Gbps streaming content such as higher definition video. However 4K video (not optimized) would require a higher speed as well as resulting in larger file sizes. Table-3 shows the performance during the large file access period showing average read /write rates and response time, bandwidth (MBps), average open and close rates with response time.

    Avg. File Read Rate

    Avg. Read Resp. Time
    Sec.

    Avg. File Write Rate

    Avg. Write Resp. Time
    Sec.

    Avg.
    CPU %
    Total

    Avg. CPU % System

    Avg. MBps
    Read

    Avg. MBps
    Write

    ENT 15K R1

    580.7

    107.9

    64.5

    19.7

    52.2

    35.5

    72.6

    8.1

    ENT 10K R1

    455.4

    135.5

    50.6

    44.6

    34.0

    22.7

    56.9

    6.3

    ENT CAP R1

    285.5

    221.9

    31.8

    19.0

    43.9

    28.3

    37.7

    4.0

    ENT 10K R10

    690.9

    87.21

    76.8

    48.6

    35.0

    21.8

    86.4

    9.6

    Table-3 Performance summary for large file access operations (90% read)

    Table-3 shows that for two-drive RAID 1, the Enterprise 15K are the fastest performance, however using a RAID 10 with four 10K HDD’s with enhanced cache features provide a good price, performance and space capacity option. Software RAID was used in this workload test.

    Figure-4 shows the relative performance of various HDD options handling large files, keep in mind that for the response line lower is better, while for the activity rate higher is better.

    large file processing
    Figure-4 Large file processing 90% read, 10% write rate and response time

    In figure-4 you can see the performance in terms of response time (reads larger dashed line, writes smaller dotted line) along with number of file read operations per second (reads solid blue column bar, writes green column bar). Reminder that lower response time, and higher activity rates are better. Performance declines moving from left to right, from 15K to 10K Enterprise Performance with enhanced cache feature to Enterprise Capacity (7.2K), all of which were hardware RAID 1. Also shown is a hardware RAID 10 (four x 10K HDD’s).

    Results in figure-4 above and table-4 below show how various drives can be configured to balance their performance, capacity and costs to meet different needs. Table-4 below shows an analysis looking at average file reads per second (RPS) performance vs. HDD costs, usable capacity and protection level.

    Table-4 is an example of looking at multiple metrics to make informed decisions as to which HDD would be best suited to your specific needs. For example RAID 10 using four 10K drives provides good performance and protection along with large usable space, however that also comes at a budget cost (e.g. price).

    Avg.
    File Reads Per Sec. (RPS)

    Single Drive Cost per RPS

    Multi-Drive Cost per RPS

    Single Drive Cost / Per GB Capacity

    Cost / Per GB Usable (Protected) Cap.

    Drive Cost (Multiple Drives)

    Protection Overhead (Space Capacity for RAID)

    Cost per usable GB per RPS

    Avg. File Read Resp. (Sec.)

    ENT 15K R1

    580.7

    $1.02

    $2.05

    $ 0.99

    $0.99

    $1,190

    100%

    $2.1

    107.9

    ENT 10K R1

    455.5

    1.92

    3.84

    0.49

    0.49

    1,750

    100%

    3.8

    135.5

    ENT CAP R1

    285.5

    1.40

    2.80

    0.20

    0.20

    798

    100%

    2.8

    271.9

    ENT 10K R10

    690.9

    1.27

    5.07

    0.49

    0.97

    3,500

    100%

    5.1

    87.2

    Table-4 Performance, capacity and cost analysis for big file processing

    Small File Size Processing

    To simulate a general file sharing environment, or content streaming with many smaller objects, 1,638,464 16KB sized files were created on each device being tested (table-5). These files were spread across 64 directories (25,600 files each) and accessed via 64 threads (workers) doing 90% reads with a 1KB I/O size over a ten hour time frame. Like the large file test, and database activity, all workloads were run at the same time (e.g. test devices were concurrently busy).

    Avg. File Read Rate

    Avg. Read Resp. Time
    Sec.

    Avg. File Write Rate

    Avg. Write Resp. Time
    Sec.

    Avg.
    CPU %
    Total

    Avg. CPU % System

    Avg. MBps
    Read

    Avg. MBps
    Write

    ENT 15K R1

    3,415.7

    1.5

    379.4

    132.2

    24.9

    19.5

    3.3

    0.4

    ENT 10K R1

    2,203.4

    2.9

    244.7

    172.8

    24.7

    19.3

    2.2

    0.2

    ENT CAP R1

    1,063.1

    12.7

    118.1

    303.3

    24.6

    19.2

    1.1

    0.1

    ENT 10K R10

    4,590.5

    0.7

    509.9

    101.7

    27.7

    22.1

    4.5

    0.5

    Table-5 Performance summary for small sized (16KB) file access operations (90% read)

    Figure-5 shows the relative performance of various HDD options handling large files, keep in mind that for the response line lower is better, while for the activity rate higher is better.

    small file processing
    Figure-5 Small file processing 90% read, 10% write rate and response time

    In figure-5 you can see the performance in terms of response time (reads larger dashed line, writes smaller dotted line) along with number of file read operations per second (reads solid blue column bar, writes green column bar). Reminder that lower response time, and higher activity rates are better. Performance declines moving from left to right, from 15K to 10K Enterprise Performance with enhanced cache feature to Enterprise Capacity (7.2K RPM), all of which were hardware RAID 1. Also shown is a hardware RAID 10 (four x 10K RPM HDD’s) that has higher performance and capacity along with costs (table-5).

    Results in figure-5 above and table-5 below show how various drives can be configured to balance their performance, capacity and costs to meet different needs. Table-6 below shows an analysis looking at average file reads per second (RPS) performance vs. HDD costs, usable capacity and protection level.

    Table-6 is an example of looking at multiple metrics to make informed decisions as to which HDD would be best suited to your specific needs. For example RAID 10 using four 10K drives provides good performance and protection along with large usable space, however that also comes at a budget cost (e.g. price).

    Avg.
    File Reads Per Sec. (RPS)

    Single Drive Cost per RPS

    Multi-Drive Cost per RPS

    Single Drive Cost / Per GB Capacity

    Cost / Per GB Usable (Protected) Cap.

    Drive Cost (Multiple Drives)

    Protection Overhead (Space Capacity for RAID)

    Cost per usable GB per RPS

    Avg. File Read Resp. (Sec.)

    ENT 15K R1

    3,415.7

    $0.17

    $0.35

    $0.99

    $0.99

    $1,190

    100%

    $0.35

    1.51

    ENT 10K R1

    2,203.4

    0.40

    0.79

    0.49

    0.49

    1,750

    100%

    0.79

    2.90

    ENT CAP R1

    1,063.1

    0.38

    0.75

    0.20

    0.20

    798

    100%

    0.75

    12.70

    ENT 10K R10

    4,590.5

    0.19

    0.76

    0.49

    0.97

    3,500

    100%

    0.76

    0.70

    Table-6 Performance, capacity and cost analysis for small file processing

    Looking at the small file processing analysis in table-5 shows that the 15K HDD’s on an apples to apples basis (e.g. same RAID level and number of drives) provide the best performance. However when also factoring in space capacity, performance, different RAID level or other protection schemes along with cost, there are other considerations. On the other hand the Enterprise Capacity 2TB HDD’s have a low cost per capacity, however do not have the performance of other options, assuming your applications need more performance.

    Thus the right HDD for one application may not be the best one for a different scenario as well as multiple metrics as shown in table-5 need to be included in an informed storage decision making process.

    Where To Learn More

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    File processing are common content applications tasks, some being small, others large or mixed as well as reads and writes. Even if your content environment is using object storage, chances are unless it is a new applications or a gateway exists, you may be using NAS or file based access. Thus the importance of if your applications are doing file based processing, either run your own applications or use tools that can simulate as close as possible to what your environment is doing.

    Continue reading part six in this multi-part series here where the focus is around general I/O including 8KB and 128KB sized IOPs along with associated metrics.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    HDDs evolve for Content Application servers

    HDDs evolve for Content Application servers

    hdds evolve server storage I/O trends

    Updated 1/23/2018

    Enterprise HDDs evolve for content server platform

    Insight for effective server storage I/O decision making
    Server StorageIO Lab Review

    Which enterprise HDD to use for content servers

    This is the seventh and final post in this multi-part series (read part six here) based on a white paper hands-on lab report I did compliments of Servers Direct and Seagate that you can read in PDF form here. The focus is looking at the Servers Direct (www.serversdirect.com) converged Content Solution platforms with Seagate Enterprise Hard Disk Drive (HDD’s). The focus of this post is comparing how HDD continue to evolve over various generations boosting performance as well as capacity and reliability. This also looks at how there is more to HDD performance than the traditional focus on Revolutions Per Minute (RPM) as a speed indicator.

    Comparing Different Enterprise 10K And 15K HDD Generations

    There is more to HDD performance than RPM speed of the device. RPM plays an important role, however there are other things that impact HDD performance. A common myth is that HDD’s have not improved on performance over the past several years with each successive generation. Table-10 shows a sampling of various generations of enterprise 10K and 15K HDD’s (14) including different form factors and how their performance continues to improve.

    different 10K and 15K HDDs
    Figure-9 10K and 15K HDD performance improvements

    Figure-9 shows how performance continues to improve with 10K and 15K HDD’s with each new generation including those with enhanced cache features. The result is that with improvements in cache software within the drives, along with enhanced persistent non-volatile memory (NVM) and incremental mechanical drive improvements, both read and write performance continues to be enhanced.

    Figure-9 puts into perspective the continued performance enhancements of HDD’s comparing various enterprise 10K and 15K devices. The workload is the same TPC-C tests used earlier in a similar (14) (with no RAID). 100 simulated users are shown in figure-9 accessing a database on each of the different drives all running concurrently. The older 15K 3.5” Cheetah and 2.5” Savio used had a capacity of 146GB which used a database scale factor of 1500 or 134GB. All other drives used a scale factor 3000 or 276GB. Figure-9 also highlights the improvements in both TPS performance as well as lower response time with new HDD’s including those with performance enhanced cache feature.

    The workloads run are same as the TPC-C ones shown earlier, however these drives were not configured with any RAID. The TPC-C activity used Benchmark Factory with similar setup and configuration to those used earlier including on a multi-socket, multi-core Windows 2012 R2 server supporting a Microsoft SQL Server 2012 database with a database for each drive type.

    ENT 10K V3 2.5"

    ENT (Cheetah) 15K 3.5"

    Users

    1

    20

    50

    100

    Users

    1

    20

    50

    100

    TPS (TPC-C)

    14.8

    50.9

    30.3

    39.9

    TPS (TPC-C)

    14.6

    51.3

    27.1

    39.3

    Resp. Time (Sec.)

    0.0

    0.4

    1.6

    1.7

    Resp. Time (Sec.)

    0.0

    0.3

    1.8

    2.1

    ENT 10K 2.5" (with cache)

    ENT (Savio) 15K 2.5"

    Users

    1

    20

    50

    100

    Users

    1

    20

    50

    100

    TPS (TPC-C)

    19.2

    146.3

    72.6

    71.0

    TPS (TPC-C)

    15.8

    59.1

    40.2

    53.6

    Resp. Time (Sec.)

    0.0

    0.1

    0.7

    0.0

    Resp. Time (Sec.)

    0.0

    0.3

    1.2

    1.2

    ENT 15K V4 2.5"

    Users

    1

    20

    50

    100

    TPS (TPC-C)

    19.7

    119.8

    75.3

    69.2

    Resp. Time (Sec.)

    0.0

    0.1

    0.6

    1.0

    ENT 15K (enhanced cache) 2.5"

    Users

    1

    20

    50

    100

    TPS (TPC-C)

    20.1

    184.1

    113.7

    122.1

    Resp. Time (Sec.)

    0.0

    0.1

    0.4

    0.2

    Table-10 Continued Enterprise 10K and 15K HDD performance improvements

    (Note 14) 10K and 15K generational comparisons were run on a separate comparable server to what was used for other test workloads. Workload configuration settings were the same as other database workloads including using Microsoft SQL Server 2012 on a Windows 2012 R2 system with Benchmark Factory driving the workload. Database memory sized was reduced however to only 8GB vs. 16GB used in other tests.

    Where To Learn More

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    A little bit of flash in the right place with applicable algorithms goes a long way, an example being the Seagate Enterprise HDD’s with enhanced cache feature. Likewise, HDD’s are very much alive complementing SSD and vice versa. For high-performance content application workloads flash SSD solutions including NVMe, 12Gbps SAS and 6Gbps SATA devices are cost effective solutions. HDD’s continue to be cost-effective data storage devices for both capacity, as well as environments that do not need the performance of flash SSD.

    For some environments using a combination of flash and HDD’s complementing each other along with cache software can be a cost-effective solution. The previous workload examples provide insight for making cost-effective informed storage decisions.

    Evaluate today’s HDD’s on their effective performance running workloads as close as similar to your own, or, actually try them out with your applications. Today there is more to HDD performance than just RPM speed, particular with the Seagate Enterprise Performance 10K and 15K HDD’s with enhanced caching feature.

    However the Enterprise Performance 10K with enhanced cache feature provides a good balance of capacity, performance while being cost-effective. If you are using older 3.5” 15K or even previous generation 2.5” 15K RPM and “non-performance enhanced” HDD’s, take a look at how the newer generation HDD’s perform, looking beyond the RPM of the device.

    Fast content applications need fast content and flexible content solution platforms such as those from Servers Direct and HDD’s from Seagate. Key to a successful content application deployment is having the flexibility to hardware define and software defined the platform to meet your needs. Just as there are many different types of content applications along with diverse environments, content solution platforms need to be flexible, scalable and robust, not to mention cost effective.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    VMware vCloud Air Server StorageIOlab Test Drive with videos

    Server Storage I/O trends

    VMware vCloud Air Server StorageIOlab Test Drive with videos

    Recently I was invited by VMware vCloud Air to do a free hands-on test drive of their actual production environment. Some of you may already being using VMware vSphere, vRealize and other software defined data center (SDDC) aka Virtual Server Infrastructure (VSI) or Virtual Desktop Infrastructure (VDI) tools among others. Likewise some of you may already be using one of the many cloud compute or Infrastructure as a Service (IaaS) such as Amazon Web Services (AWS) Elastic Cloud Compute (EC2), Centurylink, Google Cloud, IBM Softlayer, Microsoft Azure, Rackspace or Virtustream (being bought by EMC) among many others.

    VMware vCloud Air provides a platform similar to those just mentioned among others for your applications and their underlying resource needs (compute, memory, storage, networking) to be fulfilled. In addition, it should not be a surprise that VMware vCloud Air shares many common themes, philosophies and user experiences with the traditional on-premises based VMware solutions you may be familiar with.

    VMware vCloud Air overview

    You can give VMware vCloud Air a trial for free while the offer lasts by clicking here (service details here). Basically if you click on the link and register a new account for using VMware vCloud Air they will give you up to $500 USD in service credits to use in the real production environment while the offer lasts which iirc is through end of June 2015.

    Server StorageIO test drive VMware vCloud Air video I
    Click on above image to view video part I

    Server StorageIO test drive VMware vCloud Air part II
    Click on above image to view video part II

    What this means is that you can go and setup some servers with as many CPUs or cores, memory, Hard Disk Drive (HDD) or flash Solid State Devices (SSD) storage, external IP networks using various operating systems (Centos, Ubuntu, Windows 2008, 20012, 20012 R2) for free, or until you use up the service credits.

    Speaking of which, let me give you a bit of a tip or hint, even though you can get free time, if you provision a fast server with lots of fast SSD storage and leave it sit idle over night or over a weekend, you will chew up your free credits rather fast. So the tip which should be common sense is if you are going to do some proof of concepts and then leave things alone for a while, power the virtual cloud servers off to stretch your credits further. On the other hand, if you have something that you want to run on a fast server with fast storage over a weekend or longer, give that a try, just pay attention to your resource usage and possible charges should you exhaust your service credits.

    My Server StorageIO test drive mission objective

    For my test drive, I created a new account by using the above link to get the service credits. Note that you can use your regular VMware account with vCloud Air, however you wont get the free service credits. So while it is a few minutes of extra work, the benefit was worth it vs. simply using my existing VMware account and racking up more cloud services charges on my credit card. As part of this Server StorageIOlab test drive, I created two companion videos part I here and part II here that you can view to follow along and get a better idea of how vCloud works.

    VMware vCloud Air overview
    Phase one, create the virtual data center, database server, client servers and first setup

    My goal was to set up a simple Virtual Data Center (VDC) that would consist of five Windows 2012 R2 servers, one would be a MySQL database server with the other four being client application servers. You can download MySQL from here at Oracle as well as via other sources. For applications to simplify things I used Hammerdb as well as Benchmark Factory that is part of the Quest Toad tool set for database admins. You can download a free trial copy of Benchmark Factory here, and HammerDB here. Another tool that I used for monitoring the servers is Spotlight on Windows (SoW) which is also free here. Speaking of tools, here is a link to various server and storage I/O performance as well as monitoring tools.

    Links to tools that I used for this test-drive included:

    Setting up a virtual data center vdc
    Phase one steps and activity summary

    Summary of phase one of vdc
    Recap of what was done in phase one, watch the associated video here.

    After the initial setup (e.g. part I video here), the next step was to add some more virtual machines and take a closer look at the environment. Note that most of the work in setting up this environment was Windows, MySQL, Hammerdb, Benchmark Factory, Spotlight on Windows along with other common tools so their installation is not a focus in these videos or this post, perhaps a future post will dig into those in more depth.

    Summary of phase two of the vdc
    What was done during phase II (view the video here)

    VMware vCloud Air vdc trest drive

    There is much more to VMware vCloud Air and on their main site there are many useful links including overviews, how-too tutorials, product and service offering details and much more here. Besides paying attention to your resource usage and avoid being surprised by service charges, two other tips I can pass along that are also mentioned in the videos (here and here) is to pay attention what region you setup your virtual data centers in, second is have your network thought out ahead of time to streamline setting up the NAT and firewall as well as gateway configurations.

    Where to learn more

    Learn more about data protection and related topics, themes, trends, tools and technologies via the following links:

    Server Storage I/O trends

    What this all means and wrap-up

    Overall I like the VMware vCloud Air service which if you are VMware centric focused will be a familiar cloud option including integration with vCloud Director and other tools you may already have in your environment. Even if you are not familiar with VMware vSphere and associated vRealize tools, the vCloud service is intuitive enough that you can be productive fairly quickly. On one hand vCloud Air does not have the extensive menu of service offerings to choose from such as with AWS, Google, Azure or others, however that also means a simpler menu of options to choose from and simplify things.

    I had wanted to spend some time actually using vCloud and the offer to use some free service credits in the production environment made it worth making the time to actually setup some workloads and do some testing. Even if you are not a VMware focused environment, I would recommend giving VMware vCloud Air a test drive to see what it can do for you, as opposed to what you can do for it…

    Ok, nuff said for now

    Cheers
    Gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    Microsoft Diskspd (Part II): Server Storage I/O Benchmark Tools

    Microsoft Diskspd (Part II): Server Storage I/O Benchmark Tools

    server storage I/O trends

    This is part-two of a two-part post pertaining Microsoft Diskspd.that is also part of a broader series focused on server storage I/O benchmarking, performance, capacity planning, tools and related technologies. You can view part-one of this post here, along with companion links here.

    Microsoft Diskspd StorageIO lab test drive

    Server and StorageIO lab

    Talking about tools and technologies is one thing, installing as well as trying them is the next step for gaining experience so how about some quick hands-on time with Microsoft Diskspd (download your copy here).

    The following commands all specify an I/O size of 8Kbytes doing I/O to a 45GByte file called diskspd.dat located on the F: drive. Note that a 45GByte file is on the small size for general performance testing, however it was used for simplicity in this example. Ideally a larger target storage area (file, partition, device) would be used, otoh, if your application uses a small storage device or volume, then tune accordingly.

    In this test, the F: drive is an iSCSI RAID protected volume, however you could use other storage interfaces supported by Windows including other block DAS or SAN (e.g. SATA, SAS, USB, iSCSI, FC, FCoE, etc) as well as NAS. Also common to the following commands is using 16 threads and 32 outstanding I/Os to simulate concurrent activity of many users, or application processing threads.
    server storage I/O performance
    Another common parameter used in the following was -r for random, 7200 seconds (e.g. two hour) test duration time, display latency ( -L ) disable hardware and software cache ( -h), forcing cpu affinity (-a0,1,2,3). Since the test ran on a server with four cores I wanted to see if I could use those for helping to keep the threads and storage busy. What varies in the commands below is the percentage of reads vs. writes, as well as the results output file. Some of the workload below also had the -S option specified to disable OS I/O buffering (to view how buffering helps when enabled or disabled). Depending on the goal, or type of test, validation, or workload being run, I would choose to set some of these parameters differently.

    diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -w0 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noh_write000.txt

    diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -w50 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noh_write050.txt

    diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -w100 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noh_write100.txt

    diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -S -w0 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noSh_test_write000.txt

    diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -S -w50 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noSh_write050.txt

    diskspd -c45g -b8K -t16 -o32 -r -d7200 -h -S -w100 -L -a0,1,2,3 F:\diskspd.dat >> SIOWS2012R203_Eiscsi_145_noSh_write100.txt

    The following is the output from the above workload command.
    Microsoft Diskspd sample output
    Microsoft Diskspd sample output part 2
    Microsoft Diskspd sample output part 3

    Note that as with any benchmark, workload test or simulation your results will vary. In the above the server, storage and I/O system were not tuned as the focus was on working with the tool, determining its capabilities. Thus do not focus on the performance results per say, rather what you can do with Diskspd as a tool to try different things. Btw, fwiw, in the above example in addition to using an iSCSI target, the Windows 2012 R2 server was a guest on a VMware ESXi 5.5 system.

    Where to learn more

    The following are related links to read more about server (cloud, virtual and physical) storage I/O benchmarking tools, technologies and techniques.

    Drew Robb’s benchmarking quick reference guide
    Server storage I/O benchmarking tools, technologies and techniques resource page
    Server and Storage I/O Benchmarking 101 for Smarties.
    Microsoft Diskspd download and Microsoft Diskspd overview (via Technet)
    I/O, I/O how well do you know about good or bad server and storage I/Os?
    Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I and Part II)

    Comments and wrap-up

    What I like about Diskspd (Pros)

    Reporting including CPU usage (you can’t do server and storage I/O without CPU) along with IOP’s (activity), bandwidth (throughout or amount of data being moved), per thread and total results along with optional reporting. While a GUI would be nice particular for beginners, I’m used to setting up scripts for different workloads so having an extensive options for setting up different workloads is welcome. Being associated with a specific OS (e.g. Windows) the CPU affinity and buffer management controls will be handy for some projects.

    Diskspd has the flexibility to use different storage interfaces and types of storage including files or partitions should be taken for granted, however with some tools don’t take things for granted. I like the flexibility to easily specify various IO sizes including large 1MByte, 10MByte, 20MByte, 100MByte and 500MByte to simulate application workloads that do large sequential (or random) activity. I tried some IO sizes (e.g. specified by -b parameter larger than 500MB however, I received various errors including "Could not allocate a buffer bytes for target" which means that Diskspd can do IO sizes smaller than that. While not able to do IO sizes larger than 500MB, this is actually impressive. Several other tools I have used or with have IO size limits down around 10MByte which makes it difficult for creating workloads that do large IOP’s (note this is the IOP size, not the number of IOP’s).

    Oh, something else that should be obvious however will state it, Diskspd is free unlike some industry de-facto standard tools or workload generators that need a fee to get and use.

    Where Diskspd could be improved (Cons)

    For some users a GUI or configuration wizard would make the tool easier to get started with, on the other hand (oth), I tend to use the command capabilities of tools. Would also be nice to specify ranges as part of a single command such as stepping through an IO size range (e.g. 4K, 8K, 16K, 1MB, 10MB) as well as read write percentages along with varying random sequential mixes. Granted this can easily be done by having a series of commands, however I have become spoiled by using other tools such as vdbench.

    Summary

    Server and storage I/O performance toolbox

    Overall I like Diskspd and have added it to my Server Storage I/O workload and benchmark tool-box

    Keep in mind that the best benchmark or workload generation technology tool will be your own application(s) configured to run as close as possible to production activity levels.

    However when that is not possible, the an alternative is to use tools that have the flexibility to be configured as close as possible to your application(s) workload characteristics. This means that the focus should not be as much on the tool, as opposed to how flexible is a tool to work for you, granted the tool needs to be robust.

    Having said that, Microsoft Diskspd is a good and extensible tool for benchmarking, simulation, validation and comparisons, however it will only be as good as the parameters and configuration you set it up to use.

    Check out Microsoft Diskspd and add it to your benchmark and server storage I/O tool-box like I have done.

    Ok, nuff said (for now)

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    Server Storage I/O Benchmark Performance Resource Tools

    Server Storage I/O Benchmarking Performance Resource Tools

    server storage I/O trends

    Updated 1/23/2018

    Server storage I/O benchmark performance resource tools, various articles and tips. These include tools for legacy, virtual, cloud and software defined environments.

    benchmark performance resource tools server storage I/O performance

    The best server and storage I/O (input/output operation) is the one that you do not have to do, the second best is the one with the least impact.

    server storage I/O locality of reference

    This is where the idea of locality of reference (e.g. how close is the data to where your application is running) comes into play which is implemented via tiered memory, storage and caching shown in the figure above.

    Cloud virtual software defined storage I/O

    Server storage I/O performance applies to cloud, virtual, software defined and legacy environments

    What this has to do with server storage I/O (and networking) performance benchmarking is keeping the idea of locality of reference, context and the application workload in perspective regardless of if cloud, virtual, software defined or legacy physical environments.

    StorageIOblog: I/O, I/O how well do you know about good or bad server and storage I/Os?
    StorageIOblog: Server and Storage I/O benchmarking 101 for smarties
    StorageIOblog: Which Enterprise HDDs to use for a Content Server Platform (7 part series with using benchmark tools)
    StorageIO.com: Enmotus FuzeDrive MicroTiering lab test using various tools
    StorageIOblog: Some server storage I/O benchmark tools, workload scripts and examples (Part I) and (Part II)
    StorageIOblog: Get in the NVMe SSD game (if you are not already)
    Doridmen.com: Transcend SSD360S Review with tips on using ATTO and Crystal benchmark tools
    ComputerWeekly: Storage performance metrics: How suppliers spin performance specifications

    Via StorageIO Podcast: Kevin Closson discusses SLOB Server CPU I/O Database Performance benchmarks
    Via @KevinClosson: SLOB Use Cases By Industry Vendors. Learn SLOB, Speak The Experts’ Language
    Via BeyondTheBlocks (Reduxio): 8 Useful Tools for Storage I/O Benchmarking
    Via CCSIObench: Cold-cache Sequential I/O Benchmark
    Doridmen.com: Transcend SSD360S Review with tips on using ATTO and Crystal benchmark tools
    CISJournal: Benchmarking the Performance of Microsoft Hyper-V server, VMware ESXi and Xen Hypervisors (PDF)
    Microsoft TechNet:Windows Server 2016 Hyper-V large-scale VM performance for in-memory transaction processing
    InfoStor: What’s The Best Storage Benchmark?
    StorageIOblog: How to test your HDD, SSD or all flash array (AFA) storage fundamentals
    Via ATTO: Atto V3.05 free storage test tool available
    Via StorageIOblog: Big Files and Lots of Little File Processing and Benchmarking with Vdbench

    Via StorageIO.com: Which Enterprise Hard Disk Drives (HDDs) to use with a Content Server Platform (White Paper)
    Via VMware Blogs: A Free Storage Performance Testing Tool For Hyperconverged
    Microsoft Technet: Test Storage Spaces Performance Using Synthetic Workloads in Windows Server
    Microsoft Technet: Microsoft Windows Server Storage Spaces – Designing for Performance
    BizTech: 4 Ways to Performance-Test Your New HDD or SSD
    EnterpriseStorageForum: Data Storage Benchmarking Guide
    StorageSearch.com: How fast can your SSD run backwards?
    OpenStack: How to calculate IOPS for Cinder Storage ?
    StorageAcceleration: Tips for Measuring Your Storage Acceleration

    server storage I/O STI and SUT

    Spiceworks: Determining HDD SSD SSHD IOP Performance
    Spiceworks: Calculating IOPS from Perfmon data
    Spiceworks: profiling IOPs

    vdbench server storage I/O benchmark
    Vdbench example via StorageIOblog.com

    StorageIOblog: What does server storage I/O scaling mean to you?
    StorageIOblog: What is the best kind of IO? The one you do not have to do
    Testmyworkload.com: Collect and report various OS workloads
    Whoishostingthis: Various SQL resources
    StorageAcceleration: What, When, Why & How to Accelerate Storage
    Filesystems.org: Various tools and links
    StorageIOblog: Can we get a side of context with them IOPS and other storage metrics?

    flash ssd and hdd

    BrightTalk Webinar: Data Center Monitoring – Metrics that Matter for Effective Management
    StorageIOblog: Enterprise SSHD and Flash SSD Part of an Enterprise Tiered Storage Strategy
    StorageIOblog: Has SSD put Hard Disk Drives (HDD’s) On Endangered Species List?

    server storage I/O bottlenecks and I/O blender

    Microsoft TechNet: Measuring Disk Latency with Windows Performance Monitor (Perfmon)
    Via Scalegrid.io: How to benchmark MongoDB with YCSB? (Perfmon)
    Microsoft MSDN: List of Perfmon counters for sql server
    Microsoft TechNet: Taking Your Server’s Pulse
    StorageIOblog: Part II: How many IOPS can a HDD, HHDD or SSD do with VMware?
    CMG: I/O Performance Issues and Impacts on Time-Sensitive Applications

    flash ssd and hdd

    Virtualization Practice: IO IO it is off to Storage and IO metrics we go
    InfoStor: Is HP Short Stroking for Performance and Capacity Gains?
    StorageIOblog: Is Computer Data Storage Complex? It Depends
    StorageIOblog: More storage and IO metrics that matter
    StorageIOblog: Moving Beyond the Benchmark Brouhaha
    Yellow-Bricks: VSAN VDI Benchmarking and Beta refresh!

    server storage I/O benchmark example

    YellowBricks: VSAN performance: many SAS low capacity VS some SATA high capacity?
    YellowBricsk: VSAN VDI Benchmarking and Beta refresh!
    StorageIOblog: Seagate 1200 12Gbs Enterprise SAS SSD StorgeIO lab review
    StorageIOblog: Part II: Seagate 1200 12Gbs Enterprise SAS SSD StorgeIO lab review
    StorageIOblog: Server Storage I/O Network Benchmark Winter Olympic Games

    flash ssd and hdd

    VMware VDImark aka View Planner (also here, here and here) as well as VMmark here
    StorageIOblog: SPC and Storage Benchmarking Games
    StorageIOblog: Speaking of speeding up business with SSD storage
    StorageIOblog: SSD and Storage System Performance

    Hadoop server storage I/O performance
    Various Server Storage I/O tools in a hadoop environment

    Michael-noll.com: Benchmarking and Stress Testing an Hadoop Cluster With TeraSort, TestDFSIO
    Virtualization Practice: SSD options for Virtual (and Physical) Environments Part I: Spinning up to speed on SSD
    StorageIOblog: Storage and IO metrics that matter
    InfoStor: Storage Metrics and Measurements That Matter: Getting Started
    SilvertonConsulting: Storage throughput vs. IO response time and why it matters
    Splunk: The percentage of Read / Write utilization to get to 800 IOPS?

    flash ssd and hdd
    Various server storage I/O benchmarking tools

    Spiceworks: What is the best IO IOPs testing tool out there
    StorageIOblog: How many IOPS can a HDD, HHDD or SSD do?
    StorageIOblog: Some Windows Server Storage I/O related commands
    Openmaniak: Iperf overview and Iperf.fr: Iperf overview
    StorageIOblog: Server and Storage I/O Benchmark Tools: Microsoft Diskspd (Part I and Part II)
    Quest: SQL Server Perfmon Poster (PDF)
    Server and Storage I/O Networking Performance Management (webinar)
    Data Center Monitoring – Metrics that Matter for Effective Management (webinar)
    Flash back to reality – Flash SSD Myths and Realities (Industry trends & benchmarking tips), (MSP CMG presentation)
    DBAstackexchange: How can I determine how many IOPs I need for my AWS RDS database?
    ITToolbox: Benchmarking the Performance of SANs

    server storage IO labs

    StorageIOblog: Dell Inspiron 660 i660, Virtual Server Diamond in the rough (Server review)
    StorageIOblog: Part II: Lenovo TS140 Server and Storage I/O Review (Server review)
    StorageIOblog: DIY converged server software defined storage on a budget using Lenovo TS140
    StorageIOblog: Server storage I/O Intel NUC nick knack notes First impressions (Server review)
    StorageIOblog & ITKE: Storage performance needs availability, availability needs performance
    StorageIOblog: Why SSD based arrays and storage appliances can be a good idea (Part I)
    StorageIOblog: Revisiting RAID storage remains relevant and resources

    Interested in cloud and object storage visit our objectstoragecenter.com page, for flash SSD checkout storageio.com/ssd page, along with data protection, RAID, various industry links and more here.

    Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

    Software Defined Data Infrastructure Essentials Book SDDC

    What This All Means

    Watch for additional links to be added above in addition to those that appear via comments.

    Ok, nuff said, for now.

    Gs

    Greg Schulz – Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

    Posted on

    Cloud Conversations: Revisiting re:Invent 2014 and other AWS updates

    server storage I/O trends

    This is part one of a two-part series about Amazon Web Services (AWS) re:Invent 2014 and other recent cloud updates, read part two here.

    Revisiting re:Invent 2014 and other AWS updates

    AWS re:Invent 2014

    A few weeks ago I attended Amazon Web Service (AWS) re:Invent 2014 in Las Vegas for a few days. For those of you who have not yet attended this event, I recommend adding it to your agenda. If you have interest in compute servers, networking, storage, development tools or management of cloud (public, private, hybrid), virtualization and related topic themes, you should check out AWS re:invent.

    AWS made several announcements at re:invent including many around development tools, compute and data storage services. One of those to keep an eye on is cloud based Aurora relational database service that complement existing RDS tools. Aurora is positioned as an alternative to traditional SQL based transactional databases commonly found in enterprise environments (e.g. SQL Server among others).

    Some recent AWS announcements prior to re:Invent include

    AWS vCenter Portal

    Using the AWS Management Portal for vCenter adds a plug-in within your VMware vCenter to manage your AWS infrastructure. The vCenter for AWS plug-in includes support for AWS EC2 and Virtual Machine (VM) import to migrate your VMware VMs to AWS EC2, create VPC (Virtual Private Clouds) along with subnet’s. There is no cost for the plug-in, you simply pay for the underlying AWS resources consumed (e.g. EC2, EBS, S3). Learn more about AWS Management Portal for vCenter here, and download the OVA plug-in for vCenter here.

    AWS re:invent content


    AWS Andy Jassy (Image via AWS)

    November 12, 2014 (Day 1) Keynote (highlight video, full keynote). This is the session where AWS SVP Andy Jassy made several announcements including Aurora relational database that complements existing RDS (Relational Data Services). In addition to Andy, the key-note sessions also included various special guests ranging from AWS customers, partners and internal people in support of the various initiatives and announcements.


    Amazon.com CTO Werner Vogels (Image via AWS)

    November 13, 2014 (Day 2) Keynote (highlight video, full keynote). In this session, Amazon.com CTO Werner Vogels appears making announcements about the new Container and Lambda services.

    AWS re:Invent announcements

    Announcements and enhancements made by AWS during re:Invent include:

    • Key Management Service (KMS)
    • Amazon RDS for Aurora
    • Amazon EC2 Container Service
    • AWS Lambda
    • Amazon EBS Enhancements
    • Application development, deployed and life-cycle management tools
    • AWS Service Catalog
    • AWS CodeDeploy
    • AWS CodeCommit
    • AWS CodePipeline

    Key Management Service (KMS)

    Hardware security module (HSM) based key managed service for creating and control of encryption keys to protect security of digital assets and their keys. Integration with AWS EBS and others services including S3 and Redshift along with CloudTrail logs for regulatory, compliance and management. Learn more about AWS KMS here

    AWS Database

    For those who are not familiar, AWS has a suite of database related services including SQL and no SQL based, simple to transactional to Petabyte (PB) scale data warehouses for big data and analytics. AWS offers the Relational Database Service (RDS) which is a suite of different database types, instances and services. RDS instance and types include SimpleDB, MySQL, Postgress, Oracle, SQL Server and the new AWS Aurora offering (read more below).  Other little data database and big data repository related offerings include DynamoDB (a non-SQL database), ElasticCache (in memory cache repository) and Redshift (large-scale data warehouse and big data repository).

    In addition to database services offered by AWS, you can also combine various AWS resources including EC2 compute, EBS and other storage offerings to create your own solution. For example there are various Amazon Machine Images (AMI’s) or pre-built operating systems and database tools available with EC2 as well as via the AWS Marketplace , such as MongoDB and Couchbase among others. For those not familiar with MongoDB, Couchbase, Cassandra, Riak along with other non SQL or alternative databases and key value repositories, check out Seven Databases in Seven Weeks in my book review of it here.

    Seven Databases book review
    Seven Databases in Seven Weeks and NoSQL movement available from Amazon.com

    Amazon RDS for Aurora

    Aurora is a new relational database offering part of the AWS RDS suite of services. Positioned as an alternative to commercial high-end database, Aurora is a cost-effective database engine compatible with MySQL. AWS is claiming 5x better performance than standard MySQL with Aurora while being resilient and durable. Learn more about Aurora which will be available in early 2015 and its current preview here.

    Amazon EC2 C4 instances

    AWS will be adding a new C4 instance as a next generation of EC2 compute instance based on Intel Xeon E5-2666 v3 (Haswell) processors. The Intel Xeon E5-2666 v3 processors run at a clock speed of 2.9 GHz providing the highest level of EC2 performance. AWS is targeting traditional High Performance Computing (HPC) along with other compute intensive workloads including analytics, gaming, and transcoding among others. Learn more AWS EC2 instances here, and view this Server and StorageIO EC2, EBS and associated AWS primer here.

    Amazon EC2 Container Service

    Containers such as those via Docker have become popular to support developers rapidly build as well as deploy scalable applications. AWS has added a new feature called EC2 Container Service that supports Docker using simple API’s. In addition to supporting Docker, EC2 Container Service is a high performance scalable container management service for distributed applications deployed on a cluster of EC2 instances. Similar to other EC2 services, EC2 Container Service leverages security groups, EBS volumes and Identity Access Management (IAM) roles along with scheduling placement of containers to meet your needs. Note that AWS is not alone in adding container and docker support with Microsoft Azure also having recently made some announcements, learn more about Azure and Docker here. Learn more about EC2 container service here and more about Docker here.

    Docker for smarties

    Continue reading about re:Invent 2014 and other recent AWS enhancements here in part two of this two-part series.

    Ok, nuff said (for now)

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved

    Posted on

    November 2014 Server StorageIO Update Newsletter

    November 2014

    Hello and welcome to this November Server and StorageIO update newsletter. Enjoy this edition of the Server and StorageIO update newsletter and watch for new tips, articles, StorageIO lab report reviews, blog posts, videos and podcasts along with in the news commentary appearing soon.

    Cheers gs

    Industry Trends and Perspectives

    Storage trends

    A few weeks ago I attended AWS re:invent 2014 in Las Vegas for a few days. For those of you who have not yet attended this event, I recommend adding it to your agenda. If you have interest in compute servers, networking, storage, development tools or management of cloud (public, private, hybrid), virtualization and related topic themes, you should check out AWS re:invent. For those who need a AWS primer or refresher visit here.

    AWS made several announcements at re:invent including many around development tools, compute and data storage services. One of those to keep an eye on is cloud based Aurora relational database service that complement existing RDS tools. Aurora is positioned as an alternative to traditional SQL based transactional databases commonly found in enterprise environments (e.g. SQL Server, IBM DB2/UDB, Oracle among others). I will put some additional notes and perspectives together in a StorageIOblog post along with some video from AWS soon.

    Commentary In The News

    StorageIO news

    Following are some StorageIO industry trends perspectives comments that have appeared in various venues. Cloud conversations continue to be popular including concerns about privacy, security and availability.

    Over at Processor: Comments on Datacenters, Decide Whether To Build Or Not To Build, and controlling storage costs via insight and action. EdTechMagazine: has some comments on IaaS and Is Lean IT Here to Stay, while at CyberTrend perspectives on Better Servers for Better Business.

    Across the pond over at the UK based Computerweekly comments on AWS launching Aurora cloud-based relational database engine, and hybrid cloud storage. Some comments on Overland Storage RAINcloud can be found at SearchStorage, while SearchDatabackup has some comments on Symantec break-up makeing sense for storage.

    For those of you who speak Dutch, here is an interview (via it-infra.nl) I did when Holland earlier this year about storage and your business.

    View other industry trends comments here

    Tips and Articles

    View recent as well as past tips and articles here

    StorageIOblog posts

    Recent StorageIOblog posts include:

    View other recent as well as past blog posts here

    In This Issue

  • Industry Trends Perspectives
  • Commentary in the news
  • Tips and Articles
  • StorageIOblog posts

    • Events & Activities

    November 11-13, 2014
    AWS re:Invent Las Vegas

    View other recent and upcoming events here

    Webinars

    December 11, 2014 – BrightTalk
    Server & Storage I/O Performance

    December 10, 2014 – BrightTalk
    Server & Storage I/O Decision Making

    December 9, 2014 – BrightTalk
    Virtual Server and Storage Decision Making

    December 3, 2014 – BrightTalk
    Data Protection Modernization

    November 13 9AM PT – BrightTalk
    Software Defined Storage

    November 11 10AM PT
    Google+ Hangout Dell BackupU

    November 11 9AM PT – BrightTalk
    Software Defined Data Centers

    Videos and Podcasts

    VMworld 2014 review
    Video: Click to view VMworld 2014 update

    StorageIO podcasts are also available via and at StorageIO.tv

    From StorageIO Labs

    Research, Reviews and Reports

    Lenovo ThinkServer TD340
    Earlier this year I did a review of the Lenovo ThinkServer TS140 in the StorageIO Labs (see the review here), in fact I ended up buying a TS140 after the review, and a few months back picked up yet another one. This StorageIOlab review looks at the Lenovo ThinkServer TD340 Tower Server which besides having a larger model number than the TS140, it also has a lot more capabilities (server compute, memory, I/O slots and internal hot-swap storage bays. Read more about the TD340 here.

    Resources and Links

    Check out these useful links and pages:
    storageio.com/links
    objectstoragecenter.com
    storageioblog.com/data-protection-diaries-main/
    storageio.com/ssd
    storageio.com/ssd

    Ok, nuff said (for now)

    Cheers
    Gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved