Tag: algorithms

Posted on

Software Defined Storage Virtual Hard Disk (VHD) Algorithms + Data Structures

Software Defined Storage Virtual Hard Disk (VHD) Algorithms + Data Structures

server storage I/O trends

For those who are into, or simply like to talk about software defined storage (SDS), APIs, Windows, Virtual Hard Disks (VHD) or VHDX, or Hyper-V among other related themes, have you ever actually looked at the specification for VHDX? If not, here is the link to the open specification that Microsoft published (this one dates back to 2012).

Microsoft VHDX specification document
Click on above image to download the VHDX specification from Microsoft.com

How about Algorithms + Data Structures = Programs by Niklaus Wirth, some of you might remember that from the past, if not, it’s a timeless piece of work that has many fundamental concepts for understanding software defined anything. I came across Algorithms + Data Structures = Programs back in Graduate School when I was getting my masters degree in Software Engineering at night, while working during the day in an IT environment on servers, storage, I/O networking hardware and software.


Algorithms + Data Structures = Programs on Amazon.com

In addition to the Amazon.com link above, here is a link to a free (legitimate PDF) copy.

The reason I mention Software Defined, Virtual Hard Disk and Algorithms + Data Structures = Programs is that they are all directly related, or at a minimum can help demystify things.

Inside a VHD and VHDX

The following is an excerpt from the Microsoft VHDX specification document mentioned above that shows a logical view of how a VHDX is defined as a data structure, as well as how algorithms should use and access them.

Microsoft VHDX specification

Keep in mind that anything software defined is a collection of data structures that describe how bits, bytes, blocks, blobs or other entities are organized and then accessed by algorithms that are defined how to use those data structures. Thus the connection to Algorithms + Data Structures = Programs mentioned above.

In the case of a Virtual Hard Disk (VHD) or VHDX they are the data structures defined (see the specification here) and then used by various programs (applications or algorithms) such as Windows or other operating systems, hypervisors or utilities.

A VHDX (or VMDK or VVOL or qcow or other virtual disk for that matter) is a file whose contents are organized e.g. the data structures per a given specification (here).

The VHDX can then be moved around like another file and used for booting some operating systems, as well as simply mounting and using like any other disk or device.

This also means that you can nest putting a VHDX inside of a VHDX and so forth.

Where to learn more

Continue reading with the following links about Virtual Hard Disks pertaining to Microsoft Windows, Hyper-V, VMware among others.

  • Algorithms + Data Structures = Programs on Amazon.com
  • Microsoft Technet Virtual Hard Disk Sharing Overview
  • Download the VHDX specification from Microsoft.com
  • Microsoft Technet Hyper-V Virtual Hard Disk (VHD) Format Overview
  • Microsoft Technet Online Virtual Hard Disk Resizing Overview
  • VMware Developer Resource Center (VDDK for vSphere 6.0)
  • VMware VVOLs and storage I/O fundamentals (Part 1)

    • What this all means

    Applications and utilities or basically anything that is algorithms working with data structures is a program. Software Defined Storage or Software Defined anything involves defining data structures that describes various entities, along with the algorithms to work with and use those data structures.

    Sharpen, refresh or expand your software defined data center, software defined network, software defined storage or software defined storage management as well as software defined marketing game by digging a bit deeper into the bits and bytes. Who knows, you might just go from talking the talk to walking the talk, if nothing else, talking the talk better..

    Ok, nuff said (for now)

    Cheers
    Gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2023 Server StorageIO(R) and UnlimitedIO All Rights Reserved

    Posted on

    The Human Face of Big Data, a Book Review

    StorageIO industry trends cloud, virtualization and big data

    My copy of the new book The Human Face of Big Data created by Rick Smolan and Jennifer Erwitt arrived yesterday compliments of EMC (the lead sponsor). In addition to EMC, the other sponsors of the book are Cisco, VMware, FedEx, Originate and Tableau software.

    To say this is a big book would be an understatement, then again, big data is a big topic with a lot of diversity if you open your eyes and think in a pragmatic way, which once you open and see the pages you will see. This is physically a big book (11x 14 inches) with lots of pictures, texts, stories, factoids and thought stimulating information of the many facets and dimensions of big data across 224 pages.

    While Big Data as a buzzword and industry topic theme might be new, along with some of the related technologies, techniques and focus areas, other as aspects have been around for some time. Big data means many things to various people depending on their focus or areas of interest ranging from analytics to images, videos and other big files. A common theme is the fact that there is no such thing as an information or data recession, and that people and data are living longer, getting larger, and we are all addicted to information for various reasons.

    Big data needs to be protected and preserved as it has value, or its value can increase over time as new ways to leverage it are discovered which also leads to changing data access and life cycle patterns. With many faces, facets and areas of interests applying to various spheres of influence, big data is not limited to programmatic, scientific, analytical or research, yet there are many current and use cases in those areas.

    Big data is not limited to videos for security surveillance, entertainment, telemetry, audio, social media, energy exploration, geosciences, seismic, forecasting or simulation, yet those have been areas of focus for years. Some big data files or objects are millions of bytes (MBytes), billion of bytes (GBytes) or trillion of bytes (TBytes) in size that when put into file systems or object repositories, add up to Exabytes (EB – 1000 TBytes) or Zettabytes (ZB – 1000 EBs). Now if you think those numbers are far-fetched, simply look back to when you thought a TByte, GByte let alone a MByte was big or far-fetched future. Remember, there is no such thing as a data or information recession, people and data are living longer and getting larger.

    Big data is more than hadoop, map reduce, SAS or other programmatic and analytical focused tool, solution or platform, yet those all have been and will be significant focus areas in the future. This also means big data is more than data warehouse, data mart, data mining, social media and event or activity log processing which also are main parts have continued roles going forward. Just as there are large MByte, GByte or TByte sized files or objects, there are also millions and billions of smaller files, objects or pieces of information that are part of the big data universe.

    You can take a narrow, product, platform, tool, process, approach, application, sphere of influence or domain of interest view towards big data, or a pragmatic view of the various faces and facets. Of course you can also spin everything that is not little-data to be big data and that is where some of the BS about big data comes from. Big data is not exclusive to the data scientist, researchers, academia, governments or analysts, yet there are areas of focus where those are important. What this means is that there are other areas of big data that do not need a data science, computer science, mathematical, statistician, Doctoral Phd or other advanced degree or training, in other words big data is for everybody.

    Cover image of Human Face of Big Data Book

    Back to how big this book is in both physical size, as well as rich content. Note the size of The Human Face of Big Data book in the adjacent image that for comparison purposes has a copy of my last book Cloud and Virtual Data Storage Networking (CRC), along with a 2.5 inch hard disk drive (HDD) and a growler. The Growler is from Lift Bridge Brewery (Stillwater, MN), after all, reading a big book about big data can create the need for a big beer to address a big thirst for information ;).

    The Human Face of Big Data is more than a coffee table or picture book as it is full of with information, factoids and perspectives how information and data surround us every day. Check out the image below and note the 2.5 inch HDD sitting on the top right hand corner of the page above the text. Open up a copy of The Human Face of Big Data and you will see examples of how data and information are all around us, and our dependence upon it.

    A look inside the book The Humand Face of Big Data image

    Book Details:
    Copyright 2012
    Against All Odds Productions
    ISBN 978-1-4549-0827-2
    Hardcover 224 pages, 11 x 0.9 x 14 inches
    4.8 pounds, English

    There is also an applet to view related videos and images found in the book at HumanFaceofBigData.com/viewer in addition to other material on the companion site www.HumanFacesofBigData.com.

    Get your copy of
    The Human Face of Big Data at Amazon.com by clicking here or at other venues including by clicking on the following image (Amazon.com).

    Some added and related material:
    Little data, big data and very big data (VBD) or big BS?
    How many degrees separate you and your information?
    Hardware, Software, what about Valueware?
    Changing Lifecycles and Data Footprint Reduction (Data doesnt have to lose value over time)
    Garbage data in, garbage information out, big data or big garbage?
    Industry adoption vs. industry deployment, is there a difference?
    Is There a Data and I/O Activity Recession?
    Industry trend: People plus data are aging and living longer
    Supporting IT growth demand during economic uncertain times
    No Such Thing as an Information Recession

    For those who can see big data in a broad and pragmatic way, perhaps using the visualization aspect this book brings forth the idea that there are and will be many opportunities. Then again for those who have a narrow or specific view of what is or is not big data, there is so much of it around and various types along with focus areas you too will see some benefits.

    Do you want to play in or be part of a big data puddle, pond, or lake, or sail and explore the oceans of big data and all the different aspects found in, under and around those bigger broader bodies of water.

    Bottom line, this is a great book and read regardless of if you are involved with data and information related topics or themes, the format and design lend itself to any audience. Broaden your horizons, open your eyes, ears and thinking to the many facets and faces of big data that are all around us by getting your copy of The Human Face of Big Data (Click here to go to Amazon for your copy) book.

    Ok, nuff said.

    Cheers gs

    Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press, 2011), The Green and Virtual Data Center (CRC Press, 2009), and Resilient Storage Networks (Elsevier, 2004)

    twitter @storageio

    All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved