Tag: Benchmarking

Posted on

Big Files Lots of Little File Processing Benchmarking with Vdbench

Big Files Lots of Little File Processing Benchmarking with Vdbench


server storage data infrastructure i/o File Processing Benchmarking with Vdbench

Updated 2/10/2018

Need to test a server, storage I/O networking, hardware, software, services, cloud, virtual, physical or other environment that is either doing some form of file processing, or, that you simply want to have some extra workload running in the background for what ever reason? An option is File Processing Benchmarking with Vdbench.

I/O performance

Getting Started


Here’s a quick and relatively easy way to do it with Vdbench (Free from Oracle). Granted there are other tools, both for free and for fee that can similar things, however we will leave those for another day and post. Here’s the con to this approach, there is no Uui Gui like what you have available with some other tools Here’s the pro to this approach, its free, flexible and limited by your creative, amount of storage space, server memory and I/O capacity.

If you need a background on Vdbench and benchmarking, check out the series of related posts here (e.g. www.storageio.com/performance).

Get and Install the Vdbench Bits and Bytes


If you do not already have Vdbench installed, get a copy from the Oracle or Source Forge site (now points to Oracle here).

Vdbench is free, you simply sign-up and accept the free license, select the version down load (it is a single, common distribution for all OS) the bits as well as documentation.

Installation particular on Windows is really easy, basically follow the instructions in the documentation by copying the contents of the download folder to a specified directory, set up any environment variables, and make sure that you have Java installed.

Here is a hint and tip for Windows Servers, if you get an error message about counters, open a command prompt with Administrator rights, and type the command:

$ lodctr /r


The above command will reset your I/O counters. Note however that command will also overwrite counters if enabled so only use it if you have to.

Likewise *nix install is also easy, copy the files, make sure to copy the applicable *nix shell script (they are in the download folder), and verify Java is installed and working.

You can do a vdbench -t (windows) or ./vdbench -t (*nix) to verify that it is working.

Vdbench File Processing

There are many options with Vdbench as it has a very robust command and scripting language including ability to set up for loops among other things. We are only going to touch the surface here using its file processing capabilities. Likewise, Vdbench can run from a single server accessing multiple storage systems or file systems, as well as running from multiple servers to a single file system. For simplicity, we will stick with the basics in the following examples to exercise a local file system. The limits on the number of files and file size are limited by server memory and storage space.

You can specify number and depth of directories to put files into for processing. One of the parameters is the anchor point for the file processing, in the following examples =S:\SIOTEMP\FS1 is used as the anchor point. Other parameters include the I/O size, percent reads, number of threads, run time and sample interval as well as output folder name for the result files. Note that unlike some tools, Vdbench does not create a single file of results, rather a folder with several files including summary, totals, parameters, histograms, CSV among others.


Simple Vdbench File Processing Commands

For flexibility and ease of use I put the following three Vdbench commands into a simple text file that is then called with parameters on the command line.
fsd=fsd1,anchor=!fanchor,depth=!dirdep,width=!dirwid,files=!numfiles,size=!filesize

fwd=fwd1,fsd=fsd1,rdpct=!filrdpct,xfersize=!fxfersize,fileselect=random,fileio=random,threads=!thrds

rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=!etime,interval=!itime

Simple Vdbench script

# SIO_vdbench_filesystest.txt

#

# Example Vdbench script for file processing

#

# fanchor = file system place  where directories and files will be created

# dirwid = how wide should the directories be (e.g. how many directories wide)

# numfiles = how many files per directory

# filesize = size in in k, m, g e.g. 16k = 16KBytes

# fxfersize = file I/O transfer size in kbytes

# thrds = how many threads or workers

# etime = how long to run in minutes (m) or hours (h)

# itime = interval sample time e.g. 30 seconds

# dirdep = how deep the directory tree

# filrdpct = percent of reads e.g. 90 = 90 percent reads

# -p processnumber = optional specify a process number, only needed if running multiple vdbenchs at same time, number should be unique

# -o output file that describes what being done and some config info

#

# Sample command line shown for Windows, for *nix add ./

#

# The real Vdbench script with command line parameters indicated by !=

#


fsd=fsd1,anchor=!fanchor,depth=!dirdep,width=!dirwid,files=!numfiles,size=!filesize


fwd=fwd1,fsd=fsd1,rdpct=!filrdpct,xfersize=!fxfersize,fileselect=random,fileio=random,threads=!thrds


rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=!etime,interval=!itime

Big Files Processing Script


With the above script file defined, for Big Files I specify a command line such as the following.
$ vdbench -f SIO_vdbench_filesystest.txt fanchor=S:\SIOTemp\FS1 dirwid=1 numfiles=60 filesize=5G fxfersize=128k thrds=64 etime=10h itime=30 numdir=1 dirdep=1 filrdpct=90 -p 5576 -o SIOWS2012R220_NOFUZE_5Gx60_BigFiles_64TH_STX1200_020116

Big Files Processing Example Results


The following is one of the result files from the folder of results created via the above command for Big File processing showing totals.



Run totals


21:09:36.001 Starting RD=format_for_rd1


Feb 01, 2016 .Interval. .ReqstdOps.. ...cpu%...  read ....read.... ...write.... ..mb/sec... mb/sec .xfer.. ...mkdir... ...rmdir... ..create... ...open.... ...close... ..delete...

                          rate  resp total  sys   pct   rate  resp   rate  resp  read write  total    size  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp

21:23:34.101   avg_2-28 2848.2  2.70   8.8 8.32   0.0    0.0  0.00 2848.2  2.70  0.00 356.0 356.02  131071   0.0  0.00   0.0  0.00   0.1 109176   0.1  0.55   0.1  2006   0.0  0.00


21:23:35.009 Starting RD=rd1; elapsed=36000; fwdrate=max. For loops: None


07:23:35.000 avg_2-1200 4939.5  1.62  18.5 17.3  90.0 4445.8  1.79  493.7  0.07 555.7 61.72 617.44  131071   0.0  0.00   0.0  0.00   0.0  0.00   0.1  0.03   0.1  2.95   0.0  0.00

Lots of Little Files Processing Script


For lots of little files, the following is used.



$ vdbench -f SIO_vdbench_filesystest.txt fanchor=S:\SIOTEMP\FS1 dirwid=64 numfiles=25600 filesize=16k fxfersize=1k thrds=64 etime=10h itime=30 dirdep=1 filrdpct=90 -p 5576 -o SIOWS2012R220_NOFUZE_SmallFiles_64TH_STX1200_020116

Lots of Little Files Processing Example Results


The following is one of the result files from the folder of results created via the above command for Big File processing showing totals.
Run totals


09:17:38.001 Starting RD=format_for_rd1


Feb 02, 2016 .Interval. .ReqstdOps.. ...cpu%...  read ....read.... ...write.... ..mb/sec... mb/sec .xfer.. ...mkdir... ...rmdir... ..create... ...open.... ...close... ..delete...

                          rate  resp total  sys   pct   rate  resp   rate  resp  read write  total    size  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp  rate  resp

09:19:48.016    avg_2-5  10138  0.14  75.7 64.6   0.0    0.0  0.00  10138  0.14  0.00 158.4 158.42   16384   0.0  0.00   0.0  0.00 10138  0.65 10138  0.43 10138  0.05   0.0  0.00


09:19:49.000 Starting RD=rd1; elapsed=36000; fwdrate=max. For loops: None


19:19:49.001 avg_2-1200 113049  0.41  67.0 55.0  90.0 101747  0.19  11302  2.42 99.36 11.04 110.40    1023   0.0  0.00   0.0  0.00   0.0  0.00  7065  0.85  7065  1.60   0.0  0.00

Where To Learn More

View additional NAS, NVMe, SSD, NVM, SCM, Data Infrastructure and HDD related topics via the following links.

Additional learning experiences along with common questions (and answers), as well as tips can be found in Software Defined Data Infrastructure Essentials book.

Software Defined Data Infrastructure Essentials Book SDDC

What This All Means

The above examples can easily be modified to do different things particular if you read the Vdbench documentation on how to setup multi-host, multi-storage system, multiple job streams to do different types of processing. This means you can benchmark a storage systems, server or converged and hyper-converged platform, or simply put a workload on it as part of other testing. There are even options for handling data footprint reduction such as compression and dedupe.

Ok, nuff said, for now.

Gs

Greg Schulz - Microsoft MVP Cloud and Data Center Management, VMware vExpert 2010-2017 (vSAN and vCloud). Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of Server StorageIO.

Posted on

NAD recommends Oracle discontinue certain Exadata performance claims

I Received the following press release in my inbox today from the National Advertising Division (NAD) recommending that Oracle stop making certain performance claims about Exadata after a complaint from IBM.

Oracle Exadata

In case you are not familiar with ExaData, it is a database machine or storage appliance that only supports Oracle database systems (learn more here). Oracle having bought Sun microsystems a few years back moved from being a software vendor that competed with other vendors software solutions including those from IBM while running on hardware from Dell, HP and IBM among others. Now that Oracle is in the hardware business, while you will still find Oracle software products running on their competitors hardware (servers and storage), Oracle is also more aggressively competing with those same partners, particularly IBM.

Hmm, to quote Scooby Doo: Rut Roh!

Looks like IBM complained to the Better Business Bureau (BBB) National Advertising Division (NAD) that resulted in the Advertising Self-Regulatory Council (ASRC) making their recommendation below (more about NAD and ASRC can be found here). Based on a billboard sign that I saw while riding from JFK airport into New York City last week, I would not be surprised if a company with two initials that start with an H and end with a P were to file a similar complaint.

I Wonder if the large wall size Oracle advertisement that used to be in the entry way to the white plains (IATA:HPN) airport (e.g. in IBM’s backyard) welcoming you to the terminal as you get off the airplanes is still there?

The following is the press release that I received:

For Immediate Release
Contact: Linda Bean
212.705.0129

NAD Finds Oracle Took Necessary Action in Discontinuing Comparative Performance Claims for Exadata; Oracle to Appeal NAD Decision

New York, NY – July 24,  2012 –TheNational Advertising Division has recommended that Oracle Corporation discontinue certain comparative product-performance claims for the company’s Exadata database machines, following a challenge by International Business Machines Corporation. Oracle said it would voluntarily discontinue the challenged claims, but noted that it would appeal NADs decision to the National Advertising Review Board.

The advertising claims at issue appeared in a full-page advertisement in the Wall Street Journal and included the following:

  • “Exadata 20x Faster … Replaces IBM Again”
  • “Giant European Retailer Moves Databases from IBM Power to Exadata … Runs 20 Times Faster”

NAD also considered whether the advertising implied that all Oracle Exadata systems are twenty times faster than all IBM Power systems.

The advertisement featured the image of an Oracle Exadata system, along with the statement: “Giant European Retailer Moves Databases from IBM Power to Exadata Runs 20 Times Faster.” The advertisement also offered a link to the Oracle website: “For more details oracle.com/EuroRetailer.” 

IBM argued that the “20x Faster” claim makes overly broad references to “Exadata” and “IBM Power,” resulting in a misleading claim, which the advertiser’s evidence does not support.  In particular, the challenger argued that by referring to the brand name “IBM Power” without qualification, Oracle was making a broad claim about the entire IBM Power systems line of products. 

The advertiser, on the other hand, argued that the advertisement represented a case study, not a line claim, and noted that the sophisticated target audience would understand that the advertisement is based on the experience of one customer – the “Giant European Retailer” referenced in the advertisement.

In a NAD proceeding, the advertiser is obligated to support all reasonable interpretations of its advertising claims, not just the message it intended to convey.   In the absence of reliable consumer perception evidence, NAD uses its experienced judgment to determine what implied messages, if any, are conveyed by an advertisement.   When evaluating the message communicated by an advertising claim, NAD will examine the claims at issue in the context of the entire advertisement in which they appear.

In this case, NAD concluded that while the advertiser may have intended to convey the message that in one case study a particular Exadata system was up to 20 times faster when performing two particular functions than a particular IBM Power system, Oracle’s general references to “Exadata” and “IBM Power,” along with the bold unqualified headline “Exadata 20x Faster Replaces IBM Again,” conveyed a much broader message.

NAD determined that at least one reasonable interpretation of the challenged advertisement is that all – or a vast majority – of Exadata systems consistently perform 20 times faster in all or many respects than all – or a vast majority – of IBM Power systems. NAD found that the message was not supported by the evidence in the record, which consisted of one   particular comparison of one consumer’s specific IBM Power system to a specific Exadata System. 

NAD further determined that the disclosure provided on the advertiser’s website was not sufficient to limit the broad message conveyed by the “20x Faster” claim. More importantly, NAD noted that even if Oracle’s website disclosure was acceptable – and had appeared clearly and conspicuously in the challenged advertisement – it would still be insufficient because an advertiser cannot use a disclosure to cure an otherwise false claim.

NAD noted that Oracle’s decision to permanently discontinue the claims at issue was necessary and proper.

Oracle, in its advertiser’s statement, said it was “disappointed with the NAD’s decision in this matter, which it believes is unduly broad and will severely limit the ability to run truthful comparative advertising, not only for Oracle but for others in the commercial hardware and software industry.”

Oracle noted that it would appeal all of NAD’s findings in the matter.

 

###

NAD’s inquiry was conducted under NAD/CARU/NARB Procedures for the Voluntary Self-Regulation of National Advertising.  Details of the initial inquiry, NAD’s decision, and the advertiser’s response will be included in the next NAD/CARU Case Report.

About Advertising Industry Self-Regulation:  The Advertising Self-Regulatory Council establishes the policies and procedures for advertising industry self-regulation, including the National Advertising Division (NAD), Children’s Advertising Review Unit (CARU), National Advertising Review Board (NARB), Electronic Retailing Self-Regulation Program (ERSP) and Online Interest-Based Advertising Accountability Program (Accountability Program.) The self-regulatory system is administered by the Council of Better Business Bureaus.

Self-regulation is good for consumers. The self-regulatory system monitors the marketplace, holds advertisers responsible for their claims and practices and tracks emerging issues and trends. Self-regulation is good for advertisers. Rigorous review serves to encourage consumer trust; the self-regulatory system offers an expert, cost-efficient, meaningful alternative to litigation and provides a framework for the development of a self-regulatory to emerging issues.

To learn more about supporting advertising industry self-regulation, please visit us at: www.asrcreviews.org.

 

 

Linda Bean l Director, Communications,
Advertising Self-Regulatory Council

Tel: 212.705.0129
Cell: 908.812.8175
lbean@asrc.bbb.org

112 Madison Ave.
3rd Fl.
New York, NY
10016

 

Ok, Oracle is no stranger to benchmark and performance claims controversy having amassed several decades of experience. Anybody remember the silver bullet database test from late 80s early 90s when Oracle set a record performance except that they never committed the writes to disk?

Something tells me that Oracle and Uncle Larry (e.g. Larry Ellison who is not really my uncle) will treat this as any kind of press or media coverage is good and probably will issue something like IBM must be worried if they have to go to the BBB.

Will a complaint which I’m sure is not the fist to be lodged with the BBB against Oracle deter customers, or be of more use to IBM sales and their partners in deals vs. Oracle?

What’s your take?

Is this much ado about nothing, a filler for a slow news or discussion day, a break from talking about VMware acquisition of Nicira or VMware CEO management changes? Perhaps this is an alternative to talking about the CEO of SSD vendor STEC being charged with insider trading, or something other than Larry Ellison buying an Hawaiian island (IMHO he could have gotten a better deal buying Greece), or is this something that Oracle will need to take seriously?

Ok, nuff said for now

Cheers Gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press, 2011), The Green and Virtual Data Center (CRC Press, 2009), and Resilient Storage Networks (Elsevier, 2004)

twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2012 StorageIO and UnlimitedIO All Rights Reserved

Posted on

SPC and Storage Benchmarking Games

Storage I/O trends

There is a post over in one of the LinkedIn Discussion forums about storage performance council (SPC) benchmarks being miss-leading that I just did a short response post to. Here’s the full post as LinkedIn has a short post response limit.

While the SPC is far from perfect, it is at least for block, arguably better than doing nothing.

For the most part, SPC has become a de facto standard for at least block storage benchmarks independent of using IOmeter or other tools or vendor specific simulations, similar how MSFT ESRP is for exchange, TPC for database, SPEC for NFS and so forth. In fact, SPC even recently rather quietly rolled out a new set of what could be considered the basis for Green storage benchmarks. I would argue that SPC results in themselves are not misleading, particularly if you take the time to look at both the executive and full disclosures and look beyond the summary.

Some vendors have taken advantage of the SPC results playing games with discounting on prices (something that’s allowed under SPC rules) to show and make apples to oranges comparisons on cost per IOP or other ploys. This proactive is nothing new to the IT industry or other industries for that matter, hence benchmark games.

Where the misleading SPC issue can come into play is for those who simply look at what a vendor is claiming and not looking at the rest of the story, or taking the time to look at the results and making apples to apples, instead of believing the apples to oranges comparison. After all, the results are there for a reason. That reason is for those really interested to dig in and sift through the material, granted not everyone wants to do that.

For example, some vendors can show a highly discounted list price to get a better IOP per cost on an apple to oranges basis, however, when processes are normalized, the results can be quite different. However here’s the real gem for those who dig into the SPC results, including looking at the configurations and that is that latency under workload is also reported.

The reason that latency is a gem is that generally speaking, latency does not lie.

What this means is that if vendor A doubles the amount of cache, doubles the number of controllers, doubles the number of disk drives, plays games with actual storage utilization (ASU), utilizes fast interfaces from 10 GbE  iSCSI to 8Gb FC or FCoE or SAS to get a better cost per IOP number with discounting, look at the latency numbers. There have been some recent examples of this where vendor A has a better cost per IOP while achieving a higher number of IOPS at a lower cost compared to vendor B, which is what is typically reported in a press release or news story. (See a blog entry that also points to a CMG presentation discussion around this topic here.

Then go and look at the two results, vendor B may be at list price while vendor A is severely discounted which is not a bad thing, as that is then the starting list price as to which customers should start negotiations. However to be fair, normalize the pricing for fun, look at how much more equipment vendor A may need while having to discount to get the price to offset the increased amount of hardware, then look at latency.

In some of the recent record reported results, the latency results are actually better for a vendor B than for a vendor A and why does latency matter? Beyond showing what a controller can actually do in terms of levering  the number of disks, cache, interface ports and so forth, the big kicker is for those talking about SSD (RAM or FLASH) in that SSD generally is about latency. To fully effectively utilize SSD which is a low latency device, you would want a controller that can do a decent job at handling IOPS; however you also need a controller that can do a decent job of handling IOPS with low latency under heavy workload conditions.

Thus the SPC again while far from perfect, at least for a thumb nail sketch and comparison is not necessarily misleading, more often than not it’s how the results are utilized that is misleading. Now in the quest for the SPC administrators to try and gain more members and broader industry participation and thus secure their own future, is the SPC organization or administration opening itself up to being used more and more as a marketing tool in ways that potentially compromise all the credibility (I know, some will dispute the validity of SPC, however that’s reserved for a different discussion ;) )?

There is a bit of Déjà here for those involved with RAID and storage who recall how the RAID Advisory Board (RAB) in its quest to gain broader industry adoption and support succumbed to marketing pressures and use or what some would describe as miss-use and is now a member of the “Where are they now” club!

Don’t get me wrong here; I like the SPC tests/results/format, there is a lot of good information in the SPC. The various vendor folks who work very hard behind the scenes to make the SPC actually work and continue to evolve it also all deserve a great big kudos, an “atta boy” or “atta girl” for the fine work that have been doing, work that I hope does not become lost in the quest to gain market adoption for the SPC.

Ok, so then this should all then beg the question of what is the best benchmark. Simple, the one that most closely resembles your actual applications, workload, conditions, configuration and environment.

Ok, nuff said.

Cheers gs

Greg Schulz – Author Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networks (Elsevier)
twitter @storageio

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2024 Server StorageIO and UnlimitedIO LLC All Rights Reserved