Frequently Asked Questions
This document answers frequently-asked questions about WebStone.
This is a list of answers to Frequently Asked Questions (FAQ) about WebStone.
The latest copy is always available at http://www.mindcraft.com/webstone
and via the WebStone mailing list. The FAQ is periodically posted to the
WebStone mailing list, and to the USENET newsgroup
WebStone 2.x is a rewrite of the WebStone 1.1 code. Significant changes
have been made both in the code and in the fileset and run rules. Many bugs
were eliminated, support for other platforms has been included and many
new features have been added. The WebStone 1.1 and WebStone 2.x numbers
cannot be compared, since so much has changed. In general, WebStone 1.1
will give higher connections/second values, but lower throughput numbers
than WebStone 2.x.
Absolutely NOT! WebStone 1.1 numbers are based on a different fileset, as
well as an older version of the benchmarking software. The WebStone 1.1
fileset was based on a fileset with a smaller average filesize, so that
the number of connections per second will tend to be higher (all things
being equal). The WebStone 2.x fileset is based on observations of several
real world sites, and the distribution of the filesizes found there. This
fileset is also similar to the fileset chosen by the SPEC committee for
While it is possible to convert the 1.1 fileset to a 2.x format and then
test it, the resulting numbers will not be the same, because the underlying
software used to perform the testing has changed. WebStone 1.1 was also
heavily abused because of the lack of run rules, and reporting rules. It
is recommended that everyone move to WebStone 2.x.
WebStone is a highly-configurable client-server benchmark for HTTP servers.
The original WebStone benchmark was released in March, 1995. The original
white paper describing this benchmark is available from
WebStone 2.x.1 is not a proprietary benchmark - it is an open benchmark. The source
code is freely available, and anyone can examine it. By design, WebStone
does not unfairly favor SGI, Netscape, or any other company - it is simply
a performance measurement tool.
WebStone makes a user-configurable number of HTTP 1.0 GET requests for specific
pages on a Web server. Any Web server can be tested, and any HTML content
can be used.
WebStone measures the throughput and latency of each HTTP transfer. By default,
only statistical data are returned, but the user may optionally request
data for each and every transaction. WebStone also reports transaction failures,
which translate into those little "Connection Refused" alerts in the real
WebStone 2.x includes support for testing proxy servers, as well as more
flexible handling of URL's that enable WebStone to test a wide variety of
content types. The code has also been significantly rewritten so that it
is more robust and portable.
WebStone does not yet do any of the following (listed roughly in order of
- POST transactions, widely used for CGI-bin scripts
If you have additional requests for WebStone functionality, contact the
WebStone mailing list.
The latest copy of WebStone, and of this FAQ, is available at
WebStone includes a README file which may answer some of your questions.
However, here's a brief overview.
- Set up your test-bed
- Load WebStone onto your webmaster
- Write a file list
- Start the benchmark
- Collect the results
To try the GUI, make sure you have a Web browser, and run
./webstone -gui from the WebStone base directory. You don't need to hand-edit the
testbed file anymore, but you still need to edit
filelist if you want to change the workload. This may not be necessary, since we've
distributed two real-world workload models with WebStone.
These are the stepts to follow to run the GUI
- Set up your test-bed
- Load WebStone onto your webmaster
If the GUI appears to hang, you can kill stray WebStone processes with
Your test bed should include, at minimum, two machines and a network. The
first machine is your Web server - it can be any HTTP 1.0-compliant server.
As far as WebStone is concerned, it's a black box.
You'll also need a webmaster and one or more webclients. These should be
Unix hosts, since WebStone hasn't been tested on any non-Unix operating
systems (feel free to port it, if you like). The webmaster and the webclient
may be the same machine, if desired: we've run up to 120 webclients and
the webmaster on a single 32MB Indy.
You must establish a trust relationship between your webmaster and webclients.
Each webclient must be set up so that the webmaster can use
rexec to execute the WebStone on the client. This can be done with a guest account.
It's also helpful if root can
rcp to the webclients, and even to the web server. This requires editing the
/etc/host.equiv files. Here's an example:
/.rhosts (on each webclient)
/etc/hosts/equiv (on each webclient)
To make best use of WebStone, your webmaster should be equipped with a C
compiler, Perl, awk, and a Web browser. A data analysis program such as
GnuPlot may also come in handy.
Connect the webclients, the webmaster, and the web server to a common network.
To check your setup, load a browser on one of the webclients, and make sure
it can connect to the Web server.
Copy the WebStone distribution onto your webmaster. If your webmaster isn't
an SGI IRIX 5.3 machine, you'll have to make the binaries. Type make from the WebStone directory - this creates the following binaries:
Common porting errors
- If you want to use gcc instead of cc, change the CC variable in
- Many System V-based Unix implementations (such as Solaris 2.x) will need
LIBS = -lsocket -lnsl in
- Some users may also need to comment out the definition of
If you encounter other errors, please contact the WebStone mailing list.
make install to put the binaries in the
When you run WebStone, the
distribute script automatically copies the
webclient binary to the other client systems. If you're running diverse clients (e.g.,
a couple Suns, a couple BSD hosts), you'll want to comment the
distribute script out of
bin/runbench, and distribute host-specific versions of
webclient by hand.
If you use the
webstone script to automate WebStone, you'll want to edit the
conf/testbed script. The
testbed script contains several configurable parameters that WebStone relies on.
Here is an example:
### BENCHMARK PARAMETERS -- EDIT THESE AS REQUIRED
### SERVER PARAMETERS -- EDIT AS REQUIRED
# WE NEED AN ACCOUNT WITH A FIXED PASSWORD, SO WE CAN REXEC
# THE WEBSTONE CLIENTS
CLIENTS="webstone1 webstone2 webstone3 webstone4 webstone5"
Briefly, the first set of parameters means that the WebStone benchmark will
run from 8 clients to 128 clients, in increments of 8. Each increment will
run for 30 minutes, and the whole test will be repeated three times. This
test suite would take roughly 24 hours to complete.
Why multiple iterations? The WebStone benchmark is a stochastic process
so there will be variation from run to run, especially if your test file
sets have large files or if you approach overloading the server. 3 iterations
is about the minimum you should run just to see if there is variation and
to gauge the amount of variation. the TIMEPERRUN needs to be long enough to establish a steady state and allow it to dominate
the run. 30 minutes seems to be enough if the sizes of the files are small.
You may want to run the benchmark longer per run to minimize variation if
the files are large.
The second set of parameters means that we will test a server called "www"
at port 80 (note that the port number may be changed to accomodate proxy
servers or multiple servers on the same host). We will use four clients.
Also, we specify the location of a system tuning file (on Sun Solaris, one
could use /etc/system), and web server tuning files (specified for Netscape).
These files will be copied into the
runs subdirectories for later reference.
Finally, we specify the WebStone account on the clients. Here, we use the
guest account, with a fixed password: guest.
The basic WebStone tests expect a set of files to reside on the server to
be retrieved by the webstone client programs. The file list tells WebStone which files to retrieve.
It's possible to use an arbitrary set of fixed-length files for WebStone.
Although these files have the .html extension, they are used to represent files of many types. Basically we
treat "bits-as-bits". You can use the programs in the genfileset subdirectory to create the needed set of files, and copy them onto your
The sample file list shipped with WebStone uses the files created by genfiles:
# Sample filelist, abstracted from access logs
/file500.html 350 #500
/file5k.html 500 #5125
/file50k.html 140 #51250
/file500k.html 9 #512500
/file5m.html 1 #5248000
This filelist consists of 5 different files. The number following the filename
is the weight of this file in the distribution. All the weights are summed
together and the frequency of each file is the weight of that file over
the total weights.
For example, in this fileset the weights add up to 1000. So the the file500k.html
page will occur 350 out of 1000 times, and the file5m.html will occur once
every 1000 pages.
Note that the URI should be changed to a full URI when testing proxy servers,
for example, if the proxy server is called proxy, but the actual server
which stores the file is called seltzer1, you could use the following filelist:
#Sample filelist, abstracted from access logs
http://seltzer1.sgi.com/file500.html 350 #500
http://seltzer1.sgi.com/file5k.html 500 #5125
http://seltzer1.sgi.com/file50k.html 140 #51250
http://seltzer1.sgi.com/file500k.html 9 #512500
http://seltzer1.sgi.com/file5m.html 1 #5248000
This URI is the one which is passed to the proxy server, which in turn uses
it to fetch the file from seltzer1.sgi.com. Notice that the particular files
and the distribution are identical to the previous filelist. The other change
which would need to be made for testing proxy servers is to have an entry
"PROXY=proxy" in the testbed file and to specify the port where the proxy
server listens for requests.
Wherever possible, use the same pages for WebStone that you will use in
the real world. This means that you'll have a harder time comparing your
results with published results, but your results will more accurately reflect your situation.
The results of each run will be saved in a directory called runs. Note that the runbench script attempts to collect configuration information
about your client and server configurations such as netstat results. You
may see some error messages if your clients don't have netstat or other
The WebStone summary statistics generated by webmaster are saved by runbench in a date stamped subdirectory of the runs directory in the current directory similar to:
The script wscollect is provided as a tool for collected the results of
all of the runs and generating a tab delimited file with all of the results.
This file can be read into a spreadsheet or read by other analysis programs.
wscollect runs > runs.tabs
An additional script called tabs2html will take a tab delimited file and produce an HTML 3.0 style table of the
tabs2html runs.tabs > runs.html
It's fairly common for the Web server under test to run out of swap space.
As a rule of thumb, make sure that you have swap space equal to the number
of server processes times the size of the largest test file.
For instance, if you're testing a 10MB file on a Netscape server with 64
processes, you'll need to have at least 640MB of swap space. N.B.: On SGI IRIX 5.x, you can substitute large amounts of virtual swap space, since Netscape doesn't actually use all the space it asks for.
See your operating system-specific administration guide for details on adding
and configuring swap space.
webmaster -w webmaster -p 9990 -u flist -f config
Waiting for READY from 6 clients
All READYs received
Sending GO to all clients
All clients started at Tue Aug 8 11:57:30 1995
Waiting for clients completion
.Error second reading timing info from one of the clients:
Interrupted system call
web child 1 did not respond. 3456 bytes read
.Error second reading timing info from one of the clients:
Interrupted system call
web child 0 did not respond. 3456 bytes read
What does the second reading timing info contain? What might cause the second
read to fail while the first passes?
It's most likely that one of the WebStone clients died before it could report
results to the webmaster. We've squashed many circumstances in which this
happens, but bugs continue to appear, especially on systems we haven't tested.
We can't do much for this kind of problem without debugging traces. Edit
testbed, and set the
DEBUG parameter to
DEBUG=-d, so that debugging info will be written to files named /tmp/webstone-debug.<PID>.
If you can replicate this problem with debugging turned on, please let us
know. We'd love to examine the traces.
Another possible source of problems with reading timing info is when a page
in the filelist did not get read by a client, but the webmaster was expecting
to find it. This can happen when the test time, number of clients and filelist
distribution are set up so that a file which gets read infrequently does
not get read _yet_ before the test period ends.This will get ironed out
in a later release of WebStone.
WebStone primarily measures throughput (bytes/second) and latency (time
to complete a request). WebStone also reports pages/minute, connection rate
averages, and other numbers. Some of these may help you to sanity-check
the throughput measurements.
Two types of throughput are measured: aggregate and per-client. Both are
averaged over the entire test time and the entire client base. Aggregate
throughput is simply total bytes (body + header) transferred throughout
the test, divided by the total test time. Per-client throughput divides
aggregate throughput by the number of clients.
Two types of latency are reported: connection latency and request latency.
For each metric, the mean time is provided, as well as the standard deviation
of all data, plus the minimum and maximum times. Connection latency reflects
the time taken to establish a connection, while request latency reflects
the time to complete the data transfer once the connection has been established.
User-perceived latency will include the sum of connection and request latencies,
plus any network latency due to WAN connections, routers, modems, etc.
WebStone also reports a metric called Little's Ls. Ls is derived from Little's Law, and reflects how much time is spent by the
server on request processing, rather than overhead and errors. Ls. is also an indirect indicator of the average number of connections which
the web server has open at any particular instant. This number should stay
very close to the number of clients, or else some clients are being denied
access to the server at any given time.
If you load your Web servers high enough, you'll begin to see errors in
the results. That's fine (at least as far as WebStone is concerned). It
just means that your server is heavily loaded, and some clients aren't being
serviced before they time out. In fact, the number of errors at a given
load can be an excellent indicator of how your server will perform under
extremely heavy loads.
Thanks for participating. We'd like to hear from you if there's a way
that we could make this project more useful to you. Send e-mail to
The code listed on this page is provided on an unsupported basis. In
particular, Mindcraft does not verify that the binaries were compiled from
a correct copy of the WebStone source, or that the compilation was done
correctly or with the best choice of performance options.
Try the WebStone mailing list. If you
want custom extensions done to WebStone or products tested,
THIS SOFTWARE IS PROVIDED "AS IS" WITH NO WARRANTIES OF ANY
KIND INCLUDING THE WARRANTIES OF DESIGN, MERCHANTIBILITY AND FITNESS FOR
A PARTICULAR PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE
In no event will Mindcraft, Inc. or Silicon Graphics be liable for any lost revenue or profits
or other special, indirect and consequential damages, even if Mindcraft or Silicon Graphics,
Inc. has been advised of the possibility of such damages.
(c)Copyright 1997-1998, Mindcraft, Inc.