|

WebStone 2.5 Run Rules
Draft 1
March 7, 1997
The purpose of these run rules is to provide conformity in WebStone
performance reports. When running WebStone 2.5 and reporting the results these rules
should be followed:
- File lists used: The HTML file list used for the test must be the
filelist-100h0c0a
included in the distribution. Tests of CGI performance must use the file list filelist-0h100c0a.
Tests of API performance, whether using NSAPI, ISAPI, or any other API must be carried out
with filelist-0h0c100a.
Mixed HTML, CGI, and API tests must be done with a file list that specifies the weights of
the actual mix used; the total of the weights must be less than 32768. An example of such
a file list is filelist-85h15c0a included in the WebStone 2.5 distribution. When
reporting the results of mixed tests, the percentages of access for each type (HTML, CGI,
and API) must be reported along with the files sizes used. It is also necessary to report
which API access method was used.
- Test time: The run time per data point must be set for 10
minutes.
- Number of Web clients: The number of Web clients should vary from
20 to 100 Web clients in increments of 10. When it is not possible to test with this
number of Web clients, the tests must be done with an evenly spaced number of Web clients
and the number used and the reason for using a different number must be reported.
If 100 Web clients are not enough to load the system under test to 100% CPU utilization,
the testing must still be done with 20 to 100 Web clients plus enough extra Web clients to
load the system under test to the maximum CPU utilization possible. The testing with the
extra Web clients must be done in evenly spaced steps above 100. For example, if 300 Web
clients are needed to load a system, tests need to be done with 200 and 300 Web clients.
- Reporting results: Reports must include the following
information:
- Server computer under test: including the number and type of
CPUs, the amount of memory, the type of disk controller, the number of disks, which disks
hold the operating system and Web files, the number and type of network interface cards
used, the operating system version used including patches, configuration and tuning
parameters, and any TCP/IP tuning done;
- Client computers used: The number and type of computers used, the
number and type of CPUs, the amount of memory, the type of disk controller, the number of
disks, the number and type of network interface cards used, the operating system version
used including patches, configuration and tuning parameters, and any TCP/IP tuning done;
- Web server: The name of the Web server under test, the version
tested including patches, configuration and tuning parameters such as DNS lookups, and the
number of threads and cache size;
- Web clients: The total number used, how they are divided among
the client computers, and the reason for using any non-standard number of Web clients.
Warning
Do not compare WebStone 2.5 results with WebStone 1.1 results. WebStone
2.5 introduces significant changes in testing methodology and reporting.
Legal Small Print
The WebStone 2.5 run rules are copyright 1997, Mindcraft, Inc.
These rules are provided without support and without any obligation on the
part of Mindcraft, Inc. or Silicon Graphics, Inc. to assist in its use, correction,
modification
or enhancement.
THESE RUN RULES ARE PROVIDED "AS IS" WITH NO WARRANTIES OF ANY KIND INCLUDING
THE
WARRANTIES OF DESIGN, MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE,
OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.
In no event will Mindcraft, Inc. or Silicon Graphics, Inc. be liable for any lost revenue
or
profits or other special, indirect and consequential damages, even if
Mindcraft, Inc. or Silicon Graphics, Inc. has been advised of the possibility of such
damages.
|