WebStone 2.5 Run Rules

Draft 1
March 7, 1997

The purpose of these run rules is to provide conformity in WebStone performance reports. When running WebStone 2.5 and reporting the results these rules should be followed:

1. File lists used: The HTML file list used for the test must be the filelist-100h0c0a
   included in the distribution. Tests of CGI performance must use the file list filelist-0h100c0a. Tests of API performance, whether using NSAPI, ISAPI, or any other API must be carried out with filelist-0h0c100a.
   
   Mixed HTML, CGI, and API tests must be done with a file list that specifies the weights of the actual mix used; the total of the weights must be less than 32768. An example of such a file list is filelist-85h15c0a included in the WebStone 2.5 distribution. When reporting the results of mixed tests, the percentages of access for each type (HTML, CGI, and API) must be reported along with the files sizes used. It is also necessary to report which API access method was used.
   
   
2. Test time: The run time per data point must be set for 10 minutes.
   
   
3. Number of Web clients: The number of Web clients should vary from 20 to 100 Web clients in increments of 10. When it is not possible to test with this number of Web clients, the tests must be done with an evenly spaced number of Web clients and the number used and the reason for using a different number must be reported.
   
   If 100 Web clients are not enough to load the system under test to 100% CPU utilization, the testing must still be done with 20 to 100 Web clients plus enough extra Web clients to load the system under test to the maximum CPU utilization possible. The testing with the extra Web clients must be done in evenly spaced steps above 100. For example, if 300 Web clients are needed to load a system, tests need to be done with 200 and 300 Web clients.
   
   
4. Reporting results: Reports must include the following information:
   • Server computer under test: including the number and type of CPUs, the amount of memory, the type of disk controller, the number of disks, which disks hold the operating system and Web files, the number and type of network interface cards used, the operating system version used including patches, configuration and tuning parameters, and any TCP/IP tuning done;
   • Client computers used: The number and type of computers used, the number and type of CPUs, the amount of memory, the type of disk controller, the number of disks, the number and type of network interface cards used, the operating system version used including patches, configuration and tuning parameters, and any TCP/IP tuning done;
   • Web server: The name of the Web server under test, the version tested including patches, configuration and tuning parameters such as DNS lookups, and the number of threads and cache size;
   • Web clients: The total number used, how they are divided among the client computers, and the reason for using any non-standard number of Web clients.

Warning

Do not compare WebStone 2.5 results with WebStone 1.1 results. WebStone 2.5 introduces significant changes in testing methodology and reporting.

Legal Small Print

The WebStone 2.5 run rules are copyright 1997, Mindcraft, Inc.

These rules are provided without support and without any obligation on the part of Mindcraft, Inc. or Silicon Graphics, Inc. to assist in its use, correction, modification or enhancement.

THESE RUN RULES ARE PROVIDED "AS IS" WITH NO WARRANTIES OF ANY KIND INCLUDING THE WARRANTIES OF DESIGN, MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.

In no event will Mindcraft, Inc. or Silicon Graphics, Inc. be liable for any lost revenue or profits or other special, indirect and consequential damages, even if Mindcraft, Inc. or Silicon Graphics, Inc. has been advised of the possibility of such damages.