WebSTONE: The First Generation in HTTP Server Benchmarking

February 1995

ABSTRACT

With the advent of the Hyper Text Transfer Protocol (HTTP) it was just a matter of time before the commercial use would be evident. Performance testing of different hardware platforms as well as different implementations of HTTP has made it necessary to create a new benchmark that will allow a customer easily to understand the performance characterizes of different vendors. The WebSTONE, a web serving benchmark has been developed in an attempt to better understand the performance characterics of both hardware and software. The following paper describes the benchmark in technical detail and issues involved in developing this benchmark. This benchmark was developed because there is currently no other way of testing the application. This benchmark is intended for free distribution both this white paper and code. It is the intent for this benchmark to grow and better help test system performance for future systems and HTTP implementations.

Contents

• WebSTONE: The First Generation in HTTP Server Benchmarking
  • ABSTRACT
  • Contents
  • 1. Web Overview
  • 2. WebSTONE overview
    • 2.1 The WebSTONE as a measure of performance
    • 2.2 WebSTONE's measure of server performance
    • 2.3 Metrics the WebSTONE doesn't test
  • 3. WebSTONE Architecture
    • 3.1 WebSTONE Software
  • 4. Configuration Parameters
    • 4.1 Duration of test
    • 4.2 Repetition test
    • 4.3 Number of files
    • 4.4 Number of pages
    • 4.5 All of the server software and hardware configuration
    • 4.6 The number of Webchildren
    • 4.7 Number of networks
    • 4.8 Number of clients
    • 4.9 Workload of pages
    • 4.10 Logging
    • 4.11 Debugging
  • 5. Workload configuration
    • 5.1 General modem mix
    • 5.2 General mix
    • 5.3 Media rich mix
    • 5.4 General and media rich mix
  • 6. Load Generation
    • 6.1 Page selection
    • 6.2 Page Access
    • 6.3 Duration of the test
  • 7. Benchmark Results
  • 8. Future of the WebSTONE and future work
  • 9. Acknowledgments
  • 10. Author information
  • 11. References
  • 12. Trademarks

1. Web Overview

Since the advent of the internet over 20 years ago, there has not been an easy way to access information on the network unless you were proficient with a number of UNIX® commands and understood how the network functioned. But now, thanks to HTTP, the ability to reference information and to travel the net (known as surfing) has been made incredibly simple. A user with a GUI (graphical user interface) can navigate through the internet as easily as they do through their windows based personal computers with the click of mouse button.

HTTP[1] is an application-level protocol with low overhead and the speed necessary for distributed, collaborative, hypermedia information systems. It is a generic, stateless, object-oriented protocol which can be used for many tasks, such as name servers and distributed object management systems, through extensions of its request methods (commands). A feature of HTTP is the typing and negotiation of data representation, allowing systems to be built independently of the data being transferred. The protocol is typically layered on top of TCP/IP in order to guarantee data transfer. Other methods of data transfer maybe used, but, the vast majority of existing systems use TCP/IP for HTTP transfers. The protocol consists of a request and response paradigm. See reference for further information on HTTP.

Once a content or commercial provider has their HTTP server on the Internet a user with the use of an internet browser is able to access the provider with a common interface defined through the Hyper Text Markup Language (HTML), which is a subset of SGML (Standard Generalized Markup Language).

HTML is a simple markup language used to create documents that may be shared between different platforms. The language allows text, data, graphics, news, mail and a number of other utilities to interact with the HTTP server and browser. This gives the user the freedom to enjoy the benefits of the provider without having to know how it is accomplished.

2. WebSTONE overview

The WebSTONE is a new benchmark that tests the performance of HTTP in contrast to server platform's and different implementations of HTTP. Because there are many different types of server software that is currently available, as well as different hardware platforms, there needs to be a mechanism for testing benchmarking server software and hardware platform performance. The WebSTONE is a benchmark that attempts to do this. As with any new benchmark the WebSTONE is a starting point. Once the benchmark is introduced into the general World Wide Web (WWW) community, improvements and enhancements will be made to further the use of this benchmark in support of better end user performance.

Since the functionality of the Web is similar to the functionality of NFS(TM)™, the LADDIS benchmark[2] was reviewed for possible as the web benchmark. Unfortunately, there was no easy way to adapt this benchmark. However, LADDIS did offer a perspective on how to benchmark client/server environment. The WebSTONE is used to measure maximum and average response times for connecting to the server. In addition throughput data is also generated. The WebSTONE is executed simultaneously on one or more clients resident on the server's network. Each client is able to launch a number of children (called Webchildren) depending on how the system load is configured. Each of the Webchildren is able to request information from the server based on a given file load. The WebSTONE is written to be independent of the server platform or software running on it. In essence it treats the server as a blackbox.

2.1 The WebSTONE as a measure of performance

In addition to a benchmark that generates standardized data for comparison of different platforms, the WebSTONE is also a performance tester and maybe used as a tool to help identify performance characterizations of server platforms. It is our goal that the benchmark will evolve and will help define a standard the WWW community may use when comparing data.

Used as a performance tool the WebSTONE, uses workload parameters and clients to generate HTTP traffic that allows an HTTP server to be stressed in a number of different ways. This can gives insight into the server's behavior and performance in a variety of environments.

There four different workloads that represent a sample of the existing servers currently in use on the web.

• The first one is a workload for general use billboards that is sensitive to modem users.
• The second workload is a general mix that is not as concerned with the modem user, but, is still sensitive to download times.
• The third mix is a media rich mix that has very large content.
• the fourth mix is a combination of both the first and third mix.

It should be noted that since the web is still in its infancy it will take time to have well defined mixes that will be considered standard. It also should be noted that there is also the chance that there will be no such thing as a standard mix and each customer will have to define their own mix based on individual sites and run the test against the selected systems.

2.2 WebSTONE's measure of server performance

The WebSTONE is a configurable benchmark that allows performance measurement of the server in the following ways:

• Average and maximum connect time
• Average and maximum response time
• Data throughput rate
• Number of pages retrieved
• Number of files retrieved

The benchmark's goal is to control as much of the clients running environment as possible. The WebSTONE has no interest in the server configuration.

Since the object is to control as much of the client environment as possible we decided to is write the benchmark in C and not to use existing code libraries as they are not written with performance in mind. The benchmark sends HTTP requests to the server and then processes the performance data when done. This insures that only the code necessary to execute the given request is performed in the most direct path.

Though there will always be added overhead in the benchmark this ensures the most control over the accuracy of the benchmark. With this method the layer of existing libraries is removed and only what is needed to measure the pure performance of the server is used.

2.3 Metrics the WebSTONE doesn't test

Since this benchmark is concerned only with the server software and hardware, the WebSTONE does not test the browser or client side applications or libraries. Though in the future it might be of interest to add this to the benchmark. Please see future additions to the benchmark in chapter 8, later on in this paper for other metrics that this benchmark currently does not test but will need to in the near future.

3. WebSTONE Architecture

The WebSTONE architecture is shown in figure below. The Webchildren are controlled by the WebMASTER. The WebMASTER controls the operation of the benchmark run. As shown in figure below, there are 4 Webchildren connected to the HTTP server over a dedicated LAN (LAN 1). The WebMASTER is executed on a separate system outside of the Webchildren. The WebMASTER can be run on one of the clients or on a separate machine. The benchmark currently is configured to run the WebMASTER on a machine that is not on the same network as the clients and the server.

The ability to have the WebMASTER on a different system gives the flexibility to have different networks talking to the same server and to have different client configurations.

3.1 WebSTONE Software

The WebSTONE is a distributed, multi-process benchmark. The master process (WebMASTER) reads the client configuration files as well as the command line. The WebMASTER then constructs a command line for each Webchildren. The WebMASTER then remotely spawns the Webchildren. Each of the Webchildren then reads the command line and startup communication with the WebMASTER. After all the Webchildren have been initialized the WebMASTER instructs the Webchildren to commence the benchmark.

As each of the Webchildren finishes its run the WebMASTER collects the data from each client and coalesces the data into a report. During the run the Webchildren are autonomous of each other and the WebMASTER.

The figure below shows an example of a typical client running 3 Webchildren. The Webchildren are spawned by the WebMASTER. Each Webchild then reads its configuration files and opens log files if part of the test.

4. Configuration Parameters

Since the WebSTONE can has many variables this gives the flexibility to configure and run many different suites. The standard mix of files will give a general performance indicator. The parameters that are configurable are listed below:

• Duration of test
• Repetition of test
• Number of files
• Number of pages
• Server software and hardware configuration
• Number of Webchildren
• Number of networks
• Number of clients
• Workload of pages
• Logging
• Debugging

4.1 Duration of test

The WebSTONE is designed to run for a specified duration. A given test is run in units of minutes. The maximum running time is dependent on your client memory and the number of Webchildren to be spawn.

4.2 Repetition test

The WebSTONE has the ability to run for a number of iterations. This is basically a loop counter. For example: If the test consists of a set of 4 files and the test to be run was to request each file 1 time and then report back the status of that run the benchmark would then attempt to retrieve each file one time and then generate a report based on this data.

4.3 Number of files

There are two different ways to read files or UIL's into the benchmark. The configuration file "filelist" contains a list of pages and files that are already preconfigured. There is a limit of 50 files per page in the current implementation of this benchmark.

The other way to read files into the run is at the command line. The user may list the files to be tested on the command line. This is limited by the number of arguments allowed on the command line.

4.4 Number of pages

At this point the concept of a page needs to be introduced. Since the concept of a HTML document is one that has text with inline images like GIF and JPEG in it, there is a conceptual view that a page is the HTML text plus all the GIF and JPEG files that are associated with it that make to whole document. This approach was taken to mimic as closely to a real life environment usage patters where GIF and JPEG inlined images are automatically down load upon retrieval of the HTML text page. Therefore from this point on a page will be considered the set of files it takes to create an HTML viewable document. Note: Though in real life an applications program would lite off all the request for the files at one time, it was decided that this was not an issue since multiple children run on each client and that the acquired affect of simultaneously connections is stilled achieved.

The WebSTONE is designed to use the concept of pages during testing. The WebSTONE is able to handle up to 100 pages with 50 files per page.

4.5 All of the server software and hardware configuration

Though the WebSTONE is design to test different server software and hardware configurations the benchmark in itself does not discriminate as to configuration. This allows the tester to try different configurations in order to achieve optimal performance for the server software and hardware. NOTE: In fairness the tests should be run with the same hardware and software as reported in the test results. All hardware and sever software configurations should be disclosed when a test is released.

4.6 The number of Webchildren

On each client host the number of Webchildren requesting pages or files from the server is configurable. Note that as the number of Webchildren increase less memory is available on the client hosts for running tests and there could be a performance penalty depending on the speed of the clients running on the client host.

4.7 Number of networks

It is possible to have more then one network connected to the server and to have clients on different networks running the benchmark. Though this was not done in this version of the benchmark there should be no reason that it would not work provided that the server host name is the same on all the clients.

4.8 Number of clients

The WebSTONE supports the ability to have as many children on each client as long as there is enough memory on both the WebMASTER and client systems. Note there maybe an issue of performance bottlenecks on the client side if there are too many Webchildren or the client is slow.

4.9 Workload of pages

Each page in the WebSTONE has a weight associated with it. A page may be from 1 to 100% of the test. See the filelist for further information on configuration. Weighting is used to simulate that activity of a given page. The higher the percentage the more often the page is hit. The lower the percentage the less often the page will be hit.

4.10 Logging

Logging is added to the benchmark, but caution should be exercised when using this option as each Webchildren logs every connection and relevant data in a separate file. The logging information contains additional information that is not returned to the WebMASTER.

4.11 Debugging

Debugging is added to help debugged if there is a problem with the server or client. In debugging mode the HTTP header is display for each request that is sent.

5. Workload configuration

One of the goals for the WebSTONE is to model a real world workload via a synthetic workloads based on data gathered from different sites - Hotwired (http://www.hotwired.com), IUMA (http://www.iuma.com), Netscape Communications (http://www.netscape.com), and of course Silicon Graphics' Silicon Surf(SM) (http://www.sgi.com). Unfortunately this is small data set to pull from but, it has been found that it does currently represent the general use of HTTP. In this benchmark there are 4 different mixes from the data gather from those sites.

5.1 General modem mix

The General modem mix is a synthetic page mix that would be used if modem users where to be considered. Two concerns to a potential server site should be the size of the pages that are on their system and type of access to the server. If a end user is accessing the server with a 14.4kbs modem then it would be in the best interest of the site to have small pages as to not take a long time to download the data. This mix takes the end user that has to use a modem as their connection into account. This mix will consist of small pages that are less then 20k bytes and mainly text with sparse graphics files.

5.2 General mix

The General mix is one that is not concerned with modem users however, it is still concerned with the network responsiveness and throughput. File sizes in this mix will be between 1 and 100k bytes.

5.3 Media rich mix

Media rich is defined by the type and size of data stored. Media referees to multi-media content such as: MPEG and Quicktime movie files, MPEG and aiff ..etc. sound clips, and large graphics files. Media rich content sites are not worried about the size of their files. These files usually consist of movie clips and sound files. This mix is used to cover this need. These files range in size from 20KB to megabytes.

5.4 General and media rich mix

To cover the combination of a site that wishes to server both small content and Media rich content this mix was created. This mix will most suite this need.

6. Load Generation

The current load generation of the WebSTONE is to request pages and files from the server as fast as the server can send them. Reflecting the current environment in the WWW community. To generate a load there are four things that are needed.

• A number of Webchildren. The more Webchildren that are requesting pages from the server the more load that this will cause.
• The type of page. This is mainly determined by the page size and the weight of the page.
• Number of clients. The more clients that are in the test the more Webchildren that are able to be brought to bear on the server.
• The number of pages. The more pages that are requested test other aspects of server.

6.1 Page selection

Each page in the mix has a percentage associated with it. This percentage is the weighting factor. The higher the number the more frequent the page will be hit. A random number is generated and then compared to the page weight. If the random number matches the page weight then that pages' files are retrieved one at a time. Each Webchildren has its own random number sequence.

6.2 Page Access

After a page is selected by random weight then each Webchild contacts the HTTP server and requests the first file of the page. After the Webchild receives the first page it requests the next one until all the files for that page have been requested. On each page the time it takes to connect and down load the file is recorded and log if logging is turned on.

6.3 Duration of the test

The WebSTONE is designed to run until the clients run out of memory or the loop counter hits 20000. The loop counter is the ability to run a test in repetition as to time. This test is limited because of client memory. If client memory is not an issue this number can be changed and the benchmark recompiled. Most tests do not last longer than 1 hour..

7. Benchmark Results

The following is a typical run of the benchmark. The first run is of the benchmark against the Netsite server followed by the NCSA server.

The following parameters were modified to run the test:

nm_clusters 
somaxconn = 50
nm_clusters = 4000
tcp_keepidle = (1200)
tcp_keep_timer_in_close = 1

Hardware platform:

1 150 MHZ IP22 Processor
FPU: MIPS R4010 Floating Point Chip Revision: 0.0
CPU: MIPS R4400 Processor Chip Revision: 5.0
On-board serial ports: 2
On-board bi-directional parallel port
Data cache size: 16 Kbytes
Instruction cache size: 16 Kbytes
Secondary unified instruction/data cache size: 1 Mbyte
Main memory size: 256 Mbytes
Integral ISDN: Basic Rate Interface unit 0, revision 1.0
XPI FDDI controller: xpi0, firmware version 9411032038, SAS
Integral Ethernet: ec3, version 1
Integral Ethernet: ec0, version 1
Integral SCSI controller 5: Version WD33C95A, differential, revision 0
Disk drive: unit 4 on SCSI controller 5
Disk drive: unit 3 on SCSI controller 5
Disk drive: unit 2 on SCSI controller 5
Disk drive: unit 1 on SCSI controller 5
Integral SCSI controller 4: Version WD33C95A, differential, revision 0
Integral SCSI controller 0: Version WD33C93B, revision D
Disk drive: unit 5 on SCSI controller 0
CDROM: unit 4 on SCSI controller 0
Disk drive: unit 2 on SCSI controller 0

The file set used for this test is listed below. In this case a general and media rich mix was used.

#This file is used to configure the pages and files to be tested for.
6
40 3
/file3k.html
/file4k.html
/file5k.html
5 3
/file10k.html
/file17k.html
/file20k.html
5 2
/file5m.html
/file1m.html
20 3
/file6k.html
/file7k.html
/file200k.html
20 2
/file3m.html
/file21k.html
10 2
/file500k.html
/file13k.html

What follows is the results of a test ran for 10 minutes with the above page sets. The first set of data is from the Netsite server.

/usr/local/bin/webstone -w xpi0-alfalfa -c sulu:8636 -u filelist -t 10 -n %d
Client: gateweb-indy8 Number of Clients: 6
Client: gateweb-indy9 Number of Clients: 6
Client: gateweb-indy10 Number of Clients: 6
Client: gateweb-indy11 Number of Clients: 6

Waiting for READY from 24 clients
All READYs received
Sending GO to all clients
All clients started at Tue Mar 7 01:33:16 1995
Waiting for clients completion
All clients ended at Tue Mar 7 01:43:41 1995
Page # 0
================================================================================
Total number of times page was hit 1888
Total time 1262.655230 seconds 
Maximum Response time 1.758487 
Total connect time for page 21.293520
Maximum time to connect 0.038234 
Total amount of data moved 23199744
Page size 12288 
Total number of connects 5664
================================================================================

Page # 1
================================================================================
Total number of times page was hit 200
Total time 189.543349 seconds 
Maximum Response time 1.750272 
Total connect time for page 2.550292
Maximum time to connect 0.028744 
Total amount of data moved 9625600
Page size 48128 
Total number of connects 600
================================================================================

Page # 2
================================================================================
Total number of times page was hit 191
Total time 3188.731168 seconds 
Maximum Response time 21.040139 
Total connect time for page 1.605905
Maximum time to connect 0.018310 
Total amount of data moved 1201668096
Page size 6291456 
Total number of connects 382
================================================================================

Page # 3
================================================================================
Total number of times page was hit 870
Total time 816.308261 seconds 
Maximum Response time 2.034608 
Total connect time for page 9.977394
Maximum time to connect 0.033259 
Total amount of data moved 189757440
Page size 218112 
Total number of connects 2610
================================================================================

Page # 4
================================================================================
Total number of times page was hit 918
Total time 7808.708139 seconds 
Maximum Response time 11.413719 
Total connect time for page 7.645089
Maximum time to connect 0.037124 
Total amount of data moved 1387448320
Page size 3167232 
Total number of connects 1836
================================================================================

Page # 5
================================================================================
Total number of times page was hit 500
Total time 990.735685 seconds 
Maximum Response time 4.075929 
Total connect time for page 4.365119
Maximum time to connect 0.040544 
Total amount of data moved 262656000
Page size 525312 
Total number of connects 1000
================================================================================

==================================
WEBSTONE number: 456
Total number of clients: 24
Total cumulative time of test for all hosts (sec): 14298.928119
Total number of pages retrieved from server: 4567
Total number of errors to server: 0
Total number of connects to server: 12099
Average time per connect: 0.003923 seconds
Maximum time to connect: 0.037051 seconds
Total mount of data moved: 3076611677
Total bytes of body moved: 3074355200 bytes.Total bytes of header moved 2306477
Average body size: 26052 bytes. Average retrieval size 26242
Thruput = 215181 bytes/sec
Average Response time: 1.181827 seconds
Maximum Response time: 18.488714 seconds

**
The results for the NCSA server are listed below:
/usr/local/bin/webstone -w xpi0-alfalfa -c sulu:8626 -p 1081 -u filelist -t 10 -n %d
Client: gateweb-indy8 Number of Clients: 6
Client: gateweb-indy9 Number of Clients: 6
Client: gateweb-indy10 Number of Clients: 6
Client: gateweb-indy11 Number of Clients: 6

Waiting for READY from 24 clients
All READYs received
Sending GO to all clients
All clients started at Tue Mar 7 01:17:42 1995
Waiting for clients completion
All clients ended at Tue Mar 7 01:28:06 1995
Page # 0
================================================================================
Total number of times page was hit 597
Total time 2852.766161 seconds 
Maximum Response time 31.364861 
Total connect time for page 215.503592
Maximum time to connect 29.778926 
Total amount of data moved 7335936
Page size 12288 
Total number of connects 1791
================================================================================

Page # 1
================================================================================
Total number of times page was hit 129
Total time 650.500231 seconds 
Maximum Response time 10.067770 
Total connect time for page 23.714814
Maximum time to connect 5.931920 
Total amount of data moved 6208512
Page size 48128 
Total number of connects 387
================================================================================

Page # 2
================================================================================
Total number of times page was hit 72
Total time 3050.927370 seconds 
Maximum Response time 53.652579 
Total connect time for page 5.801699
Maximum time to connect 5.563671 
Total amount of data moved 452984832
Page size 6291456 
Total number of connects 144
================================================================================

Page # 3
================================================================================
Total number of times page was hit 328
Total time 2131.366331 seconds 
Maximum Response time 12.596820 
Total connect time for page 42.219253
Maximum time to connect 5.932938 
Total amount of data moved 71540736
Page size 218112 
Total number of connects 984
================================================================================

Page # 4
================================================================================
Total number of times page was hit 221
Total time 5043.018456 seconds 
Maximum Response time 29.971733 
Total connect time for page 17.345611
Maximum time to connect 5.594679 
Total amount of data moved 699958272
Page size 3167232 
Total number of connects 442
================================================================================

Page # 5
================================================================================
Total number of times page was hit 75
Total time 498.577637 seconds 
Maximum Response time 33.667678 
Total connect time for page 58.567395
Maximum time to connect 29.853669 
Total amount of data moved 39398400
Page size 525312 
Total number of connects 150
================================================================================

==================================
WEBSTONE number: 142
Total number of clients: 24
Total cumulative time of test for all hosts (sec): 14289.247361
Total number of pages retrieved from server: 1422
Total number of errors to server: 0
Total number of connects to server: 3920
Average time per connect: 0.092649 seconds
Maximum time to connect: 29.851746 seconds
Total mount of data moved: 1278775218
Total bytes of body moved: 1278055424 bytes.Total bytes of header moved 719794
Average body size: 326034 bytes. Average retrieval size 326218
Thruput = 89492 bytes/sec
Average Response time: 3.645216 seconds
Maximum Response time: 43.999927 seconds

8. Future of the WebSTONE and future work

As this is the first version of the benchmark it is to be considered a living benchmark that will continue to grow and improve.

In the future there are a number of things that need to be added to the benchmark that this author at this time did not have time to add. The following is but a small list.

• Effects of the CGI (Common Gate Interface)
• Security (encryption and authentication)
• HTML parsing
• RDBMS performance

These and others are of importance in the near future.

9. Acknowledgments

There are a number of people that have helped with this benchmark over the past 2 months. First and foremost Mark Sake who helped code and developed/architected the first version of the benchmark. Steffen Low for support and belief that this should happen. To Helena Parentay for all her help. I would also like to specially thank David Ciemiewicz for contributions to this paper.

Additional thanks to Neal Nucklus and Bill Nowicki for critical input to this paper and the benchmark.

10. Author information

Gene Trent is a Member of Technical staff in the Advance Data Division of Silicon Graphics. Mr. Trent is the principal Software Engineer that developed the WebSTONE benchmark and author of this white paper. Gene holds a bachelors degree in Electronic Engineer technology from Devry Institute of Technology. He can be reached at et@sgi.com or at Silicon Graphics 2011 N. Mountain View, Ca

Mark Sake is a Member of Technical staff in the Advance Data Division of Silicon Graphics. Mr. Sake co-authored the WebSTONE benchmark.

11. References

[1] Berners-Lee, Fielding, and Frystyk Nielsen.

"Hypertext Transfer Protocol - HTTP/1.0"

Network Working Group INTERNET_DRAFT

<draft-fielding-http-spec-01.ps>

<URL:http://www.ics.uci.edu/pub/ietf/http/> December 19, 1994

[2] Mark Wittle, Bruce E. Keith

"LADDIS: The Next Generation In NFS File Server Benchmarking"

USENIX Association Conference Proceeding, 1993

12. Trademarks

NFS is a trademark of Sun Microsystems, Inc.

UNIX is a registered trademark of UNIX Systems Laboratories.

Silicon Graphics is a registered trademark of Silicon Graphics, Inc.

Silicon Surf is a service mark of Silicon Graphics, Inc.