Executive Summary

March 10, 1997

This report compares the performance of Netscape Enterprise Server 2.0.1 and Lotus Domino Web Server 1.5a serving HTML text. All tests were done on a single-processor Hewlett-Packard NetServer 5/133 LS2 running Microsoft Windows NT 4.0.

Three sets of tests were run: Netscape Enterprise Server serving HTML files, Lotus Domino serving the same files, and Lotus Domino serving from a Notes database containing the HTML files.

The WebStone 2.5 benchmark was used to test the system. All three configurations were tested over the range of 4 to 24 WebStone client processes. We restricted the testing to small loads because at higher loads the Domino server experienced very high error rates and lower performance (see data summary). In addition, the Netscape server was tested with loads from 30 to 240 WebStone client processes.

Figure 1 shows the peak connection rates measured in the range from 4 to 24 client processes. Figure 2 shows the corresponding data throughput rates. A table summarizing the peak performance follows Figure 2.

Introduction

This Performance Report was commissioned by Netscape Communications Corporation to compare the performance of their Enterprise Server 2.0.1 to the performance of Lotus Domino Web Server 1.5a. The WebStone 2.5 run rules were followed, with two exceptions: the fileset used was the Silicon Surf fileset from WebStone 1.1, and the range of loads used was from 4 to 24 WebStone client processes.

The Silicon Surf fileset was chosen because the standard WebStone 2.5 fileset includes one data file that is five megabytes in size, and we were not confident that we could load this file into the Notes database properly. The fileset was translated from the format used by WebStone 1.1 to that used by WebStone 2.0 and WebStone 2.5. The narrower range of loads was used because even at loads of 20 and 24 WebStone client processes tests of the Domino server gave error rates of 61 to 442 errors per second. Other servers we have tested give error rates of 0.0 to 0.1 error per second at the highest attainable load. See the data summary for the fileset used and the data summary tables.

The NetScape Enterprise Server was also tested at loads ranging up to 240 WebStone client processes.

Mindcraft's Certification

Mindcraft, Inc. conducted the performance tests described in this report between March 3 and March 7, 1997, in the Netscape performance laboratory in Mountain View, California. Mindcraft used the WebStone 2.5 benchmark to measure performance.

Mindcraft certifies that the results reported herein fairly represent the performance of the Netscape Enterprise Server 2.0.1 and of the Lotus Domino Web Server 1.5a running on a Hewlett-Packard NetServer 5/133 LS2 under Microsoft's Windows NT Server 4.0 operating system as measured by the WebStone 2.5 benchmark. Our test results should be reproducible by others who use the same test lab configuration as well as the computer and software configurations and modifications documented in this report.

Performance Analysis

This analysis is based on the complete WebStone benchmark results for the tests described here.

The WebStone 2.5 benchmark stresses a system's networking ability in addition to other aspects of server performance. The best way to see if there is unused capacity on a server computer running WebStone is to look at the CPU utilization. It was 100% for all of the tests when the peak performance was attained. The maximum network bandwidth used was 33.5 Mbits/second, for the Netscape server with 240 WebStone client processes. This is well below the maximum expected available bandwidth of a 100Base-TX Ethernet, and below the aggregate performance of the seven 10Base-T Ethernet connections to the client systems. Therefore, the networks did not significantly limit the performance of the Web servers as measured in these tests.

Figures 3 and 4 show the network throughput and connection rates, respectively, of the Lotus and Netscape servers running on a Hewlett Packard Web Server LS2. The performance curves are nearly flat for all three configurations over the range 8 through 24 clients. Above this load level the performance of the Lotus Domino server degrades rapidly, with increased latency (Figure 5) and greatly elevated error rates. The performance of the Netscape Enterprise Server increases slightly with increasing demand, up to the maximum test load of 240 WebStone client processes.

The WebStone load that a server computer can support depends on four primary factors:

• The bandwidth of the networks available;
• The ability of the operating system to utilize the available network bandwidth;
• The ability of the operating system to maximize the CPU time available to the Web server; and
• The rate at which the Web server can process requests

For all the tests presented here the network throughput was well below the maximum capacity of the network hardware. The server platform was the same for all the test runs. Each server used all available CPU cycles when peak performance was measured. This leaves only Web server efficiency as the determining factor of the observed performance.

Test Procedures and Benchmark Configuration

Test Procedures

Mindcraft followed the standard WebStone 2.5 run rules with the following changes:

• Instead of the standard run from 10 to 100 clients, we ran all three configurations from 4 to 24 clients. In addition, we ran tests for 30 to 240 clients in order to more fully demonstrate the capacity of the Netscape server.
• Access logging was turned off, so the results would not be skewed by the overhead to record the very long URLs for the data in the the Notes database.

The following basic set of procedures was used for performing these tests:

• Error logging was turned on.
• All test runs were done with a user logged in to the server console.

Test Data

The data files used for the static HTML testing were the default "Silicon Surf" fileset, described in filelist.ss as distributed with the WebStone 1.1 benchmark. The fileset was translated from the format used by WebStone 1.1 to that used by WebStone 2.0 and WebStone 2.5. See the data summary for the translated fileset file.

This fileset was designed to represent a real-world server load, based on analysis of the access logs of Silicon Graphics, Inc.'s external Web site, http://www.sgi.com. Netscape's analysis of logs from other commercial sites indicated that Silicon Surf access patterns were fairly typical for the Web when they were designed.

The Silicon Surf model targets the following characteristics:

• 93% of accessed files are smaller than 30 KB.
• Average accessed file is roughly 7 KB.

Configuration of the System Tested

Test Lab Configuration

In order to cause the Netscape Enterprise Server to use all available CPU cycles on the server computer system, we used seven WebStone client systems. One of these systems also served as the Webmaster, controlling the WebStone driver. The test lab network configuration used for this work is shown below:

The tests described in this report were performed on an isolated LAN that was quiescent except for the test traffic.

Glossary

Clients

Number of processes or threads simultaneously requesting Web services from the server.

Connections per second

Average rate of creation and destruction of client/server connections.

Errors per second

Error rate for this run. Errors detected include:

-- socket() or connect() failure

-- error sending a request to the server

-- read() failure while receiving data from the server

-- malformed response or error code from server

Latency

Average client wait for data to be returned.

Throughput

Average data transfer rate, in megabits per second.