Mindcraft Certified Performance Comparison Report

File Server Comparison:

Microsoft Windows NT Server 4.0
and
Novell IntranetWare 4.11
(PDF version 240 KB)

Contents

Executive Summary
Performance Analysis
What Are the Bottlenecks?
Price/Performance
Products Tested
Test Lab
Mindcraft Certification
NetBench Configuration and Results

Executive Summary

Microsoft Windows NT Server 4.0 Is 13.5% Faster Than Novell IntranetWare 4.11 and Has 2.6 Times Better Price/Performance

Mindcraft tested the file-server performance of Microsoft Windows NT Server 4.0 and Novell IntranetWare 4.11 on a Compaq ProLiant 5500. We tested file sharing using TCP/IP for both servers. Table 1 shows the peak throughput measured for each system in megabits per second (Mbits/S), the price of the software tested, and the price/performance in dollars per Mbits/S.

Table 1: Summary
(larger numbers are better for Throughput, smaller numbers are better for Software Price and Price/Performance)


File Server

Peak Throughput

Software Price


Price/Performance

Windows NT Server 4.0
One-Processor Compaq ProLiant 5500

95.2 Mbits/S $2,659 $27.93/Mbits/S

IntranetWare 4.11
One-Processor ProLiant 5500

83.9 Mbits/S $6,100 $72.71/Mbits/S

Windows NT Server 4.0
Two-Processor ProLiant 5500

121.5 Mbits/S $2,659 $21.89/Mbits/S

IntranetWare 4.11
Two-Processor ProLiant 5500

114.3 Mbits/S $6,100 $53.37/Mbits/S

Mindcraft tested these file servers with the Ziff-Davis Benchmark Operation NetBench 5.01 benchmark. The Performance Analysis section gives the detailed configuration of the NetBench test suite we used. The price/performance calculations are described in the Price/Performance section.

For a one-processor system at peak file server performance, Windows NT Server 4.0 is 13.5% faster than IntranetWare 4.11 and its price/performance is 2.6 times better. For a two-processor system, Windows NT Server 4.0 is 6.4% faster than IntranetWare 4.11 at peak performance. Its price/performance is 2.4 times better. In addition, on a two-processor system when the maximum load is applied, Windows NT Server 4.0 delivers 14% more throughput than IntranetWare 4.11.

Windows NT Server 4.0 is a more cost-effective and higher performance file server than Novell IntranetWare 4.11.

Performance Analysis

Looking at the Results

The NetBench 5.01 benchmark measures file server performance. Its primary performance metric is throughput in bytes per second. The NetBench documentation defines throughput as "The number of bytes a client transferred to and from the server each second. NetBench measures throughput by dividing the number of bytes moved by the amount of time it took to move them. NetBench reports throughput as bytes per second." We report throughput in megabytes per second to make the charts easier to read.

We tested file-sharing performance on Windows NT Server 4.0 and IntranetWare 4.11 using TCP/IP as the underlying protocol for both systems. Figure 1 shows the throughput we measured plotted against the number of test systems that participated in each data point.

Figure 1: NetBench Throughput Performance (larger numbers are better)

Throughput Graph (8945 bytes)

How NetBench 5.01 Works

You need to know how NetBench 5.01 works in order to understand what the NetBench throughput measurement means. NetBench is designed to stress a file server by using a number of test systems to read and write files on it. Specifically, a NetBench test suite is made up of a number of mixes. A mix is a particular configuration of NetBench parameters, including the number of test systems used to load the server. Typically, each mix increases the load on the server by increasing the number of test systems involved while keeping the rest of the parameters the same. We modified the standard NetBench NBDM_60.TST test suite in order to test each product to its maximum performance level and to make the test run in a reasonable amount of time. The parameters we used are shown in NetBench Test Suite Configuration Parameters.

NetBench does a good job of testing a file server under heavy load. To do this, each NetBench test system (called a client in the NetBench documentation) executes a script that specifies a file access pattern. As the number of test systems is increased, the load on a server is increased. You need to be careful, however, not to correlate the number of NetBench test systems participating in a test mix with the number of simultaneous users that a file server can support. This is because each NetBench test system represents more of a load than a single user would generate. NetBench was designed to behave this way in order to do benchmarking with as few test systems as possible while still generating large enough loads on a server to saturate it.

When comparing NetBench results, be sure to look at the configurations of the test systems because they have a significant effect on the measurements that NetBench makes. For example, the test system operating system may cache some or all of the workspace in its own RAM causing the NetBench test program not to go over the network to the file server as frequently as expected. This can significantly increase the reported throughput. In some cases, we’ve seen reported results that are 75% above the available network bandwidth. If the same test systems and network components are used to test multiple servers with the same test suite configuration, you can make a fair comparison of the servers.

With this background, let us look at the results in Figure 1 (the supporting details for this chart are in NetBench Configuration and Results).

The three major areas to look at are:

Peak Performance

This tells you the maximum throughput you can expect from a file server.

  • For the one-processor configuration, Windows NT Server peak performance was 95.2 Mbits/second and IntranetWare reached a peak of 83.9 Mbits/second. This means that Windows NT Server is 13.5% faster than IntranetWare.
  • For the two-processor configuration, Windows NT Server performance peaked at 121.5 Mbits/second while IntranetWare attained a maximum throughput of 114.3 Mbits/second. So Windows NT Server is 6.4% faster than IntranetWare at peak performance.

Where Peak Performance Occurs

You can tell how quickly a server reaches its saturation point by looking at where the peak performance occurs in relation to the number of test systems.

  • For the one-processor configuration, Windows NT Server reached its peak at 56 test systems while IntranetWare reached its peak at 32 test systems. This means that Windows NT Server is able to serve more users well after IntranetWare has reached its peak.
  • For the two-processor configuration, both operating systems reached peak performance at 56 test systems.

Shape of the Performance Curve

  • For the one-processor configuration, below a load of 32 test systems both operating systems follow essentially the same performance curve. At 32 test systems, IntranetWare abruptly reaches its peak and performance slowly degrades after that. The abruptness of the change from increasing performance with load to decreasing performance means that an IntranetWare system administrator may have little warning before the server is saturated. Windows NT Server performance gradually increases from 32 test systems to its peak at 56 test systems. The gradual increase in performance as Windows NT Server approaches its peak and the gradual decline in performance after that means that a system administrator can see the server becoming saturated and take appropriate steps to make more file server capacity available.
  • For the two-processor configuration, IntranetWare performance grows smoothly to its peak at 56 test systems and then declines at faster rate. So as the server becomes saturated, user productivity will decline as they wait longer to access the files they need. Windows NT Server performance increases smoothly along essentially the same curve as IntranetWare up to 48 test systems and keeps on growing to its peak at 56 test systems. Windows NT Server continues to perform within 1.8% of its peak out to 72 test systems. This means that user productivity on Windows NT Server under heavy loads will decrease far more gradually than with IntranetWare.

What Are the Bottlenecks?

We normally include an analysis of internal server performance measurements in our reports to find what bottlenecks limited the overall performance of the products tested. We gather these statistics during the tests by using monitoring programs included in the operating system. Windows NT Server provides a standard performance-monitoring tool called perfmon. It allows you to select which performance statistics you want to monitor and lets you see them in a real-time chart as well as save them in a log file for later analysis. Typically we log the processor, memory, network interface, and disk subsystem performance counters.

IntranetWare does not provide performance-monitoring tools that could be used to log server statistics. Because we could not obtain comparable internal server performance measurements for both operating systems, we will not offer any bottleneck analysis.

Conclusion

Windows NT Server 4.0 offers high-performance file sharing on one- and two-processor systems. It outperforms IntranetWare 4.11 and its performance characteristics help keep users more productive and aid system administrators in providing appropriate file-server capacity.

Price/Performance

We calculated price/performance by dividing the street price of the software tested by the peak throughput measured in megabits per second. We left out the cost of the computer because the tests were run on the same system and because we assumed you were making a decision about which file server software to use.

We obtained a street price of $2,659 for a 72-user license of Windows NT Server 4.0 by requesting a quote from a value-added reseller (VAR). Likewise, a VAR quoted us a $6,100 street price for a 75-user license for IntranetWare 4.11 (because of the way Novell licenses IntranetWare 4.11, it is less expensive to get a license for the extra three users than it would be to get a license for exactly 72 users).

Products Tested

Configuration and Tuning

We used the same Compaq ProLiant 5500 to test both Windows NT Server 4.0 and IntranetWare 4.11. Table 2 shows the system configuration we used.

Table 2: Compaq ProLiant 5500 Configuration

Feature

Configuration

CPU 2 x 200 MHz Pentium Pro (only one was enabled for all one-processor tests)

Cache: L1: 16 KB (8 KB I + 8 KB D); L2: 512 KB

RAM 256 MB EDO ECC
Disk SMART-2/P Array Controller, RAID 0, acceleration enabled (50% read and 50% write) with two logical drives:

Drive 1: 4.3 GB (1 x 4.3 GB disk) holds OS and paging file
Drive 2: 16.8 GB (8 x 2.1 GB disks) holds NetBench data

Networks 1 x Dual NetFlex 3 PCI Network Interface Card (2 x 100Base-TX networks)

Windows NT Server 4.0 Configuration

  • Service Pack 3 installed
  • Server set to maximize file sharing
  • Foreground application boost set to NONE
  • Set registry entries HKEY_LOCAL_MACHINE | SYSTEM | CurrentControlSet | Services:
  • In Cpqnf31 and Cpqnf32 | Parameters set MaxReceives = 200
  • In Tcpip | Parameters set Tcpwindowsize = 17520

IntranetWare 4.11 Configuration

  • Support Pack v5.0 installed
  • NetWare over IP service installed
  • SMP support installed for two-processor configuration
  • Changes in Startup.ncf:
  • Set maximum packet receive buffers=2000
  • Set minimum packet receive buffers=1000
  • Set maximum physical receive packet size=1514
  • Changes in Autoexec.ncf:
  • Set enable file compression=off
  • Set maximum concurrent disk cache writes=4000
  • Set maximum service processes=100
  • Set immediate purge of deleted files=on

Test Lab

The Test Systems and Network Configuration

Mindcraft ran these tests using a total of 72 test systems configured as shown in Table 3.

Table 3: Test Systems Configuration

Feature

Configuration

CPU 133 MHz Pentium. All are identical Mitac systems.
RAM 32 MB
Disk 1 GB IDE; standard Windows 95 driver
Network All systems used Intel Pro/100B LAN Adapter (100Base-TX) using e100b.sys driver version 2.02

6 x Bay Networks LS28115 switches set up as two networks – one for even numbered test systems and one for odd numbered ones.

Network software: Windows 95 TCP/IP driver.

Operating System Windows 95, version 4.0

By having one network for even numbered test systems and another for the odd numbered ones, the load on each network into the server was balanced. Figure 2 shows the test lab configuration.

Figure 2: Test Lab Configuration

Test Lab (11924 bytes)

Mindcraft Certification

Mindcraft, Inc. conducted the performance tests described in this report on May 26 and 27, 1998.

Mindcraft certifies that the results reported herein represent the file-server performance of Microsoft Windows NT Server 4.0 and Novell IntranetWare 4.11 running on a Compaq ProLiant 5500 as measured by NetBench 5.01.

Our test results should be reproducible by others who use the same test lab configuration as well as the computer and software configurations and modifications documented in this report.

NetBench Configuration and Results

NetBench Test Suite Configuration Parameters

 

Parameter

Value

Comment

Ramp Up 1 iteration Iteration is one complete execution of the Disk Mix script. This is the number of iterations at the beginning of a test mix during which NetBench ignores any file operations that occur.
Ramp Down 1 iteration Iteration is one complete execution of the Disk Mix script. This is the number of iterations at the end of a test mix during which NetBench ignores any file operations that occur.
Length 3 iterations This is the number of iterations during which NetBench measures any file operations that occur.
Delay 5 seconds How long a test system is to wait before starting a test after it is told by the controller to start. Each test system will pick a random number less than or equal to this value to stagger the start times of all test systems.
Think Time 2 seconds How long each test system will wait before performing the next piece of work.
Workspace 20 MB The size of the data files used by a test system, each of which has its own workspace.
Save Workspace Yes The last mix has this parameter set to No to clean up after the test is over.
Number of Mixes 10 Each mix tests the server with a different number of test systems. Mix 1 uses 1 system, Mix 2 uses 8 systems, and subsequent mixes increment the number of test systems by 8.
Number of Clients 72 The maximum number of test systems available to be used by any test mix. The actual number of test systems that participate in a mix depends on the number specified in the mix definition and whether an error occurred to take a test system out of a particular mix.

Windows NT Server 4.0 on a One-Processor ProLiant 5500


Mix Name

Clients Participating

Total Throughput (bytes/sec)

Total Throughput (Mbits/sec)

dm_1_client

1

480,154

3.7
dm_8_clients

8

3,757,741

28.7
dm_16_clients

16

7,206,831

55.0
dm_24_clients

24

9,436,741

72.0
dm_32_clients

32

11,584,102

88.4
dm_40_clients

40

12,100,342

92.3
dm_48_clients

48

12,303,331

93.9
dm_56_clients

56

12,475,980

95.2
dm_64_clinets

64

12,054,814

92.0
dm_72_clients

72

11,510,079

87.8

Windows NT Server 4.0 on a Two-Processor ProLiant 5500


Mix Name

Clients Participating

Total Throughput (bytes/sec)

Total Throughput (Mbits/sec)

dm_1_client

1

481,333 3.7
dm_8_clients

8

3,767,163 28.7
dm_16_clients

16

7,261,316 55.4
dm_24_clients

24

10,285,091 78.5
dm_32_clients

32

12,946,213 98.8
dm_40_clients

40

14,331,940 109.3
dm_48_clients

48

15,125,577 115.4
dm_56_clients

56

15,928,040 121.5
dm_64_clinets

64

15,821,999 120.7
dm_72_clients

72

15,648,458 119.4

IntranetWare 4.11 on a One-Processor ProLiant 5500


Mix Name

Clients Participating

Total Throughput (bytes/sec)

Total Throughput (Mbits/sec)

dm_1_client

1

501,639 3.8
dm_8_clients

8

3,964,508 30.2
dm_16_clients

16

7,658,403 58.4
dm_24_clients

24

9,597,453 73.2
dm_32_clients

32

10,994,043 83.9
dm_40_clients

40

10,608,408 80.9
dm_48_clients

48

10,956,236 83.6
dm_56_clients

56

10,747,647 82.0
dm_64_clinets

64

10,435,372 79.6
dm_72_clients

72

10,217,225 78.0

IntranetWare 4.11 on a Two-Processor ProLiant 5500


Mix Name

Clients Participating

Total Throughput (bytes/sec)

Total Throughput (Mbits/sec)

dm_1_client

1

499,475 3.8
dm_8_clients

8

3,885,759 29.6
dm_16_clients

16

7,641,611 58.3
dm_24_clients

24

10,746,267 82.0
dm_32_clients

32

12,405,177 94.6
dm_40_clients

40

14,300,491 109.1
dm_48_clients

48

14,748,965 112.5
dm_56_clients

56

14,976,923 114.3
dm_64_clinets

64

14,353,086 109.5
dm_72_clients

72

13,724,784 104.7
 

NOTICE:

The information in this publication is subject to change without notice.

MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR USE OF THIS MATERIAL.

This publication does not constitute an endorsement of the product or products that were tested. This test is not a determination of product quality or correctness, nor does it ensure compliance with any federal, state or local requirements.

The Mindcraft tests discussed herein were performed without independent verification by Ziff-Davis and Ziff-Davis makes no representations or warranties as to the results of the tests.

Mindcraft is a registered trademark of Mindcraft, Inc.

Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.


Copyright 1997-98. Mindcraft, Inc. All rights reserved.
Mindcraft is a registered trademark of Mindcraft, Inc.
For more information, contact us at: info@mindcraft.com
Phone: +1 (408) 395-2404
Fax: +1 (408) 395-6324