Contents

Executive Summary 
Server Environments Tested
Test Lab
Test Procedures
Performance Analysis
Mindcraft's Certification
Glossary

Executive Summary

Mindcraft used our LDAP server benchmark, DirectoryMark, to measure the performance of Netscape Directory Server 3.0, Netscape Directory Server 1.0, and  Novell LDAP Services for NDS 1.0. Figure 1 summarizes the peak performance measured in search operations/second. It shows that Netscape Directory Server 3.0 (NSDS3) is 4.8 times faster than its predecessor, Netscape Directory Server 1.0 (NSDS1), and that NSDS3 is 229.5 times faster than Novell LDAP Services for NDS 1.0 (LDAP for NDS). We ran the tests on a Hewlett-Packard NetServer LX Pro system using a database of 10,000 entries with random, realistic values in each data item.

Figure 1: Peak Performance

Figure 2 shows the average response times for these LDAP servers. NSDS3 responded to queries an average of 5.2 times faster than NSDS1 and 254 times faster than LDAP for NDS.

Figure 2: Average Response Times

Server Environments Tested

Table 1 shows the detailed server computer configuration. Tables 2, 3, and 4 give the software configurations for NSDS3, NSDS1, and LDAP for NDS, respectively.

Table 1: HP NetServer LX Pro Configuration

Table 2: Netscape Directory Server 3.0 Configuration

Table 3: Netscape Directory Server 1.0 Configuration

Table 4: Novell LDAP Services for NDS Configuration

Test Lab

Figure 3 shows the test lab configuration we used. We needed only one client test system, configured as shown in Table 5, to load the LDAP servers.

Figure 3: LDAP Server Test Lab Configuration

Table 5: Client Test System

We used a dedicated 100Base-TX network for all tests.

Test Procedures

Mindcraft used the DirectoryMark benchmark for these tests. It is made up of four major programs:

1. dbgen: The database generation program. It generates an LDIF file containing information for the specified number of people.
2. scriptgen: The test script generation program. It generates test scripts that use various LDAP operations. The frequency for each type of operation is specified by a percentage. The LDAP operations, and their associated parameters, used in the testing for this report were:

 

Parameter	Meaning
-u	Unique ID search. This search uses the UID attribute in the database. This type of search is an exact match on an attribute with a unique value. UID searches would be used, for example, by an e-mail program to get information associated with an e-mail address.
-s	General search. General searches are further subdivided in the following mix: 10% Searches resulting in a "not found" status 20% Wildcard search on CN attribute with * at the beginning of the value searched for 30% Exact match search on CN attribute 40% Exact match search on an attribute randomly selected from: CN, SN, GN, UID.

3. dm_master: The master controller for the DirectoryMark benchmark. It starts the dm_client program, collects test results, and formats a report.
4. dm_client:The client program. For the duration of the test time it loops through the test script issuing the specified operations against the LDAP server under test.

Table 6 shows the DirectoryMark configuration parameters we used.

Table 6: DirectoryMark Configuration

For the Netscape tests, we booted Windows NT 4.0 on the NetServer directly from its hard disks. For the Novell tests, we first booted MS-DOS on the NetServer from a floppy disk and then started the IntranetWare server from the C: hard disk.

We ran each test with a 15 minute warm-up time to fill the server's cache and then a 10 minute test time. The test was done by automatically restarting the dm_client at the end of the warm-up time.

We ran perfmon on the NetServer during the Netscape tests and had it log processor, memory, and physical disk performance information to disk every 5 seconds. During the Novell tests we ran monitor.nlm to track performance on the server and logged LDAP transaction summaries to a file.

Other than connection and anonymous LDAP BIND operations, we used only search operations for our testing. This was done because searches account for almost all of the LDAP operations encountered by real servers. We chose the particular mix of general searches vs. UID searches to simulate a forward-looking scenario where e-mail and other programs will be large users of LDAP servers.

We determined via preliminary testing that a single client test system would be sufficient to load the LDAP servers we tested. When we looked at the client test system CPU utilization we found it averaged 75% for the NSDS3 tests, 16% for the NSDS1 tests, and less than 2% for the LDAP for NDS tests. At the same time we were able to drive the NetServer's CPU to full utilization for the Netscape products and to 95% utilization for LDAP for NDS.

Performance Analysis

These tests were constructed to facilitate comparing the performance of NSDS3, NSDS1, and LDAP for NDS. We used the same computer system as the server and the same client test system for all tests. However, the Netscape and Novell products run on different operating systems. The overall performance we measured is a function of both the LDAP server and its underlying operating system. This combined performance is just what users of LDAP servers will experience across a network (impacted by network traffic, of course).

The four primary factors that limit performance of LDAP servers are:

1. The server's CPU and  memory;
2. The disk subsystem;
3. The network; and
4. The software, including the operating system and the LDAP server.

We'll examine each factor individually.

Server CPU and Memory Performance

When we tested both NSDS3 and NSDS1, the NetServer's CPU was 100% utilized. For the LDAP for NDS test, the NetServer's CPU was 95% utilized. Since the client test system's CPU utilization was low to moderate for all tests, we conclude that the performance we measured in our tests of NSDS3 and NSDS1 was limited by the server system's CPU, not the performance of the client test system. Because LDAP for NDS did not saturate the NetServer's CPU, another factor must have limited its performance. CPU utilization alone does not tell us if the CPU was being used efficiently by the software; we will examine that below.

Memory was not a performance-limiting factor for any of the products. Perfmon showed that the Netscape products used less than 100 MB of memory. Similarly, LDAP for NDS used less than 100 MB.

Disk Subsystem Performance

The performance monitoring we did for NSDS3 and NSDS1 showed an average of less than one disk access per second once the cache was filled. We could not log disk activity for LDAP for NDS, however, the disk activity lights on the drives indicated very few disk accesses. These are reasonable observations given that the entire 10,000 person LDAP database can easily fit into memory on the NetServer system we used. Thus, we conclude that the disk subsystem was not a performance limiting factor.

Network Performance

The network was not a limiting factor for any of the LDAP servers. Network bandwidth utilization averaged under 40% in all cases because the operation rate was low and so little data was transferred for each operation.

Operating System and LDAP Software Performance

The performance of LDAP server software is affected by the overall performance of the operating system on which it is running. However, because we were doing "black box" testing, we could not readily separate the performance of the operating system from that of the LDAP server we tested. So, our analysis will focus on the LDAP servers themselves.

NSDS3 showed significant performance improvements over the pervious version, NSDS1. During the 10-minute test, NSDS3 handled 110,169 transcations while NSDS1 handled 23,060 transactions. Because the NetServer's CPU was 100% utilized and because the same operating system configuration was used for both Netscape Directory Server products, we conclude that NSDS3 makes more efficient use of the available resources than NSDS1. Even though the performance of both NSDS3 and NSDS1 will increase as the speed of the server's CPU increases, you will get more of a bang for your buck with NSDS3.

The performance of LDAP for NDS is poor - it did only 474 transactions for the 10-minute test period. While the test was running, we looked at the CPU utilization in the scheduling option of the monitor.nlm. It showed that all of the LDAP processes or threads combined used under 1% of the CPU. In fact, all of the monitored threads used under 5% of the CPU, yet the overall CPU utilization averaged about 95%. What caused this to happen?

We believe that most of the overhead is in searching the NDS data structures. We could not find any documentation for Novell LDAP Services for NDS 1.0 that specified how to create different types of attribute indexing nor were there indexing options available in the LDAP or NDS management programs. This leads us to conclude that some types of searches, such as wildcard searches, may go through the entire LDAP database sequentially to resolve the search because more optimal indexing techniques are not available. An additional inefficiency of LDAP for NDS comes from its mapping LDAP attributes to NDS attributes, although we couldn't measure how much overhead this mapping contributed.

Conclusion

The performance of the Netscape Directory Server 3.0 is a substantial improvement over that of Netscape Directory Server 1.0 and is high enough that it can be used in very demanding applications. Netscape Directory Server 3.0 can take full advantage of high-speed CPUs.

Regardless of the cause, search times for Novell LDAP Services for NDS are too long and its performance is unacceptably low.

Mindcraft's Certification

Mindcraft, Inc. conducted the performance tests described in this report between October 30 and December 12, 1997, and April 24 to 27, 1998.

Mindcraft certifies that the results reported herein fairly represent the performance of systems tested as measured by the DirectoryMark benchmark configured as specified. Our test results should be reproducible by others who use the same test lab configuration as well as the computer and software configurations and modifications documented in this report.

Glossary

LDAP

An acronym for Lightweight Directory Access Protocol.

LDIF

An acronym for LDAP Data Interchange Format. It is an ASCII format for specifying values for LDAP schema attributes.

NDS

An acronym for Novell Directory Services.

Thread

Short for "thread of control in a process." As used in this report, it means the number of DirectoryMark requestors threads simultaneously making requests of an LDAP server.