Mindcraft used our LDAP server benchmark, DirectoryMark, to measure the performance of Netscape Directory Server 3.0, Netscape Directory Server 1.0, and Novell LDAP Services for NDS 1.0. Figure 1 summarizes the peak performance measured in search operations/second. It shows that Netscape Directory Server 3.0 (NSDS3) is 4.8 times faster than its predecessor, Netscape Directory Server 1.0 (NSDS1), and that NSDS3 is 229.5 times faster than Novell LDAP Services for NDS 1.0 (LDAP for NDS). We ran the tests on a Hewlett-Packard NetServer LX Pro system using a database of 10,000 entries with random, realistic values in each data item.
Figure 1: Peak Performance
Figure 2 shows the average response times for these LDAP servers. NSDS3 responded to queries an average of 5.2 times faster than NSDS1 and 254 times faster than LDAP for NDS.
Figure 2: Average Response Times
Table 1 shows the detailed server computer configuration. Tables 2, 3, and 4 give the software configurations for NSDS3, NSDS1, and LDAP for NDS, respectively.
Table 1: HP NetServer LX Pro Configuration
Table 2: Netscape Directory Server 3.0 Configuration
Table 3: Netscape Directory Server 1.0 Configuration
Table 4: Novell LDAP Services for NDS Configuration
Figure 3 shows the test lab configuration we used. We needed only one client test system, configured as shown in Table 5, to load the LDAP servers.
Figure 3: LDAP Server Test Lab Configuration
Table 5: Client Test System
We used a dedicated 100Base-TX network for all tests.
Mindcraft used the DirectoryMark benchmark for these tests. It is made up of four major programs:
Table 6: DirectoryMark Configuration
For the Netscape tests, we booted Windows NT 4.0 on the NetServer directly from its hard disks. For the Novell tests, we first booted MS-DOS on the NetServer from a floppy disk and then started the IntranetWare server from the C: hard disk.
We ran each test with a 15 minute warm-up time to fill the server's cache and then a 10 minute test time. The test was done by automatically restarting the dm_client at the end of the warm-up time.
We ran perfmon on the NetServer during the Netscape tests and had it log processor, memory, and physical disk performance information to disk every 5 seconds. During the Novell tests we ran monitor.nlm to track performance on the server and logged LDAP transaction summaries to a file.
Other than connection and anonymous LDAP BIND operations, we used only search operations for our testing. This was done because searches account for almost all of the LDAP operations encountered by real servers. We chose the particular mix of general searches vs. UID searches to simulate a forward-looking scenario where e-mail and other programs will be large users of LDAP servers.
We determined via preliminary testing that a single client test system would be sufficient to load the LDAP servers we tested. When we looked at the client test system CPU utilization we found it averaged 75% for the NSDS3 tests, 16% for the NSDS1 tests, and less than 2% for the LDAP for NDS tests. At the same time we were able to drive the NetServer's CPU to full utilization for the Netscape products and to 95% utilization for LDAP for NDS.
These tests were constructed to facilitate comparing the performance of NSDS3, NSDS1, and LDAP for NDS. We used the same computer system as the server and the same client test system for all tests. However, the Netscape and Novell products run on different operating systems. The overall performance we measured is a function of both the LDAP server and its underlying operating system. This combined performance is just what users of LDAP servers will experience across a network (impacted by network traffic, of course).
The four primary factors that limit performance of LDAP servers are:
We'll examine each factor individually.
Server CPU and Memory Performance
When we tested both NSDS3 and NSDS1, the NetServer's CPU was 100% utilized. For the LDAP for NDS test, the NetServer's CPU was 95% utilized. Since the client test system's CPU utilization was low to moderate for all tests, we conclude that the performance we measured in our tests of NSDS3 and NSDS1 was limited by the server system's CPU, not the performance of the client test system. Because LDAP for NDS did not saturate the NetServer's CPU, another factor must have limited its performance. CPU utilization alone does not tell us if the CPU was being used efficiently by the software; we will examine that below.
Memory was not a performance-limiting factor for any of the products. Perfmon showed that the Netscape products used less than 100 MB of memory. Similarly, LDAP for NDS used less than 100 MB.
Disk Subsystem Performance
The performance monitoring we did for NSDS3 and NSDS1 showed an average of less than one disk access per second once the cache was filled. We could not log disk activity for LDAP for NDS, however, the disk activity lights on the drives indicated very few disk accesses. These are reasonable observations given that the entire 10,000 person LDAP database can easily fit into memory on the NetServer system we used. Thus, we conclude that the disk subsystem was not a performance limiting factor.
The network was not a limiting factor for any of the LDAP servers. Network bandwidth utilization averaged under 40% in all cases because the operation rate was low and so little data was transferred for each operation.
Operating System and LDAP Software Performance
The performance of LDAP server software is affected by the overall performance of the operating system on which it is running. However, because we were doing "black box" testing, we could not readily separate the performance of the operating system from that of the LDAP server we tested. So, our analysis will focus on the LDAP servers themselves.
NSDS3 showed significant performance improvements over the pervious version, NSDS1. During the 10-minute test, NSDS3 handled 110,169 transcations while NSDS1 handled 23,060 transactions. Because the NetServer's CPU was 100% utilized and because the same operating system configuration was used for both Netscape Directory Server products, we conclude that NSDS3 makes more efficient use of the available resources than NSDS1. Even though the performance of both NSDS3 and NSDS1 will increase as the speed of the server's CPU increases, you will get more of a bang for your buck with NSDS3.
The performance of LDAP for NDS is poor - it did only 474 transactions for the 10-minute test period. While the test was running, we looked at the CPU utilization in the scheduling option of the monitor.nlm. It showed that all of the LDAP processes or threads combined used under 1% of the CPU. In fact, all of the monitored threads used under 5% of the CPU, yet the overall CPU utilization averaged about 95%. What caused this to happen?
We believe that most of the overhead is in searching the NDS data structures. We could not find any documentation for Novell LDAP Services for NDS 1.0 that specified how to create different types of attribute indexing nor were there indexing options available in the LDAP or NDS management programs. This leads us to conclude that some types of searches, such as wildcard searches, may go through the entire LDAP database sequentially to resolve the search because more optimal indexing techniques are not available. An additional inefficiency of LDAP for NDS comes from its mapping LDAP attributes to NDS attributes, although we couldn't measure how much overhead this mapping contributed.
The performance of the Netscape Directory Server 3.0 is a substantial improvement over that of Netscape Directory Server 1.0 and is high enough that it can be used in very demanding applications. Netscape Directory Server 3.0 can take full advantage of high-speed CPUs.
Regardless of the cause, search times for Novell LDAP Services for NDS are too long and its performance is unacceptably low.
Mindcraft, Inc. conducted the performance tests described in this report between October 30 and December 12, 1997, and April 24 to 27, 1998.
Mindcraft certifies that the results reported herein fairly represent the performance of systems tested as measured by the DirectoryMark benchmark configured as specified. Our test results should be reproducible by others who use the same test lab configuration as well as the computer and software configurations and modifications documented in this report.
|Copyright © 1997-98. Mindcraft, Inc. All rights reserved.
Mindcraft is a registered trademark of Mindcraft, Inc.
For more information, contact us at: firstname.lastname@example.org
Phone: +1 (408) 395-2404
Fax: +1 (408) 395-6324