Netegrity SiteMinder 4.51
AuthMark Performance

By Bruce Weiner
(PDF version, 62 KB)
March 26, 2001
Update to March 22, 2001 version

Contents

Executive Summary
   Conclusions
   Mindcraft Certification
Analysis
Methodology
Configuration
iLOAD MVP
AuthMark

Disclosure

Netegrity Inc. sponsored the testing in this report. Mindcraft, Inc. conducted the performance tests described in this report at Sun’s test lab in Menlo Park, California.

Acknowledgement

Executive Summary

Netegrity SiteMinder 4.51 delivers 112,202 logins per minute and 20,179 Extranet Sequences (221,969 operations) per minute for 1,000,000 users, achieving the highest overall and per CPU performance that we've seen

Netegrity SiteMinder 4.51 delivers dramatic performance improvement over earlier versions and sets a new performance standard for authentication/authorization products. Throughout our tests, SiteMinder outperformed all other similar products we've tested both in overall performance and in performance per policy/security server CPU.

Mindcraft^® tested Netegrity SiteMinder 4.51 running on a mix of Sun Enterprise servers. For these tests, we used Mindcraft’s iLOAD MVP™ test tool running the AuthMark™ Login and Extranet Scenarios.  

Login Scenario

The Login Scenario represents the type of load commonly seen at portal sites. It simulates users accessing protected resources via Web servers. The Login Scenario assumes that 10% of a portal's user population logs in concurrently to use portal resources. All tests were done using a 1,000,000-user directory with 100,000 active users. 

The Login Scenario measures the combination of one user authentication and one authorization for access to a resource (called a Login). The Result Analysis section in the second part of this white paper explains the benchmark results.

The SiteMinder Policy Server is the control point for all authentication and authorization. Our tests were structured to push the Policy Server systems as closely as possible to 100% CPU utilization. Table 1 summarizes the Login Scenario performance as a function of the SiteMinder Policy Server system(s) configuration. The Scaling Factor in Table 1 shows how much faster a configuration is compared to a single system with one CPU using one directory server, the smallest configuration. 

Table 1: SiteMinder Login Performance Scalability - 1,000,000 Users

The CPU utilizations for the Policy Server configurations with one system, three CPUs and with two systems, two and three CPUs show that we did not have enough load generators and Web servers, or fast enough ones, to drive the Policy Servers to full CPU utilization. If the lab had enough load generators (which ran at 70% CPU utilization for the highest performance tests) and Web servers available, we fully expect that SiteMinder would have achieved more logins per minute than it did.

Figure 1 shows SiteMinder's Login performance from Table 1 by server configuration.

Figure 1: SiteMinder Login Scalability for 1,000,000 Users

Extranet Scenario

The Extranet Scenario measures the combination of one user authentication and 10 authorizations for access to resources (these 11 operations constitute one Extranet sequence). The Extranet Scenario, because it uses a more realistic mix of operations than the Login Scenario, provides a better basis for capacity planning purposes.

Table 2 compares the SiteMinder Extranet Scenario performance to that of the Login Scenario for the same hardware configuration - one Policy Server with one CPU. The results in Table 2 demonstrate that the SiteMinder Policy Server performs authorizations several times faster than authentications. It is not possible to calculate the exact performance difference because the CPU utilization of the Policy Server CPU was 50% for the Extranet test and was 100% for the Login test. The CPUs in the Web servers averaged 95% utilization while the load generator CPUs averaged 70% utilization. This means that the Extranet performance would have been significantly higher, if there were enough load generators and Web servers available in the lab to drive the Policy Server's CPU to full utilization. 

Table 2: SiteMinder Extranet and Login Performance - 1 Policy Server with 1 CPU

Conclusions

The benchmark results lead us to conclude that:

• Netegrity SiteMinder 4.51 outperforms all other products we've tested so far for the AuthMark Login and Extranet Scenarios.
• SiteMinder 4.51 delivers the highest Login and Extranet performance per policy/security server CPU of any product we have tested to date.
• SiteMinder delivers outstanding performance scaling as CPUs and Policy Servers are added to a configuration.

Mindcraft Certification

Mindcraft certifies that the results reported accurately represent the performance of Netegrity SiteMinder 4.51 running on Sun Enterprise servers configured as specified herein and as measured by AuthMark benchmark.

Our test results should be reproducible by others using the same test lab configuration, the same Sun server configurations, and the same software configurations documented in this white paper.