Setting the Record Straight:
Where Arcot Systems Got It Right and Wrong

By Bruce Weiner

April 10, 2001

In an April 9, 2001 press release entitled "ARCOT ANNOUNCES ACCESS CONTROL SERVER 4X FASTER THAN MARKET LEADER," Arcot Systems, Inc. misrepresented Mindcraft's performance test results and made apples-to-oranges comparisons. This press release harks back to the days before an industry-standard benchmark was available.

We want to set the record straight. In addition, we invite Arcot to show that they are the performance leader they claim to be by using the AuthMark Benchmark, which has become the de facto industry-standard benchmark for measuring authentication and authorization performance.

What's Right

The Arcot press release makes a couple of points quite accurately:

• "Companies today are aggressively pursuing e-business initiatives with the goal of delivering more services, to more customers, more cost effectively. Business portals are enforcing more sophisticated access privileges for select customers and partners, while banks are building integrated financial services portals offering online banking, bill payment, mortgage, and brokerage services that can be seamlessly accessed by millions of consumers."
  
  
• "At the same time, companies must protect customer and vendor relationships, and confidential information."
  

What's Wrong

Unfortunately, the Arcot press release went on to compare a poorly documented performance test of Arcot AccessFort with results from Mindcraft's performance test reports for Netegrity, Oblix, Securant, and Entrust (they acquired enCommerce for whom we did the test).

What's Missing from Arcot's Claims

1. The press release does not say how to get the testing report it references and the report is not available at Arcot's Web site as well. How can one find out the details of what was tested and how the testing was conducted?

2. There are charts at Arcot's Web site that purport to show AccessFort's performance relative to products W, X, Y, and Z. What's the relation between these products and the ones from the competitors mentioned in the press release?

3. These charts show response times thereby implying that the times were published in Mindcraft's reports. But we did not publish response times. Instead, Arcot made up the response times simply by taking the reciprocal of the operation rate. As anyone experienced in benchmarking server products can tell you, the actual response times typically are not the reciprocal of the operation rate.

4. Why didn't Arcot's press release and Web site mention that they were comparing their test results to Mindcraft's?

5. What percentage of the authorizations were made from a Web server and from non-Web applications?

6. What was the nature of the non-Web applications used in the Arcot test?

7. What tools were used to generate the non-Web load?

8. What was the configuration of both the Web and the non-Web load generator systems and how many were there?

9. How many Web and non-Web servers were used and what were their configurations?

10. What information was cached and where?

11. What was used to store the user authentication and authorization information? An LDAP directory server? A DBMS?

12. How many servers were used to store the user authentication and authorization information and what was their configuration?

13. Why did Arcot compare their tests with both Web and non-Web applications to Mindcraft's tests that used only Web applications?

14. Who "independently audited" the test? What did they audit? Why didn't they publish a report?

The Wrong Stuff

• Except for the Entrust/enCommerce test, all of the Mindcraft test results are based on login (authentication) performance, not authorization performance as purported by the Arcot press release. This is a huge misrepresentation! It is quite common for a security product to have authorization performance that is 5- to 10-times faster than its authentication performance.

• The performance numbers attributed to the products from Netegrity, Oblix, and Securant were manufactured by Arcot and do not appear in Mindcraft's reports. Arcot took login performance numbers and doubled them for their press release. It is naive and misleading to extrapolate authorization performance from authentication performance.

• All of Mindcraft's security product performance tests use Web access only. Non-Web applications put an entirely different load on a security server thereby invalidating any performance comparison between the two.

• The Entrust product's authorization performance in the press release includes a system and CPUs used to store the user authentication and authorization information while the Arcot configuration does not.

Why Performance Is Important?

Here are some reasons why performance should be important to you:

1. A product's performance has a direct relationship to how much it will cost to deploy. For example, a product that is twice as fast as another can be deployed on fewer systems saving money for the unneeded hardware and software.

2. Your customers and users will find your Web site, applications, and the like to be more responsive. This is especially important for Web sites because users quickly become frustrated with poor performance and go to other sites resulting in lost revenue.

3. User productivity will improve because they are not waiting for the information they need.