Setting the Record Straight:
Where Arcot Systems Got It Right and Wrong

By Bruce Weiner

April 24, 2001
Update to our April 10, 2001 commentary

On April 16, 2001 Arcot published the benchmark report that they referenced in their April 9, 2001 press release entitled "ARCOT ANNOUNCES ACCESS CONTROL SERVER 4X FASTER THAN MARKET LEADER." Also, Arcot updated the product overview for Arcot AccessFort and the press release itself.

Did these changes address the issues we raised in our original commentary? Yes to some and no to others. Below, the issues they didn't address and the changes they made that raised more questions are discussed below.

The Issues

1. In our original commentary, we said that Arcot Systems misrepresented Mindcraft's performance test results and made apples-to-oranges comparisons.
   
   That hasn't changed. On the overall product overview page, Arcot still compares the AccessFort authorization rate to a rate they manufactured for Vendors W, X, and Y (which are Netegrity, Oblix, and Securant, respectively according to the press release). The vendor Z in the press release is Entrust and that number does appear in our report. There is still no correspondence between the vendors named in the press release and those named with single letters at the Arcot Web site. This paragraph gives you the correspondence.
   
   Arcot did make an attempt to correct their chart by modifying the legend below their Performance Comparison Chart to define "Operation = Authorization or Authentication." That does not hide the apples-to-oranges comparison, it highlights it. The overhead for doing an authorization is definitely not the same as the overhead for an authentication, at least for all of the products we've tested so far.
   

2. Arcot, if you want to extrapolate an authorization rate from our reports (something we will not do), why don't you base it on the authorization rate we reported for the AuthMark Extranet Scenario test?
   
   For example, you could have extrapolated Netegrity SiteMinder's authorization performance as follows:
   

   Item	Authorizations/minute	Comments
   Measured authorizations/minute	201,790	Measured in the Extranet Scenario.
   Authorizations/minute instead of reported authentications/minute	201,790	Since the Netegrity Policy server during the Extranet Scenario test performed at slightly more than half the authentication rate it did for the Login Scenario where the CPU was utilized 100%, one could logically extrapolate that the 20,179 authentications/minute would have used half of the total 50% CPU utilization (in reality it probably used more). This would result in doubling the measured authorizations/minute.
   Available authorizations/minute	403,580	From the Extranet Scenario accounting for the 50% CPU utilization that was unused (the sum of the above numbers).
   Total	807,160

   While this is a more reasonable way to extrapolate authorization performance than what Arcot did, we don't believe that it is accurate and we do not claim that SiteMinder will achieve this authorization rate.

   So, one could then compare the extrapolated SiteMinder authorization rate in authorizations/minute to that of AccessFort as follows:

   Policy Server Configuration	Arcot AccessFort	Netegrity SiteMinder
   1 server, 1 CPU	461,460	807,160
   1 server, 2 CPUs	807,120	807,160 (1 server, 1 CPU)

   
   

3. The following table shows the inconsistencies between the numbers reported in Arcot's press release and those in the detailed report at their Web site:

   Arcot Press Release	Arcot Detailed Report
   	Test Scenario 1 - No Web Server	Test Scenario 2 - Web Server
   809,160	810,840	807,120

   
   

4. Why didn't Arcot's press release and Web site mention that they were comparing their test results to Mindcraft's? Still no mention. The obvious place in the Arcot press release for a link to Mindcraft's reports ("Comparison figures were obtained from published benchmark reports.") is actually a link to the Arcot report, which does not include any comparison figures.

5. The detailed Arcot report does not address what percentage of the authorizations were made from a Web server and from non-Web applications. Why is it important? Because the presence of any non-Web load means that the comparisons are invalid.

   We believe that there was a very small Web load for the Arcot Scenario 2, which used a Web server, because the results are so very close to the Scenario 1 tests without a Web server (from 0.5% to 4.5% depending on the number of CPUs). Also, the report says that all Web pages were 14KB. So, 850 Web page requests/second each of which requires an authorization would use 95% of the single 100Base-TX NIC Arcot claims to have used for the Web server.

   The importance about authorization requests coming from a Web server is that the overhead of going through the Web server plug-in gets added to the authorization overhead of the AccessFort server. In the Mindcraft tests, 100% of the authorizations had this overhead.

6. What was the nature of the non-Web applications used in the Arcot test? This was answered to some degree. What wasn't answered is how many different simulated users were represented in the non-Web application? The number of simulated users wasn't covered for the Web application case either.

7. What was the configuration of both the Web and the non-Web load generator systems and how many were there? This was answered partially. But that answer raised a couple of questions:



• Tables 8 and 9 in the Arcot detailed report cover Test Client machines (does the Test Client here refer to the software load generator for the non-Web application that the report calls Test Client or to the generic term test client?) but there is no description of the systems used to run WebLoad, the Web server load generator.

• The Arcot detailed report is unclear with respect to the number of NICs in the load generators - 5 (3 x 3Com 3C905D, 3Com 3C905, Intel Pro/100+ Management Adapter). How many were really used for the tests and which ones?

8. How many Web and non-Web servers were used and what were their configurations?

• Why does Figure 9 in the Arcot detailed report show two separate networks connected by a Web server when it is supposed to be the test that doesn't use a Web server?

• Why does Figure 10 in the Arcot detailed report, which covers the test configuration that used a Web server(s?), say "Web servers" but Table 6 and the comments before it say there is only one Web server?

• The Arcot detailed report is unclear about the NICs in the servers. It shows 1 for authorization, 5 for Web and DBMS. How many were really used for the tests and which ones?

9. Why did Arcot compare their tests with both Web and non-Web applications to Mindcraft's tests that used only Web applications? Still unanswered.

10. The detailed report raises a new question:

    The report shows percentage of authorization requests with a response time less than 100 microseconds.
    
    How was this measured? There's a well known problem with measuring sub-millisecond times on Windows-based systems that results in extremely fast and incorrect times. Did the time measurement tool use the high resolution performance counter or did it take the difference between two time of day calls? Or, was the time computed from the inverse of the operation rate? The detailed report doesn't answer this.

Invitation

We invite Arcot to show that they are the performance leader they claim to be by using the AuthMark Benchmark, which has become the de facto industry-standard benchmark for measuring authentication and authorization performance.