Hi Folks,

I've spent a few days on studying various OpenAM setups in the context
of the OpenTox AA API. After several sleepless nights I've settled on
the following configuration for our local  test deployment:

OpenAM 9.5.3 RC1 (Build date 20110506)
http://download.forgerock.org/downloads/openam/snapshot9.5/openam_953RC1.war
OpenDJ Release 2.4.3 (Build date 20110617)
http://download.forgerock.org/downloads/opendj/2.4.3/install/QuickSetup.jnlp
Java 1.6.0_26 http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u26-download-400750.html
Tomcat 7.0.16 (http://tomcat.apache.org/download-70.cgi)
MySQL 5.0.91

The JVM and MySQL versions are both 64-bit to allow more flexibility
memory-wise. I've also chosen to store the defined policies in an
external LDAP server along with the user attributes, rather than
relying on the OpenAM's built-in LDAP data store.

The OpenAM and Policy services are deployed online at:

https://pirin.uni-plovdiv.bg:8443/opensso
https://pirin.uni-plovdiv.bg:8443/Pol

You can access them with your OpenTox credentials and through the same
OpenTox AA API as the production services, running at
http://opensso.in-silico.ch/

Some findings of general interest:

1) The CPU usage proportion between OpenDJ, MySQL, and OpenAM+Policy
is roughly 10:2:1. This highlights the need to have a dedicated
powerful server (even better -- a pool of cooperating servers) for
LDAP or eventually use a different backend (e.g. general purpose
RDBMS) that scales better for workloads that are not predominantly
read-only.

2) The services scale reasonably well at least up to 15000 policies in
a workload of random policy creations, evaluations and removals; some
provisional statistics are available online at:

http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV

You might notice that this AA service instance is several times faster
than OpenTox's current production AA services, even though it holds a
substantially higher number of policies and runs on a (almost)
pre-historical piece of hardware. In particular, compare the following
test workflows:

Production:
http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-1
http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-3

Test:
http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-1a
http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-2
http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-3a

The sharp latency increase in the beginning of the SmokePing's graph
for the test instance corresponds to the insertion of 15K policies,
accompanied by simultaneous policy evaluations (circa 30K) and about
1K policy removals. I'll be running more policy insertion/evaluation
scripts in the following hours/days to check how far we can get with
such setup.

3) As AM has noted already, there's some room for improvements in the
Policy service, especially regarding the use of a db connection pool
and table indexes, however such improvements are not likely to
influence the overall performance dramatically. Nevertheless, it makes
sense to implement them and Nina will take care of this.

4) There's some room for further tuning of the JVM, the servlet
container, as well as OpenAM and OpenDJ, that could help running the
AA service smoothly with hundreds of thousands of defined resource
access policies (if necessary :-).

Kind regards,
Vedrin