Dear Vedrin,

thank you very much for these experiments. So is a powerful LDAP server
and container tuning the key to performance improvement? Or rather
optimizing the policy service code (you mentioned MySQL connection pool
usage, but it didn't sound like that was the main step). 
Do you know what effect the newer OpenAM release (9.5.3) has? As you
may know, I have tried upgrading several times, but didn't succeed, most
likely due to a bug in OpenAM 9.0 (see
https://bugster.forgerock.org/jira/browse/OPENAM-464). We also use an
external LDAP server (the one feeding the main website), but it might be
not too powerful.
Again, thank you for investigating the stuff! To make best use of it
and transition to a more powerful deployment, we will need a way to
migrate our existing policies (see the above topic on the issue
tracker).

Best regards
Andreas

On Mon, 4 Jul 2011 10:23:34 +0300, Vedrin Jeliazkov
<vedrin.jeliazkov at gmail.com> wrote:
> Hi Folks,
> 
> I've spent a few days on studying various OpenAM setups in the context
> of the OpenTox AA API. After several sleepless nights I've settled on
> the following configuration for our local  test deployment:
> 
> OpenAM 9.5.3 RC1 (Build date 20110506)
> http://download.forgerock.org/downloads/openam/snapshot9.5/openam_953RC1.war
> OpenDJ Release 2.4.3 (Build date 20110617)
> http://download.forgerock.org/downloads/opendj/2.4.3/install/QuickSetup.jnlp
> Java 1.6.0_26
> http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u26-download-400750.html
> Tomcat 7.0.16 (http://tomcat.apache.org/download-70.cgi)
> MySQL 5.0.91
> 
> The JVM and MySQL versions are both 64-bit to allow more flexibility
> memory-wise. I've also chosen to store the defined policies in an
> external LDAP server along with the user attributes, rather than
> relying on the OpenAM's built-in LDAP data store.
> 
> The OpenAM and Policy services are deployed online at:
> 
> https://pirin.uni-plovdiv.bg:8443/opensso
> https://pirin.uni-plovdiv.bg:8443/Pol
> 
> You can access them with your OpenTox credentials and through the same
> OpenTox AA API as the production services, running at
> http://opensso.in-silico.ch/
> 
> Some findings of general interest:
> 
> 1) The CPU usage proportion between OpenDJ, MySQL, and OpenAM+Policy
> is roughly 10:2:1. This highlights the need to have a dedicated
> powerful server (even better -- a pool of cooperating servers) for
> LDAP or eventually use a different backend (e.g. general purpose
> RDBMS) that scales better for workloads that are not predominantly
> read-only.
> 
> 2) The services scale reasonably well at least up to 15000 policies in
> a workload of random policy creations, evaluations and removals; some
> provisional statistics are available online at:
> 
> http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV
> 
> You might notice that this AA service instance is several times faster
> than OpenTox's current production AA services, even though it holds a
> substantially higher number of policies and runs on a (almost)
> pre-historical piece of hardware. In particular, compare the following
> test workflows:
> 
> Production:
> http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-1
> http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-3
> 
> Test:
> http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-1a
> http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-2
> http://ambit.uni-plovdiv.bg/cgi-bin/smokeping.cgi?target=IDEA-DEV.AA.TestWorkflow-3a
> 
> The sharp latency increase in the beginning of the SmokePing's graph
> for the test instance corresponds to the insertion of 15K policies,
> accompanied by simultaneous policy evaluations (circa 30K) and about
> 1K policy removals. I'll be running more policy insertion/evaluation
> scripts in the following hours/days to check how far we can get with
> such setup.
> 
> 3) As AM has noted already, there's some room for improvements in the
> Policy service, especially regarding the use of a db connection pool
> and table indexes, however such improvements are not likely to
> influence the overall performance dramatically. Nevertheless, it makes
> sense to implement them and Nina will take care of this.
> 
> 4) There's some room for further tuning of the JVM, the servlet
> container, as well as OpenAM and OpenDJ, that could help running the
> AA service smoothly with hundreds of thousands of defined resource
> access policies (if necessary :-).
> 
> Kind regards,
> Vedrin
> _______________________________________________
> Development mailing list
> Development at opentox.org
> http://www.opentox.org/mailman/listinfo/development

-- 
http://www.maunz.de

      Not always does the squeaky wheel get the grease- sometimes it
gets replaced.