Hi Andreas,

On 4 July 2011 10:51, Andreas Maunz <andreas at maunz.de> wrote:
> Dear Vedrin,
>
> thank you very much for these experiments. So is a powerful LDAP server
> and container tuning the key to performance improvement?

LDAP seems to be the real bottleneck, when used as data store for
policies and you have thousands of them. Unfortunately OpenAM seems to
have become too dependent on LDAP, which makes it somehow hard to
switch to another backend (e.g. MySQL) for config data storage.
However, there are some more or less supported ways to achieve this (I
haven't tried this yet). It should be fine though to use LDAP as user
store -- this doesn't seem to be much of an issue, provided that the
users are not expected (at least initially) to be more than a few
hundreds or thousands.

> Or rather
> optimizing the policy service code (you mentioned MySQL connection pool
> usage, but it didn't sound like that was the main step).

We should of course do this, but it doesn't seem to be the main issue
we're facing performance-wise.

> Do you know what effect the newer OpenAM release (9.5.3) has?

No -- I haven't compared it to older versions, but I presume that it
might have some effect in policy evaluation speed-up. I don't think
though that policy removal (which seems to be the most demanding
operation currently) would benefit much from the version upgrade,
since this looks like an inherent performance issue of LDAP servers in
general -- they're too much optimised towards reading data that is
carved in stone and occasionally adding some more :-) Another possible
bottleneck might be the XML processing/conversion that happens between
the webservice and LDAP, but this is also a rather generic issue
outside of OpenAM's specific context, so it is unlikely to change
dramatically with version updates. Perhaps the most promising
direction would be to try using MySQL (or some other RDBMS) instead of
LDAP for policies storage. There's already some experimental support
for this in OpenAM.

> As you
> may know, I have tried upgrading several times, but didn't succeed, most
> likely due to a bug in OpenAM 9.0 (see
> https://bugster.forgerock.org/jira/browse/OPENAM-464). We also use an
> external LDAP server (the one feeding the main website), but it might be
> not too powerful.

If you're using it only for user attributes retrieval, it should be
more or less OK. However, if you're using it also for policies
(config) storage, then it must be quite overloaded... OpenAM comes by
default with a built-in instance of OpenDS, which is used by default
for config storage, but I think that it is better to run the policy
(config) LDAP in a separate JVM -- so that's why I've opted for
OpenDJ.

> Again, thank you for investigating the stuff! To make best use of it
> and transition to a more powerful deployment, we will need a way to
> migrate our existing policies (see the above topic on the issue
> tracker).

Yes, the migration is yet another problem... I guess that if this bug
doesn't get fixed reasonably soon, we could be forced to forget about
upgrading and rebuild the policies from scratch (or semi-automatically
with some custom script running through the ssoadm).

Kind regards,
Vedrin

PS: Currently I have 4 GB of RAM allocated to OpenDJ with half of it
dedicated to db cache, another 2 GB to the Tomcat instance hosting
OpenAM, Policy and Ambit2, and yet another 1 GB to the MySQL instance
-- all running on the same server which has 2 processors with a total
of 8 CPU cores, running at 2.67 GHz. However it has a rather slow
(outdated) memory bus, so I'm sure the figures could be improved quite
a lot by using more recent hardware.