2008-02-17 02:45:19

The OpenLDAP way of concurrency

OpenLDAP is a multithreaded implementation of an LDAP server, licensed under a special derivate of a 2-clause BSD license called «OpenLDAP license». It uses LDBM or BerkeleyDB as a storage backend or can alternatively be configured to read its information from an SQL database or other formats (which effectively disables writing to the database). Like for BSDprojects.net, BerkeleyDB is frequently used for user management (authorization, authentification, mail addresses, etc.).

Up to this point, all attempts to upgrade to OpenLDAP's BerkeleyDB backend have failed. There was always the same error: when the database has been filled with some data, any LDAP search request would crash the LDAP server. The stack trace usually ended in some call instruction or some other instruction immediately following a call.

In an answer to NetBSD PR 37997, I was reminded by Christos Zoulas that the problem might actually be no crash at all but an issue with the rlimit facility of NetBSD, so I experimented a bit with the stack limit.

By default, NetBSD imposes a limit of 2 MB on the program stack. However, not even 4 or 8 MB seemed enough. A stack limit of 32 MB however solved the problem entirely.

It seems that OpenLDAP attempts to avoid locking problems between threads by putting all information on the function local stack. This only works well with a low amount of data or with low parallelism. However, in the case of an LDAP server, shared data structures would indeed be desirable.

LDBM itself does not use so much stack space, so it did not trigger the stack space limit. However, with parallel requests being handled by multiple LDAP threads, iit would undoubtedly only be a matter of time.

Let's see how well the 32 MB stack limit works...

Posted by Tonnerre Lombard | Permanent link | File under: programming, general