New mod_mono/xsp in 1.2.5

Unlike stated in the previous post, the changes described there will make it into Mono 1.2.5 as they turned out to fix more bugs they were intended to (wish me more fixes like that!). mod_mono and xsp were copied from svn trunk and are part of the 1.2.5 Preview 4 release.

New mod_mono and xsp developments

It's been a while since I blogged. The reason for this is simple - all the work I've done on ASP.NET in the past weeks was bugfixing and most of it wasn't blog-worthy. But recently a couple of important bugs were closed which resulted in several nice feature additions to mod_mono and XSP.

One of the important bug fixes was a modification to Mono assembly shadow copying code. We had a race condition that showed up when one thread was shutting down an application domain and another would be starting a new one. The latter would overwrite the shadowed assemblies which would cause the former to throw an exception about invalid opcodes present in the assembly. The fix was to introduce an internal AppDomain serial number which would be unique for each new domain. The serial number is then used in generating the names of shadow copy directories. That way, even for the same application, new AppDomain created by the same application, with the same root directory, setup etc. (e.g. an ASP.NET hosting domain) saves its shadow assemblies in another target directory.

Another bug was reported to happen on ASP.NET sites powered by Apache/mod_mono. The problem was another race condition which occurred in the request startup/teardown sequence inside the Mono.WebServer assembly (shared by both xsp and mod_mono backends). The race was in the assignment and release of request ids which sometimes ended up reused even though the original request with that id wasn't done yet. The solution was to introduce a few lock statements in the code as well as an delegate way of releasing the request id in the code using Mono.WebServer.

The third bug was reported by a Mono user running a very busy website which triggerred the behavior described in the paragraph above, but also exhibited problems with memory consumption for the long-running mod_mono backends. We haven't solved the memory consumption problem yet (it's probably related to the GC), but as part of workaround for the issue we implemented set of options and features for mod_mono which make sites with such problems more robust. I will enumerate and shortly describe the features below.

Startup optimization

mod_mono used to start the backends in the child init hook, which is called after apache forks a new child and has already switched from the root user to the target user/group named by the User/Group directives, respectively. The problem with this solution was that if we had 5 child processes, then 5 copies of mod_mono would launch that many backends simultaneously. Eventually all but one backends would die, but the initial impact on performance would be quite severe. New code moved the backend launch to a server init hook function which runs when apache is still executing as root (i.e. in the main apache process). The code looks up the User/Group directives and switches temporarily to the defined uid/gid in order to launch the backend. That way the vhost backends are launched only once.

To lookup User/Group properly, mod_mono must be loaded after the two directives in the apache config.

The dashboard

The dashboard is a shared memory area which stores information about all instances of mod_mono backends. Each virtual host has its own dashboard, so the performance impact of one vhost's dashboard locking is none for the other vhosts present in the server. The dashboard is used by mod_mono to synchronize (re)starts of the backends so that only one copy of mod_mono stops or starts the backend (Apache can run several sub-processes each of which loads its own copy of mod_mono). The dashboard is also used to store accounting information used by the auto-restart feature described below.
In order to synchronize access to the dashboard, mod_mono must be able to acquire a lock before accessing the data. By default it uses whatever the underlying Apache runtime chooses, but there are cases when it might fail on some systems. Should such event occur, you can use the MOD_MONO_LOCKING_MECHANISM environment variable to specify one of the available mechanisms. For the list of currently recognized ones, please consult the mod_mono manual page (man mod_mono)

Auto-restart

If your mod_mono site uses patterns which make Mono allocate large amounts of RAM, and the site is busy enough for the Mono garbage collector not being able to free enough memory on time, you can configure mod_mono to restart the backend(s) automatically based on two criterions: backend uptime or the number of requests served. The feature is configured on a per virtual host basis. Configuration is done by means of three Apache configuration directives:

• MonoAutoRestartMode. This directive selects the restart mode and can take one of three values: None (default, no automatic restarts are done), Requests (restart after the given number of requests) and Time (restart after the backend has been up for the specified period of time)


• MonoAutoRestartRequests. Specifies the number of requests after which the backend should be restarted. Defaults to 10000 requests.


• MonoAutoRestartTime. Specifies the minimum uptime after which the backend will be restarted. Takes values in the generic format DD[:HH[:MM[:SS]]]. Defaults to 12h (00:12:00:00)

If your site stores data in the ASP.NET session storage, you must make sure that you use out of process session storage, since the in-process data wil l be destroyed when the restart occurs.

Lamentably, the changes described above will not be part of the upcoming 1.2.5 release of Mono. You need to use the svn trunk versions of mod_mono and xsp in order to take advantage of them.