Apache: No space left on device: Cannot create SSLMutex

Today, for the first time on one of our servers, I was unable to restart Apache. Running a configtest showed no errors, so something else was up. Looking in the Apache error logs, there was a line: "No space left on device: Cannot create SSLMutex".

If you ever get this, chances are you are suffering from Apache leaving a bunch of stray semaphore sets lying around after an attempted restart of apache. A semaphore in simple terms, is a lock on an operation. The fix, with many thanks to Carlos Rivero (his site is dead now) solved my problem instantly. Note that his solution does not fix the root of the problem, rather it simply gets you going again.

In terminal do the following using the ipcs app, which according to the man pages "ipcs - report XSI interprocess communication facilities status" . The -s flag will show a list of semaphores sets:

ipcs -s | grep apache

Most likely you will see a fairly large list here. You need too, and it is safe too, have these deleted. The following command will again do the trick:

ipcs -s | grep apache | awk ' { print $2 } ' | xargs -n 1 ipcrm -s

Note: If your apache is running as nobody or another user, be sure to substitute that other user in place of  apache above.

Cannot create SSLMutex solution

At the heart of the problem, is most likely a poorly configured Apache server. By default, SSMutex is configured to the default setting, as it was on this one server of ours. If you read the Apache.org pages for mod_ssl configuration, they have this to say about the default setting:

The following Mutex types are available: 

none | no

This is the default where no Mutex is used at all. Use it at your own risk. But because currently the Mutex is mainly used for synchronizing write access to the SSL Session Cache you can live without it as long as you accept a sometimes garbled Session Cache. So it's not recommended to leave this the default. Instead configure a real Mutex.

There are of course optional configuration settings. At the very least, it is suggested that you set SSLMutex to sem, which will let Apache choose which SSLMutex type to use.

You will most likely find this setting in the ssl.conf file located at /etc/httpd/conf.d.