Apache refuses to start, with a No space left on device error, despite all partitions being far from full

I have been running into some unusual problems in Apache lately, and thought I would document them.

If you get this error when trying to start Apache, and only a reboot will fix it:

[emerg] (28)No space left on device: Couldn't create accept lock

Run this bash command:

ipcs -s | grep apache | awk ' { print $2 } ' | xargs ipcrm sem

I would also recommend adding this function to the end of the stop function in /etc/init.d/httpd if the problem persists. The cause of the problem seems to be crashed apache forks that refuse to respond to a SIGTERM, and must be killed. The problem is that the semaphores are used up for the Apache user, and that results in the apache startup failing. One possible workaround is reducing the max pages served by a process, but testing hasn't proven this out yet.