Recently we encountered some strange issues with httpd on some of our CentOS 7 machines during my current project.

Through our pipeline we restart httpd several times which sometimes leads to this error:

Apache: [error] (28)No space left on device

After some research we found out the semaphores were all being used blocking httpd daemon to restart.

The list of semaphores can be fetched by issuing

# ipcs -st

------ Semaphore Operation/Change Times --------
semid    owner      last-op                    last-changed
753664   apache      Not set                    Mon Feb 17 20:20:47 2020
786433   apache      Not set                    Mon Feb 17 20:20:47 2020
720898 …

I decided to bootstrap my new machine with btrfs as filesystem instead of ext4 LVM volumes. By following the excellent arch-wiki btrfs page I successfully crafted a base system with sub volumes, limited on size and snapshots enabled.

Everything went fine, installed all the other stuff I needed, pulled in my data and was ready to go.

Obviously on that very moment disaster happened.. Due to an unexpected interrupt the journal went corrupt. When trying to boot I got stuck right after decrypted the disk failing to mount my btrfs root volume.

Uncool, unpleasant, .. I almost got insane..

So back …