Old Drupal Semaphore Lock Could Bring Down Your Entire Site
One day on our company's Intranet we saved a view and took down the entire website. We tried a few things but since it was a Production site we had to restore to a restore from a nightly backup.
New Lesson Learned
After tearing apart the database I eventually found the issue was an old lock was still in the database from about a month ago. And when a view is saved, it triggers a menu rebuild. We'll turns out this old lock was preventing the menu rebuild from ever happening. So the entire site went into a white screen of death.
What are these locks?
Turns out its an age old problem, to prevent two actions from happening at once, the first process will engage a lock until it finishes. This is what this locking table is used for, so actions such as rebuilding the sites menu don't happen at the same time. This table will normally be empty and only for a very short period of time should a 'lock' record exist here.
The problem is however if some process fails and a lock is never released, no other process can ever precede. This is what happened in our case, our PHP errors are hidden on production, so everyone was getting a white screen of death.
See the issue I created for updates (if there are any) /www.drupal.org/node/2917643
The Fix
One handy snippet of code in my Drupal toolkit is now the below code. It will remove any old locks in the database and fairly safely in my opinion. You could just drop the entire table, but I prefer to use a scalpel instead of a hammer if you know what I mean.
<?php
// This is to fix orphaned locks in the semaphore table
// UPDATE the below variable to match the name of the lock
$lockToDelete = 'menu_rebuild';
// Only locks that have expire dates older than the following
$locksOlderThan = strtotime("-2 day");
// no need to edit beyond here
$result = db_delete('semaphore')
->condition('name', $lockToDelete)
->condition('expire', $locksOlderThan, '<')
->execute();
if($result) {
echo $lockToDelete.' lock removed';
}else{
echo $lockToDelete.' LOCK DOES NOT EXIST';
}
?>