SatelliteGuys was offline this morning from around 3:30 am until about 5 am.
When I woke up we were offline. The server was throwing database errors and normal systems were not functioning.
I restarted the machine which normally fixes things however this did not. So I logged in to the machine via VNC to find the issue. It took a couple of minutes but I found it... we were out of drive space. Since it was the first of the month, I figured the monthly backup had issues. So I deleted that and a bunch of older backups and opened up some space. But still services would still start and our drive space was still 94% used. I then did a drive search for files on the server that were bigger than 1 gig and found a 460 gig error_log file in our forum software directory. I took a look at it and it was one error that started February 16th of last year and repeated and repeated on every page load here from our CleanTalk security plugin. Yeah every time a page was loaded here that plugin wrote to the error_log file. I disabled the plugin and the errors stopped. I cleared that log file and now we are only using 43% of our drive space.
I restarted the server again and all services came fully back online again. (phew!)
I alerted the CleanTalk people of the issue and it appears their plugin does not create a needed table to fully function properly, without that table the forum software throws an error into that error_log file for every page load.
So now I got a ticket in with them and they are checking why that table was not created. This may be happening on other Xenforo forum sites but no one noticed it yet because this error_log is not the normal one that is written elsewhere.
Upon the advice of CleanTalk I have disabled their plugin which has stopped the errors in the log. And CleanTalk is working on the issue.
Yesterday I upgraded the system from PHP 8.3 to PHP 8.4 and thought maybe the swap had something to do with it. But it did not, thankfully.
So for you early birds, sorry for the down time.
When I woke up we were offline. The server was throwing database errors and normal systems were not functioning.
I restarted the machine which normally fixes things however this did not. So I logged in to the machine via VNC to find the issue. It took a couple of minutes but I found it... we were out of drive space. Since it was the first of the month, I figured the monthly backup had issues. So I deleted that and a bunch of older backups and opened up some space. But still services would still start and our drive space was still 94% used. I then did a drive search for files on the server that were bigger than 1 gig and found a 460 gig error_log file in our forum software directory. I took a look at it and it was one error that started February 16th of last year and repeated and repeated on every page load here from our CleanTalk security plugin. Yeah every time a page was loaded here that plugin wrote to the error_log file. I disabled the plugin and the errors stopped. I cleared that log file and now we are only using 43% of our drive space.
I restarted the server again and all services came fully back online again. (phew!)
I alerted the CleanTalk people of the issue and it appears their plugin does not create a needed table to fully function properly, without that table the forum software throws an error into that error_log file for every page load.
So now I got a ticket in with them and they are checking why that table was not created. This may be happening on other Xenforo forum sites but no one noticed it yet because this error_log is not the normal one that is written elsewhere.
Upon the advice of CleanTalk I have disabled their plugin which has stopped the errors in the log. And CleanTalk is working on the issue.
Yesterday I upgraded the system from PHP 8.3 to PHP 8.4 and thought maybe the swap had something to do with it. But it did not, thankfully.
So for you early birds, sorry for the down time.
