Been monitoring the system and found another issue with ElasticSearch, which is the search engine we use here. It was trying to do its monthly rotation of its logs and was running out of memory. I think I have that fixed now as well.
Well my fix worked for overnight... sort of. It kept things running however the Apache server still kept locking up giving a Connection Refused message a few times overnight. Luckily with the change I made it restarted the Apache server a few seconds after the Connection Refused issues happened. The log show the ModSecurity seems to be the one causing it.
I am in the office early today, had a vendor in doing some work here and I had to let them in and meet with them, so now I am here at my desk and I plan to work on the server again and hopefully fix this mess once and for all.
Because of this the site may be unavailable for a few minutes at a time. I will be recompiling and upgrading our Apache web server, as well as ModSecurity.
I would rather not put us behind a Sucuri setup again as that caused issues itself, not to mention the expense, but if its something we need to do then I will have to do it.
Ok one issue I kept seeing in the logs was issues with ElasticSearch, which is our search engine we use here on SatelliteGuys. We were running version 6.8.19 which I have removed and have installed the latest version which is 7.15.0.
The search index is still building as I type this almost 3 million out of the 4.5 million posts have been indexed. As you can imagine this is both CPU and Memory intensive as its doing its indexing. But it should be done shortly within a half hour.
As I am going through the logs and looking at the processes I can see other things I can adjust as well to help with performance.
Thank god for Google. Its helping me find, identify and fix some of these things.
The issue is the attack is trying to use the Xenforo proxy.php to pull bad images from other sites. Mod security sees the proxy.php of our server being the bad guy as its the one that is trying (and failing) to grab the bad images... so it keeps trying to ban our server from the server.
I have been tweaking things all morning, updated APache, Php and ModSecurity, upgraded ElasticSearch and have just removed my cron job which was restarting the services every 45 minutes.... let see how this goes.