Oh boy...

[Scott Greczkowski]

So as everyone may know we have been having strange issues over the past few days...

I have been monitoring the server closely trying to squish everything... and today I have been monitoring the webserver (apache) and php error logs for issues.

It has been clean up until about an hour and a half ago.

Went to do something and came back and pulled up the screens monitoring the system and we went from no errors to OH MY GOD.

Here is what I saw...

[Wed May 01 12:56:12.256151 2024] [security2:error] [pid 323107:tid 23332770461440] [client 20.2.80.25:59696] [client 20.2.80.25] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/RESPONSE-980-CORRELATION.conf"] [line "37"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 15 - SQLI=5,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=5,HTTP=0,SESS=0): SQL Injection Attack Detected via libinjection"][tag "event-correlation"] [hostname "www.satelliteguys.us"] [uri "/xen/whats-new/posts/"] [unique_id "ZjJ0LKJEHT3noqj4-42QIQAAABc"]
[Wed May 01 12:56:12.282728 2024] [security2:error] [pid 323107:tid 23332774663936] [client 20.2.80.25:59697] [client 20.2.80.25] ModSecurity: Warning. Matched phrase "sqlmap" at REQUEST_HEADERS:User-Agent. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-913-SCANNER-DETECTION.conf"] [line "33"] [id "913100"] [rev "2"] [msg "Found User-Agent associated with security scanner"] [data "Matched Data: sqlmap found within REQUEST_HEADERS:User-Agent: sqlmap/1.8.4.7#dev (https://sqlmap.org)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-reputation-scanner"] [tag "OWASP_CRS/AUTOMATION/SECURITY_SCANNER"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "www.satelliteguys.us"] [uri "/xen/threads/dish-hopper-and-caller-id.372482/"] [unique_id "ZjJ0LKJEHT3noqj4-42QIgAAABU"]

But THOUSANDS of them. All from the same IP in Hong Kong.

I had to block the IP and I also blocked Hong Kong.

Errors have stopped again. And server load dropped greatly.

I did apply some more software to help with security but I need to reboot the server for it to take affect. (Is it affect or effect, I always get those two confused.)

I will be rebooting in a few moments, the site will be unavailable for about 20 seconds while we reboot. (yeah its FAST!)

If its not one thing its another.

Just keeping you guys informed of whats going on.

 

[Scott Greczkowski]

And we rebooted... (If you got an Oops that is why)

 

[mikew]

nice work Scott!

 

[telstar_1]

No hiccups here. "affect" is influencing action (as in, "will this affect the server?"), "effect" is resulting condition. Affects of a medication may include side effects. So "take effect".

Oop, was going to say no hiccups, but then a delayed response to edit post.

 

[navychop]

So Scott has gotten the attention of the CCP?

 

[arlo]

At times a bit laggy opening the page. And delays opening posts. If this server is in the CONUS maybe block that whole region.

 

[Stuart Sweet]

Well done sir. I have had to do much the same at The Solid Signal Blog.

 

[Scott Greczkowski]

I am working on this security software in the background.

But at least this is gone. Also PHP and Apache error logs are now showing no issues again.

 

[Scott Greczkowski]

ModSecurity must flush the blocked database after a half hour. As it came back again in full force.

It is now blocked in the IPtables.

If I need to then I may have to install Firewall software to better manage things... but I don't want to possibly slow the server down by having CSF running all the time.

 

[Scott Greczkowski]

Some good news...

On our new server a lot of email going out is being blocked by AT&T. I worked with them the past 2 days and have been able to get our IP address off of their blacklist, so those users who have AT&T provided emails will again be receiving your SatelliteGuys notifications again within the next 48 hours.

 

[Foxbat]

But THOUSANDS of them. All from the same IP in Hong Kong.

I had to block the IP and I also blocked Hong Kong.

I guess I made too many Polestar posts in the EVs Forum…

 

[KE4EST]

Glad things are back and running as they should.

 

[a33]

... but I need to reboot the server for it to take affect. (Is it affect or effect, I always get those two confused.)

As a non-native speaker I would say:
The changes take effect. After that, the HongKong activity no longer affects the running of our website.

I always have to think: is it allways or always?
Is it than or then?
Is it choose or chose, loose or lose?
Boy, language is difficult sometimes!


Scott, thanks for the good work you're doing!

Greetz,
A33

 

[Scott Greczkowski]

Ok more hopeful good news.

I have been having a hard time working on the site at home as my 2019 MacBook Pro is having issues. Apple wants over $700 to fix and I don’t have a spare $700 at the moment. So with the issues with the screen I have been limited sometime the screen works and sometimes it don’t. And when it does not I have to work on things on my iPhone.

Anyways at work I am blessed to have three screens on my computer and today was the first time in a long time I was able to sit in my office all day and work on things. And today I took full advantage of the three screens.

On the screen on the left I had ssh open and was constantly monitoring the Apache Web Server error log…. On the right monitor I had another ssh window opened and was monitoring the PHP error log.

And in the center I was doing my work… work. (No SatelliteGuys is not my job, believe it or not I don’t get paid to run this place…. I did at one time but that’s a different story.

So in the center monitor I was doing my real job work but had a third ssh window open so if I saw anything strange happen in the left or right screen I could immediately go into the ssh window and start working on things.

As you saw above besides getting hammered by someone on Hong Kong trying to hack the site I did finally notice two things pop up.

First I saw an error message about workers_2 pop up on the screen. I had no idea what h2_workers was so I researched and found it has something to do with the httpd2 protocol. A week or two ago I added httpd2 support as it was recommended by one of the site testing sites.

I couldn’t really find out much about the error though, which frustrated me. But I sat and watched the error logs as I worked.

Then it happened.

At the time I was browsing SatelliteGuys and all a sudden it paused. I looked at my left screen and saw an error message about running out of MaxRequestWorkers appear in the screen.

I searched for that and httpd2 and boom had a solution!

When we moved to the new server I did tune our Apache server with the Apache optimizer wizard and used the suggestions it gave me for settings.

Did more searching and found out the tuning optimizer does not take into effect (did I use it correctly that time?) httpd2 workers. So in reality the Apache server was crashing as it was maxing out based on its settings.

Some more searching and I found someone with a similar spec server as our and they posted their Apache settings for http2 enabled systems. So I copied their settings and applied them here.

And so far so good. In fact the settings brought down our average server load from around 2.30 and now we are doing around 0.80!

I never gave http2 any thought. I was just told to enable it in Apache but there are no direct settings for it.

Is this the final fix? I sure hope so.

I will be back in the office all day tomorrow as my coworker will be out again, so I will be watching the error logs again just in case.

But I am hopeful that I may have finally got it

Enjoy the speed and than you for your patience.


Sent from my iPhone using Tapatalk

 

[Scott Greczkowski]

BTW I also removed a bunch of things we were no longer using, or no longer worked...

For example, the automated Twitter Posting, which would post all new threads to our SatelliteGuys Twitter account. When Elon took over they changed the API to prevent automated posting like this. (I guess they want you to pay to do that now... no thanks!)

I also removed the ChatGPT feature which would suggest new, more SEO friendly titles for new topics to staff. With that it would make topic names more search friendly. But it didn't get much use.

The above two add ons would take time to call out to their respective servers whenever someone posted something... and could (and sometimes did) slow down when posting something.

I removed the Chat Feature that the Cutting Edge guys were using but stopped using as it was just easier to post messages in the CE forums directly.

I removed the AMP support. AMP was a way to make the site more friendly for mobile phones. However phones have advanced so far (as has the forum software) so that software was no longer necessary. In the past Google would rank your site better if you supported AMP, but they no longer do. Not to mention there was an annual cost for the AMP service which submitted the posts to Google.

With these things I am hoping to give you the fastest SatelliteGuys experience possible.

Thanks for being here.

 

[navychop]

would post all new threads to our SatelliteGuys Twitter account

“ALL?”

Might I safely assume Pub postings did not go there?

 

[Scott Greczkowski]

“ALL?”

Might I safely assume Pub postings did not go there?

No only new topics in the DISH, DIRECTV and FTA areas.

 

[dweber]

I have been having a hard time working on the site at home as my 2019 MacBook Pro is having issues. Apple wants over $700 to fix and I don’t have a spare $700 at the moment. So with the issues with the screen I have been limited sometime the screen works and sometimes it don’t. And when it does not I have to work on things on my IPhone.
Sent from my iPhone using Tapatalk

Scott,

You put in a lot of time and effort keeping the site running smoothly. So if you need to repair your laptop or buy a new one I am sure that many of your loyal users such as myself would contribute. You should have the tools you need to do the job.
David Weber


Sent from my iPhone using Tapatalk

 

[Scott Greczkowski]

I will get it fixed eventually. Or maybe go for a new M3 MacBook Air. This one is a 2019 and has served me well.

Just have a bunch of things all going on here at home so money is tight at the moment... Thanks!

 

[charlesrshell]

Maybe it's time for a donation maraton, Scott.