Process spikes

[Sir Osis of Liver]

This is becoming a chronic problem.  2.0.19 forum in WP website, things are crashing because of spikes in server cpu usage and processes.  Processes especially are running continuously at max.  From host support -

The process "host -W" is found in the following two files of the forum:

/home/ihq7n15g5yh7/public_html/forum/Sources/Subs.php:                  $test = @shell_exec('host -W 1 ' . @escapeshellarg($ip));
/home/ihq7n15g5yh7/public_html/forum/Sources/Subs.php~:                $test = @shell_exec('host -W 1 ' . @escapeshellarg($ip));

The SMF developers would be able to better answer what is the function of this and why do the processes for it keep increasing.

Subs.php -

Code Select Expand


    // Try the Linux host command, perhaps?
    if (!isset($host) && (strpos(strtolower(PHP_OS), 'win') === false || strpos(strtolower(PHP_OS), 'darwin') !== false) && mt_rand(0, 1) == 1)
    {
        if (!isset($modSettings['host_to_dis']))
            $test = @shell_exec('host -W 1 ' . @escapeshellarg($ip));
        else
            $test = @shell_exec('host ' . @escapeshellarg($ip));



[Illori]

have you turned off hostname lookup? if not you should.

[Sir Osis of Liver]

Did not recognize $modSettings['host_to_dis'].  It was enabled, I've disabled it.  Will see what happens.  Thx.

[Arantor]

It keeps increasing because your hosting company doesn't have a DNS proxy that SMF can call to get the hostname.

Personally I'm at the point where I think the entire hostname functionality should be at the very least disabled by default and removed in a future version because the work required to make it work *correctly* vastly outweighs the actual benefits derivable from it.

[shawnb61]

I'm willing to bet that the real problem, once again, is actually bots.

They are totally hyperactive these days.

Have you looked into who is hitting your forum so much?  Analyzed the logs? 

What are you doing for bot mitigation?

[Sir Osis of Liver]

Do ban triggers hit hostname lookup?

[shawnb61]

I believe so, yes.  (Out of town, can't confirm.)

Have you looked into who is hitting your forum so much?  Analyzed the logs? 

What are you doing for bot mitigation?

[Aleksi "Lex" Kilpinen]

Would expect so, since you can ban by hostname too.

[Sir Osis of Liver]

Someone else handles new member screening and bans, and has set up many ban triggers over many years. Forum has been logging 7000-8000 ban errors daily and increasing.  I'm in the process of copying the banned IPs from error log to CIDR range bans in ,htaccess.  Am beginning to think either ban triggers have reached a critical mass, or adding ,htaccess bans has pushed processes over the top.  Will see tomorrow if disabling hostname lookup has any effect.

[vbgamer45]

Bots sometimes can be crazy like shawn said. I had 200k most online one forum i host on Saturday.

[Arantor]

.htaccess bans are done before they ever get to SMF. You want to put bans in there if at all possible.

If HostnameLookups are enabled, any online session that hasn't recently gotten its host name looked up will get looked up. Which for bits means fair chance you'll get spammed with requests because bots have a habit of not carrying sessions properly meaning they end up creating new sessions every request - meaning every request triggers a host name lookup.

[Sir Osis of Liver]

.htaccess bans are done before they ever get to SMF. You want to put bans in there if at all possible.

That's what I've been doing.  Support tells me .htaccess bans should not be affecting processes.  They're way down since I disabled hostname lookups last night.  Still getting some cpu spikes, but I think those are normal.  What I'll have to do is continue adding bans to .htaccess as they're logged in forum, then at some point remove the ban triggers.  I can kill all processes in SSH, that seems to be the only way to clear things when everything starts to crash.

[shawnb61]

The longer term effort is monitoring those spikes & determining what to do about them.

Maybe it's a new bot with a clear useragent.   Maybe it's an IP address range known to be spammy.  Or yet another AI feeder.

Maybe it's Google ramping up for some unknown reason (likely AI feeding).

If you ignore them, they come back.

Once you've drawn their attention, there's no going back...  Monitoring needs to become routine.

I feel sorry for all the smaller sites, with basically users as admins.  It's getting harder & harder for normal non-admin folks to run a site.

[Sir Osis of Liver]

Looks like if I remove the ban triggers and have all IP bans done in .htaccess it should solve the processes problem.  Don't want to do that until I've reduced the ban errors, I'm using those to identify the bad IPs.  Over a thousand errors since last night.

[shawnb61]

My point is you're never really "done".  It will require ongoing monitoring and maintenance.

[shawnb61]

Case in point...  This is my MySQL CPU chart, which spiked last night...

Confirmed to be Googlebot, valid IPs.  Some time yesterday afternoon, Google started crawling me 4-6x the normal crawl rate.

And those scumbuckets discontinued their crawl rate limiting tool as of Jan 2024.  And they do not honor crawl-delay in robots.txt.

This is a few weeks after I blocked GoogleOther in .htaccess...  Which I am convinced is feeding their AI...  And so I am now convinced they just chose another route to feed their AI.

Their site says to limit the crawl rate, you need to start returning status 500 to them. 
https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate

"Don't worry!  We'll slow down if we detect your server is nearing capacity!  TRUST US!!!"
https://www.searchenginejournal.com/google-removing-crawl-rate-limiter-tool-from-search-console/502176/

You cannot view this attachment.

[Sir Osis of Liver]

Don't expect to be done, but by moving IP bans to .htaccess that should clean up the forum.  Processes are flat since I disabled hostname lookups -

You cannot view this attachment.

They were maxed last night, ftp crashed, then cpanel crashed, then website and forum crashed.  That apparently was all from hostname lookups.  I have 1756 ban errors in forum log, less than 24 hours.  Trying to set range bans in .htaccess so I can delete the forum ban triggers, but they're staying way ahead of me. 

[Sir Osis of Liver]

If hostname lookup is disabled are ban errors still logged?  Not seeing any tonight.

[Arantor]

Ban errors are still logged if they get to the forum (all circumstances), but htaccess bans stop them getting that far so it never ever hits SMF's ban system.

[Sir Osis of Liver]

Yeah, I know, but I'm using the forum ban errors to get the rogue IPs and add them to .htaccess.  Only two IPs logged since last night, it's been over 1,000-2,000 per day past couple weeks, mostly overnight.  Stopped rather suddenly.  Processes are still flat, just a few cpu usage spikes, but support tells me that's normal.