(888) 555-5555

Stuck Websites

pole
Home » Stuck Websites

Stuck Websites

Stuck Websites Fixing Server

Of late my server which has 192GB or ram and rarely goes over 20% utilization seems to get “stuck” could that be caused by AI robots scraping the sites on there 2 sites each have 8500 pages.

Yes, AI crawlers and automated scraping bots can easily cause your server to get “stuck” even with 192 GB of RAM sitting mostly unused.

A machine with 192 GB of RAM running under 20% utilization rules out memory exhaustion (OOM), but web servers rarely freeze because of raw RAM. Instead, unthrottled scrapers cause concurrency exhaustion and queue bottlenecks.

With 8,500 pages across two sites, an aggressive AI scraper (or multiple bots scraping simultaneously) can request hundreds of un-cached URLs per second.

Why the Server “Freezes” While RAM Stays Empty

  1. Worker / Thread Pool Depletion Web servers (like Nginx, Apache, or LiteSpeed) and application backends (such as PHP-FPM, Node.js, or Python WSGI) allocate a fixed pool of worker processes or threads. If bots trigger 150 concurrent dynamic requests, all worker slots fill instantly. New requests (including yours) wait in the TCP listen backlog until they time out. The server looks dead from the outside, even though RAM usage barely budged.
  2. Database Connection Pool & Query Lockups Each scraped page often runs multiple database queries. If scrapers bypass the cache by adding random query strings, sorting parameters, or paginated archives, MySQL/PostgreSQL will quickly hit max_connections or lock up on slow table scans.
  3. Storage I/O Bottlenecks (Disk Wait) If scraping forces heavy disk reads or frequent write-logging, the CPU enters high iowait (waiting on storage I/O). The CPU utilization percentage may still look low because cores are idling while waiting for the disk to respond.

How to Diagnose and Resolve the Issue

Step 1: Check Active Scraping in Real Time

Run these commands in your SSH terminal while the server is experiencing a slowdown:

Check top IP addresses making connections right now

netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n | tail -n 20

Identify the top user-agents hitting your web server logs (e.g., Nginx)

tail -n 10000 /var/log/nginx/access.log | awk -F” ‘{print $6}’ | sort | uniq -c | sort -rn | head -20

Look for common AI/crawler User-Agents: Bytespider, GPTBot, ClaudeBot, CCBot, anthropic-ai, or generic Python/cURL clients.

Step 2: Check Process Pool Backlogs

  • If using PHP-FPM, check your status page or log for warnings like:[pool www] server reached pm.max_children setting, consider raising it.
  • Check database connection saturation:

mysqladmin status

or in MySQL:

SHOW PROCESSLIST;

Step 3: Block or Throttle Aggressive AI Bots

  1. At the Web Server / Firewall Layer (Fastest Fix): Block well-known aggressive scrapers that ignore crawl-delay (e.g., Bytespider is notorious for ignoring polite crawling guidelines). In Nginx:

if ($http_user_agent ~* (Bytespider|ClaudeBot|GPTBot|CCBot|PerplexityBot)) {
return 429;
}

Set Rate Limiting: Configure connection or request rate-limiting per IP (e.g., limit_req_zone in Nginx) to ensure no single IP can make more than 5–10 requests per second.

Use a Reverse Proxy / CDN (Cloudflare):Enabling Cloudflare’s free “Bot Fight Mode” or turning on Web Application Firewall (WAF) rate limiting handles AI crawlers at the edge before they touch your server’s socket pool.

OK my server is Linux running php and almost all the websites are WordPress could I block some of these robots. Clearly I want the sites to be indexed by Google and Bing and reputable search engines but not ones which are just AI.

You can block aggressive AI scrapers without touching search engine crawlers like Google and Bing.

Legitimate search engines identify themselves as Googlebot and bingbot, whereas AI scrapers use distinct identifiers (such as Bytespider, GPTBot, ClaudeBot, CCBot, Diffbot, and PerplexityBot).

Because WordPress loads the entire PHP and MySQL stack for every uncached dynamic request, blocking bots at the web server layer (Nginx or Apache) is essential. Dropping them before PHP or WordPress ever executes protects your worker pool and prevents the server from freezing.

Method 1: Block at Web Server Level (Recommended)

Identify whether your server uses Nginx or Apache (nginx -v or httpd -v / apache2 -v).

If Running Nginx

In your Nginx site configuration (e.g., /etc/nginx/sites-available/your-site.conf or inside the http {} block in /etc/nginx/nginx.conf), add a map or regex check to immediately drop AI bots with code 403 (Forbidden) or 444 (Nginx drop connection without sending data):

Add inside the server { … } block for your WordPress sites

if ($http_user_agent ~* (Bytespider|ClaudeBot|GPTBot|CCBot|PerplexityBot|Diffbot|Amazonbot|Applebot-Extended|cohere-ai)) {
return 403;
}

  • Verification: Test the configuration with sudo nginx -t (it must report syntax is ok), then reload with sudo systemctl reload nginx. Run curl -I -A "GPTBot" [https://yourdomain.com](https://yourdomain.com) in your terminal — you should receive an immediate HTTP/2 403. Testing with curl -I -A "Googlebot" [https://yourdomain.com](https://yourdomain.com) should return 200 OK.

If Running Apache (.htaccess)

If your sites run behind Apache, add the following to the top of your WordPress .htaccess file before the # BEGIN WordPress block:

RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (Bytespider|ClaudeBot|GPTBot|CCBot|PerplexityBot|Diffbot|Amazonbot|Applebot-Extended|cohere-ai) [NC] RewriteRule .* – [F,L]

  • Verification: Run curl -I -A "Bytespider" [https://yourdomain.com](https://yourdomain.com) — it should return 403 Forbidden without triggering WordPress or MySQL.

Method 2: Update robots.txt for Compliant Crawlers

Well-behaved AI bots (like OpenAI’s GPTBot or Anthropic’s ClaudeBot) check robots.txt before crawling. Adding disallow rules stops them politely without affecting Googlebot or Bingbot:

Add this to your root robots.txt file (or manage it via your SEO plugin like Yoast or Rank Math):

Allow search engines

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Block AI training bots

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Diffbot
Disallow: /

  • Verification: Visit [https://yourdomain.com/robots.txt](https://yourdomain.com/robots.txt) in a web browser to confirm the directives are live and visible.

Note: Scrapers like Bytespider often ignore robots.txt directives. Do not rely on robots.txt alone; the server-level block in Method 1 or a CDN firewall is required to stop them from consuming server resources.

Method 3: Protect at the DNS / Edge (Cloudflare)

If your domains route through Cloudflare:

  1. Navigate to Security > Bots in the Cloudflare dashboard.
  2. Toggle on “Bot Fight Mode” (or turn on “Block AI scrapers and crawlers” under the AI bot controls).
  3. Cloudflare automatically maintains a verified search engine list (Google, Bing, Yahoo, DuckDuckGo) and lets them pass while blocking or challenging known AI scrapers before any request reaches your 192GB server.

Well it is 8.45pm and I am feeling tired so the option 2 robots file edit feels like a start for today and then look into the other options in the coming days.

That is a sensible, zero-risk move for tonight—editing robots.txt cannot break your site, won’t crash PHP, and immediately tells compliant AI crawlers to back off.

Here is the exact block of text to add, followed by the fastest ways to update it depending on how you prefer to access your site.

The Content to Add

Copy and paste this block into your robots.txt:

———————————————–

Allow Search Engines (SEO)

———————————————–

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

———————————————–

Block AI Scrapers & Data Harvesters

———————————————–

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Cohere-ai
Disallow: /

Choose the Quickest Way to Update It

Option A: Inside WordPress Admin (If using Yoast or Rank Math)

  1. Yoast SEO: Go to Yoast SEO > Tools > File editor. Paste the lines into the robots.txt box and click Save changes to robots.txt.
  2. Rank Math: Go to Rank Math > General Settings > Edit robots.txt. Paste the lines and click Save Changes.

Option B: Via SSH / Terminal

If your WordPress installation lives in a standard web directory (e.g., /var/www/html or /var/www/yourdomain.com):

  1. Navigate to the site’s document root:

cd /var/www/yourdomain.com

Open or create the file with nano:

Bash

nano robots.tx
  1. Paste the snippet above.
  2. Press Ctrl + O, then Enter to save, and Ctrl + X to exit.

How to Verify It Worked

Open your web browser and visit:

https://yourdomain.com/robots.txt

  • Expected Result: You should see the plain text with the Disallow: / directives for GPTBot, ClaudeBot, and the other scrapers. If it appears, the update is live.

Get some rest tonight. When you’re ready in the coming days, we can tackle the server-level firewall rules or Cloudflare settings to handle the stubborn scrapers that ignore this file.

this is one of the huge sites https://verniersprings.com/robots.txt and this is the robots.txt file – any comments

The robots.txt file is structured correctly in intent, but there is one major technical detail to watch out for: standard robots.txt rules are case-sensitive or group-specific for certain bot user-agents, and text editors can occasionally insert invisible formatting issues.

Here is a quick review of what to check to make sure it functions as intended:

1. Check the Syntax Structure

Make sure your file adheres to the standard spacing where every distinct user-agent block is separated by a blank line, like this:

———————————————–

1. Rules for All Standard Crawlers

———————————————–

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/uploads/wc-logs/
Disallow: /wp-content/uploads/woocommerce_transient_files/
Disallow: /wp-content/uploads/woocommerce_uploads/
Disallow: /wp-content/uploads/wpo/wpo-plugins-tables-list.json
Disallow: /?add-to-cart=
Disallow: /?*add-to-cart=

———————————————–

2. Allow Search Engines (SEO)

———————————————–

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

———————————————–

3. Block AI Scrapers & Data Harvesters

———————————————–

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Cohere-ai
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Google-Extended
Disallow: /

———————————————–

Sitemap

———————————————–

Sitemap: https://verniersprings.com/sitemap_index.xml

This is the other website with 8500 pages the colour scheme is slightly different and the prices but otherwise they are virtually the same. Can you adjust the most recent txt file to work for the domain https://shop.daviessprings.com/ please

———————————————–

1. Rules for All Standard Crawlers

———————————————–

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/uploads/wc-logs/
Disallow: /wp-content/uploads/woocommerce_transient_files/
Disallow: /wp-content/uploads/woocommerce_uploads/
Disallow: /wp-content/uploads/wpo/wpo-plugins-tables-list.json
Disallow: /?add-to-cart=
Disallow: /?*add-to-cart=

———————————————–

2. Allow Search Engines (SEO)

———————————————–

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

———————————————–

3. Block AI Scrapers & Data Harvesters

———————————————–

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Cohere-ai
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Google-Extended
Disallow: /

———————————————–

Sitemap

———————————————–

Sitemap: https://shop.daviessprings.com/sitemap_index.xml

Individually, a 20-to-50-page site is not the primary driver of a massive queue backup, but they can still contribute to server lockups depending on how bots interact with them.

Here is how small WordPress sites interact with server load, when they cause problems, and how to manage them:

Why Small Sites Usually Aren’t the Primary Problem

  • Limited Link Traversal: An AI crawler scraping an 8,500-page catalog makes tens of thousands of requests as it follows pagination, category links, product variations, and filters. On a 20-page site, a bot finishes crawling in a few seconds and moves on.
  • Effective Full-Page Caching: On small brochure or portfolio sites, caching plugins (like WP Super Cache, LiteSpeed, or WP Rocket) keep almost every page in memory or pre-rendered HTML. Serving a pre-rendered static page takes almost 0 CPU and negligible worker time.

When Small Sites Do Cause Server Lockups

  1. Cumulative Traffic Across Many Sites If you host 20 or 30 small sites on the same server, and multiple crawlers hit several of them at the same time, the collective concurrency can quickly saturate your PHP worker pool.
  2. Dynamic Endpoint Probing (Bypassing Cache) Scrapers don’t just request standard pages. They often hammer uncached WordPress endpoints:
    • WordPress search queries: /?s=keyword (bypasses page cache and runs heavy SQL LIKE queries).
    • Comment/form endpoints: /wp-comments-post.php.
    • Feed endpoints: /feed/, /feed/atom/.
    • XML-RPC / Login attacks: /xmlrpc.php and /wp-login.php.
  3. No Page Caching Enabled If page caching is turned off, every single bot hit forces PHP to bootstrap WordPress, load active plugins, and run 25–50 database queries. Fifty rapid requests to a 20-page site will still tie up 50 PHP-FPM workers simultaneously.

Do You Need to Update robots.txt on All the Small Sites?

  • Priority Order: Your two 8,500-page sites were by far the biggest risk factors because of their crawl depth. Updating those two resolves the vast majority of crawler load.
  • For the Small Sites: You do not need to urgently rush to update robots.txt on every single small site tonight.
  • The Long-Term Fix: Rather than managing dozens of individual robots.txt files across all your small sites, blocking AI scrapers at the web server level (Nginx/Apache configuration) or via a CDN/WAF (like Cloudflare) is far more effective. A single server-level rule protects every existing site—and any new site you add—automatically.
back to previous page