All websites are plagued by bots. Some are good bots. Some are bad bots. Some are damn right malicious bots. Here are few tips to help WordPress websites defend themselves from bad bots and control the flow of good bots.
The Do’s
- Configure the web server
- Use Wordfence.
- Use a cache plugin.
- Enable page caching by browsers
- Use an SEO plugin
- Use a good host.
- Use Akismet
- Use Jetpack Protect
- Use Jetpack’s Photon CDN
The Don’ts
- Do not ping content syndication services unnecessarily
- Do not ping search engines unnecessarily
- Don’t worry about robots.txt
- Do not spam other websites
- Do not upset hackers, teens or fanatics
The Reasons
Confirm your web server is configured to use the latest version of PHP available to the server (the latest release is PHP 7). Check that PHP is optimally configured for WordPress. Tell the web server how to handle requests for missing files. This will improve the efficiency of the server.
Wordfence will provide a firewall. Look through the Wordfence settings page to configure Wordfence properly. Make sure you Disable ‘Live Traffic Monitor’ because the monitor will fill up your database with unnecessary data. Other WordPress firewalls exist. Try Sucuri, with the Firewall Addon, if Wordfence isn’t your cup of tea, or search the WordPress plugin repository for other Firewall plugins.
Akismet and Jetpack Protect defend WordPress sites against spam bots and brute force attacks. Akismet ships with WordPress and Jetpack can be installed from the WordPress plugin repository.
A cache plugin will let your server be lazy and do less work when your website is visited.
A cache will reduce the amount of work your website performs when visited and so prevents server, so website, slowdowns. Comet Cache Pro is my favorite cache plugin and has been my favorite for a few years. Wordfence has a Performance Cache feature which works well but sometimes fails to refresh expired documents in the cache.
PHP 5.6 and PHP 7 both use the ACPu memory cache. ACPu, and other memory caches like Opcache, memcache and memcached, allow the server to store the results of PHP processes and database queries in memory for quick access. This means less server work so faster websites. Configure the server to use memory caching but be careful not to overload the server by using more RAM than the server can spare you. Comet Cache Pro monitors and clears the Opcache cache.
Add a few browser cache enabling directives to your web server’s .htaccess file.
A good SEO plugin like Yoast SEO will have features that let you manage the rate pages are indexed and crawled by bots. Website speed can grind to a halt and servers have been known to crash when search bots crawl pages too aggressively. Use a good SEO plugin to control the pages that are indexed and to control the frequency of crawls by bots. Yoast SEO is well maintained and stays up to date with SEO best practices. Yoast provides tools to manage search bot crawlers too.
Good web hosts detect and block bots. A good host will also have good security to defend against hackers and DOS / DDOS attacks. I like Namecheap managed servers for this reason. If you want to you could use an unmanaged VPS or Dedicated server or an Amazon EC2 container and configure your own anti bot defense system. Get hosted with Namecheap if you want the easy option.
When you ping search engines, broadcast sites or content aggregation sites you effectively tell them to visit your website and crawl all over it. This will slowdown your website. Remember that
- many content aggregation sites are only looking for content to copy and republish for their own purposes;
- provided they are aware your website exists, search engines will regularly crawl your website whether pinged or not; and
- WordPress automatically pings Pingdom when new content is created or old content is updated.
Ping manually only if you really must. At all other times let WordPress decide when to ping Pingdom and let bots decide when to crawl pages.
Photon CDN is the image content delivery network (CDN ) provided to Jetpack users by the organization behind the development of the WordPress CMS software. Install Jetpack and enable Photon (disable any modules that are not needed). Doing so will help your web server do less work.
Robots.txt is used to tell bots which parts of a website they should not crawl. Robots.txt is also used to ban bots. It is next to useless for banning bad bots. Bad bots ignore it and hackers read it to discover the parts of a site they are not meant to see. Good bots pay attention to robots.txt sometimes. Leave robots.txt management in the capable hands of your SEO plugin and WordPress.
There are web masters who have ‘connections’, if you know what I mean. Upset them at your own risk. Spam their sites and you could find yourself under retaliatory attack. It happens. Do not upset those who could attack your website.
And Then There’s Cloudflare
Cloudflare is a popular content delivery network (CDN). I’m not a fan of Cloudflare but it is a middleman between a website and its visitors that also happens to shield against attacking bots and malicious visitors. Cloudflare can help to keep a site functional when traffic is heavy.
Rounding Up
Aggressive bots can be tamed and the website slowdown effects of over-frequent bot crawls can be mitigated. Use firewalls and security software to defend your website against bad bots and malicious visitors. Use cache plugins and CDN storage to reduce the work your web server needs to do to display content to visitors. Use a good secure web host and apply the do’s and don’ts given above to keep your website functional when bots attack or when visitor numbers are high.