A large share of the traffic to any website is automated. Understanding which is which matters, because some of it is useful, some is waste, and some is a problem.
The categories
- Search engine crawlers – Googlebot and its equivalents. You want these. Blocking them removes you from search results.
- Monitoring and preview services – uptime checks, link previews when someone shares your page. Harmless.
- SEO and marketing crawlers – gathering data to sell. They consume your resources and give you nothing.
- Scrapers – copying your content or, on a shop, your prices. Often aggressive.
- Attack traffic – probing for known vulnerabilities and guessing logins.
Why it matters
An aggressive crawler working through a large shop can generate more requests than all your real customers combined, and can push the account into its resource limits. A 508 error at your busiest moment is often a bot rather than success.
What you can do
- robots.txt asks well-behaved crawlers to slow down or stay out of certain areas. Polite bots obey it; bad ones ignore it entirely.
- Crawl delay in robots.txt helps with the ones that respect it.
- Blocking by user agent or address in .htaccess works on the rest.
- Caching reduces the cost of bot traffic dramatically, since a cached page is cheap to serve.
Be careful what you block
Blocking Googlebot, or blocking your images and scripts, damages your search visibility. If in doubt, ask us before adding a block – see The correct robots.txt for an online shop.
