User-agent: * Crawl-delay: 10 <---- Slows the crawl rates your website. (Prevents excessive load spikes..) ---> User-agent: * Disallow: /wp-includes/ Disallow: /feed/ Disallow: /trackback/ Disallow: /cgi-bin/