User-agent: * Disallow: Crawl-Delay: 5 Request-rate: 18/1m 2100-0529 Request-rate: 6/1m 0530-2059