User-agent: * Disallow: /archives/ Disallow: /public_html/archives/ User-agent: WebCrawler Disallow: /archives/ Disallow: /public_html/archives/