# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: User-agent: AdsBot-Google Disallow: User-agent: Googlebot Disallow: User-agent: Googlebot-image Disallow: