# www.robotstxt.org/ User-agent: SemrushBot Disallow: / # Allow crawling of all content User-agent: * Disallow: