SEO · Client-side only
robots.txt + sitemap generator
Generate a clean robots.txt with explicit rules for the 11 AI crawlers that matter in 2026 — plus a starter sitemap.xml.
robots.txt
# Generated with Build on Vibe · buildonvibe.site/tools/free/robots-generator User-agent: * Allow: / Disallow: /admin Disallow: /api/private Disallow: /*.json$ User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / Sitemap: https://example.com/sitemap.xml
sitemap.xml (starter)
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url><loc>https://example.com/</loc><changefreq>weekly</changefreq><priority>1.0</priority></url> <url><loc>https://example.com/pricing</loc><changefreq>monthly</changefreq><priority>0.8</priority></url> <url><loc>https://example.com/about</loc><changefreq>monthly</changefreq><priority>0.6</priority></url> <url><loc>https://example.com/blog</loc><changefreq>weekly</changefreq><priority>0.7</priority></url> </urlset>
Replace the sample routes with your real public pages. Upload both files to your site root (same domain).