Robots.txt Generator
Create a custom robots.txt file to guide search engine crawlers on your website. Whether you need to block specific paths, reduce crawl load, or point crawlers to your sitemap, this free generator builds a valid robots.txt with support for multiple user-agent rules.
Crawl Rules
Sitemap URL
Preview
How to Create a Robots.txt File
Add one or more user-agent rule sets using the editor on the left. Each rule set targets a specific crawler (use * for all crawlers) and defines which paths to allow or disallow. Add your sitemap URL at the bottom. Preview the generated robots.txt in real time, then copy or download it. Upload the file to the root of your domain, for example, https://yoursite.com/robots.txt.
Robots.txt Best Practices for SEO
A properly configured robots.txt helps search engines crawl your site efficiently without wasting budget on irrelevant pages.
Disallow vs. Noindex
There is a critical distinction between Disallow in robots.txt and a noindex meta tag. Disallowing a page in robots.txt prevents crawlers from fetching it, which means they will never see a noindex directive on that page, so it can still appear in search results if linked externally. For sensitive pages, use noindex (so the page is crawled and the directive is seen) rather than blocking it entirely in robots.txt. Use robots.txt to block low-value URL parameters, staging environments, and duplicate content that wastes crawl budget.
Always Include a Sitemap Reference
Adding your sitemap URL to robots.txt is the simplest way to ensure every crawler that visits your site can discover it. Use the Sitemap.xml Generator to create your sitemap first, then reference it here.
Testing Your Robots.txt
Google Search Console includes a robots.txt tester that lets you simulate how Googlebot interprets your file. Use it before deploying changes, especially if you are blocking new paths. A single misplaced Disallow can hide important pages from search results.
Common Robots.txt Mistakes
- Blocking CSS, JS, or image files. Preventing crawlers from accessing these resources can cause Google to render pages incorrectly, hurting your rankings.
- Using Disallow instead of Noindex for sensitive pages. If a disallowed page gets linked from another site, it can still appear in search results without your intended noindex directive ever being seen.
- Missing sitemap reference. Crawlers may still find your sitemap, but adding it explicitly speeds up discovery.
- Blocking the entire site accidentally. A single
Disallow: /on theUser-agent: *rule blocks all crawlers from your entire site.
Example: Well-Configured Robots.txt
This example allows all standard crawlers to access the main site while blocking the admin area and checkout pages. It also blocks GPTBot entirely (a common pattern for sites that do not want their content used for AI training) and ensures crawlers find the sitemap.
Frequently Asked Questions
Does robots.txt hide pages from Google?
No. Robots.txt controls crawling, not indexing. If a page is disallowed in robots.txt but linked from other sites, it can still appear in search results (without the content or any noindex directive). Use noindex tags to prevent indexing.
What is the difference between Disallow and Noindex?
Disallow prevents crawlers from fetching a page. Noindex tells crawlers not to include the page in search results. Use Disallow for crawl budget management and Noindex for preventing indexing.
Can I target specific crawlers with robots.txt?
Yes. Use different User-agent lines to target specific crawlers like Googlebot, Bingbot, or GPTBot. Rules are applied in order of specificity, so more specific user-agent rules override general ones.
Should I block /admin/ in robots.txt?
Yes, blocking admin paths prevents them from appearing in search results. For truly sensitive pages, combine robots.txt blocking with authentication and noindex tags as a defense-in-depth approach.
How do I test my robots.txt file?
Use the robots.txt tester in Google Search Console to simulate how Googlebot reads your file. You can test specific URLs to verify they are allowed or blocked as intended before deploying changes to production.
Need a complete SEO audit? The SEOflow desktop application analyzes technical SEO, metadata, sitemaps, robots.txt, and more, all in one workspace.