Robots.txt Generator

Generate robots.txt file for your website online free — control search engine crawling with proper directives instantly.

Last updated: 10 October 2026

Generate a valid robots.txt controlling which paths search engines may crawl. A sample allowing /public/ while blocking /admin/ and /private/, with a sitemap reference, produced correct syntax including wildcard support.

Generated robots.txt

# robots.txt

User-agent: *

Allow: /
Allow: /images/
Allow: /css/
Allow: /js/

Worked Examples

Tested output

Allow /public/, block /admin/ and /private/ — valid syntax confirmed
Wildcard rules (* and $) supported for path patterns
Sitemap line appended pointing to the site's sitemap.xml

Placement rules

robots.txt lives at the domain root: example.com/robots.txt.
Blocking a URL does not remove it from the index — it only stops crawling; use noindex for removal.
Every user-agent group applies independently: a specific bot rule does not inherit from User-agent: *.

About Robots.txt Generator

Generate a valid robots.txt file for your website. Control which paths search engines can crawl and which they should skip. Configure rules for all bots or specific user-agents like Googlebot or Bingbot. Test data: generated rules allowing /public/ but blocking /admin/ and /private/ — valid robots.txt syntax confirmed with correct path formatting and wildcard support. Also added a sitemap reference pointing to https://example.com/sitemap.xml, producing a complete robots.txt ready for deployment.

Limits: Does not validate whether the sitemap URL actually exists or returns a 200 status; Cannot test whether search engines will actually obey the rules — compliance is voluntary; Does not generate crawl-delay directives (supported by some search engines); Path rules are case-sensitive — /Admin/ and /admin/ are different paths in robots.txt; Cannot verify if your site already has a robots.txt file that might conflict

Who Uses This Tool

A web developer generating a robots.txt that blocks /api/ and /admin/ but allows all public pages for a new website
An SEO specialist creating rules that direct Googlebot to crawl the /blog/ section but block Bingbot from staging URLs
A site owner adding a sitemap reference to help search engines discover pages more efficiently

Key Features

Allow/disallow rules for any path
User-agent specific configuration (Googlebot, Bingbot, etc.)
Sitemap URL reference
Wildcard path matching (* and $)
Preview valid robots.txt syntax
Copy or download the generated file

How to use Robots.txt Generator

Choose to allow or disallow all bots by default.
Add specific path rules (allow or disallow individual directories).
Optionally add a sitemap URL.
Copy the generated robots.txt content.

How It Compares

Unlike robots-txt.com which generates basic rules but doesn't support user-agent-specific configuration, this tool provides full control over per-bot rules. It also shows a preview of the exact file content — most generators only show the rules without the complete file format.

Privacy & Security

Because processing happens directly in your browser, your files and input are not uploaded to our servers.

Written and checked by Mohammad Sohail

Generated syntax re-checked on 10 October 2026 against the robots.txt specification.

About FreeDigitalToolkit →

Frequently Asked Questions

What is robots.txt?

robots.txt is a plain text file placed at the root of your website (e.g., example.com/robots.txt) that instructs search engine crawlers which pages to access and which to skip. It's part of the Robots Exclusion Protocol.

Does robots.txt block hackers or bad bots?

No. robots.txt is a voluntary guideline — legitimate search engines respect it, but malicious bots can ignore it entirely. For security, use server-side authentication, firewalls, or .htaccess rules, not robots.txt.

How do I block a specific directory?

Add a rule like 'Disallow: /admin/' to block the /admin/ path and everything under it. Use 'Disallow: /admin/login' to block only a specific page.

Can I allow some bots but block others?

Yes. Use separate User-agent blocks. For example, allow Googlebot to crawl everything but block all other bots. Search engines only read the block matching their user-agent name.

Where do I put the robots.txt file?

Upload it to the root directory of your website (e.g., https://example.com/robots.txt). It must be accessible at this exact URL for search engines to find it.

Browse all Generator Tools or explore our full collection of 86 free tools.