Clixcel Free tools

Free tools

52 tools, free to use

Analytics3

SEO9

Web performance3

Paid ads4

Social5

CRO1

Web development7

Local SEO2

Email2

E-commerce11

Documents5

See all tools on one page

Free tool · No sign-up · Nothing stored

Free robots.txt generator

Build a robots.txt with ready rules for WordPress, WooCommerce and Shopify, choose which AI crawlers may visit, add your sitemap, then copy or download the file.

How to use this tool

Common blocks

Search engines such as Google and Bing follow the rules under "User-agent: *", so they can crawl everything you have not blocked above.

AI crawlers to block

Training crawlers collect pages to train models. Blocking them has no effect on your search rankings. Search and answer crawlers fetch pages so AI search tools and chat apps can show and link to them, so blocking them keeps your pages out of those answers. On-request crawlers visit only when a person asks about a page. Every crawler you tick gets its own "Disallow: /" group; the rest follow your rules above.

Quick choice:
Custom rules

Use * as the user-agent for every crawler, or name one, such as Googlebot. In a path, * matches any characters and $ marks the end of the address.

One full address per line, starting with https://. WordPress SEO plugins usually use /sitemap_index.xml.

    Then test your rules with the robots.txt checker to see how each crawler reads them.

    What robots.txt does, and what it does not

    robots.txt is a plain text file that tells crawlers which addresses on your site they may request. Search engines read it before they crawl, and the well-known AI crawlers do too. It is a request, not a lock: a crawler that ignores it can still fetch your pages, so use a password, firewall or CDN rule for anything that must stay private.

    It controls crawling, not indexing. Blocking a page does not remove it from search results. If a page should not appear in Google, let it be crawled and add a noindex tag instead, which the meta tag generator can write for you.

    Where the file goes

    • At the root of the host: https://www.example.com/robots.txt. A robots.txt in a subfolder is ignored.
    • One per subdomain: shop.example.com and blog.example.com each need their own file.
    • Plain text, UTF-8, named robots.txt in lower case. Google reads the first 500 KiB.
    • WordPress: WordPress serves a basic file of its own. Edit it in your SEO plugin (Yoast and Rank Math both have a file editor), or upload a real robots.txt to the folder that holds wp-config.php, which replaces it.
    • Shopify: in your admin, go to Online Store, Themes, then Edit code. Add a new template, choose robots, and paste the template this tool writes.

    Test it before and after you publish

    Paste the file into the robots.txt checker to see how Googlebot and each AI crawler read your rules, and whether a firewall turns crawlers away before robots.txt is even read. After publishing, open the robots.txt report in Google Search Console to see the version Google fetched and any lines it could not read.

    Then make sure the pages you want found are ready for search: describe them with the schema generator and run the SEO checker on your key pages.

    Questions

    Does robots.txt keep a page out of Google?

    No. It stops crawling, not indexing. A blocked page that other sites link to can still appear in results, usually with no description. To keep a page out, let it be crawled and add a noindex meta tag or X-Robots-Tag header. Do not block it in robots.txt as well, or crawlers never see the noindex.

    Should I block AI crawlers?

    That is a business decision, and robots.txt lets you make it crawler by crawler. Blocking training crawlers keeps your pages out of future training data and has no effect on search rankings. Blocking search and answer crawlers keeps your pages out of AI search results and chat answers, along with any visits those answers would send. You can block one kind and allow the other.

    Does Google read Crawl-delay?

    No. Google ignores Crawl-delay and sets its own pace from how quickly your server answers. Bing still reads it, but Bing Webmaster Tools has a crawl control setting that does the same job without touching the file, so this generator leaves it out.

    How do Allow and Disallow work together?

    Google uses the most specific rule: the one with the longest matching path, and Allow wins a tie. So Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php blocks the admin area but keeps the one file many themes and plugins need. In paths, * matches any characters and $ marks the end of the address.

    Why can't I upload a robots.txt to Shopify?

    Shopify writes the file for you and already blocks the admin, cart, checkout and account pages, internal search, sorted collections and pages with two or more filters. To add rules, create the robots.txt.liquid template in your theme code, which this tool writes for you. Shopify support does not help with edits to that template, so keep a copy of what you change.

    How quickly do changes take effect?

    Google usually keeps a copy of robots.txt for up to 24 hours. After a big change, open the robots.txt report in Search Console (under Settings) and request a recrawl. Other crawlers check again on their own schedules.

    Pages missing from search?

    Our technical SEO team checks robots.txt, sitemaps, canonicals and crawl settings, and fixes whatever keeps your pages out of Google.

    See our SEO service