Free tool · No sign-up

Robots.txt Generator

Build a correct robots.txt file in a few clicks and test which URLs Google is allowed to crawl.

Start from:
All search engines (User-agent: *)
AI crawlers
Block specific bots completely

robots.txt

    Test a URL

    Tests the file above using Google’s rules: the most specific (longest) matching rule wins; Allow wins a tie.

    How to use

    How to use the Robots.txt Generator & Tester

    1. 1

      Start from a preset (Allow all, WordPress, Shopify…) or set the default rule.

    2. 2

      Add folders to block, your sitemap URL and, if you want, block AI training bots.

    3. 3

      Test any path in the tester – it uses the same rules as Google (longest match wins).

    4. 4

      Download the file and upload it to the root of your site, so it opens at yourdomain.com/robots.txt.

    FAQ

    Frequently asked questions

    What is robots.txt?

    A small text file at the root of your website that tells crawlers which parts of the site they may crawl. It must be at exactly /robots.txt.

    Does Disallow remove a page from Google?

    No. Disallow only stops crawling – a blocked page can still be indexed if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

    Should I block AI bots?

    It is your choice. Blocking bots such as GPTBot, CCBot or Google-Extended asks AI companies not to use your content for training. It does not affect your normal Google rankings.

    Should I block CSS and JavaScript files?

    No. Google needs CSS and JavaScript to render your pages. Blocking them can hurt how your pages are understood and ranked.

    Want to master SEO like a pro?

    Learn SEO, Google Ads, Meta Ads and AI tools hands-on in our AI & Digital Marketing Course. Next batch starts 26 November 2026 · Only 15 seats per batch

    Request a Callback

    Share your details – our counsellor will call you within 24 hours.

    Prefer to talk now? Call 7665-542-542