Robots.txt Generator
Build a correct robots.txt file in a few clicks and test which URLs Google is allowed to crawl.
robots.txt
Test a URL
Tests the file above using Google’s rules: the most specific (longest) matching rule wins; Allow wins a tie.
How to use the Robots.txt Generator & Tester
- 1
Start from a preset (Allow all, WordPress, Shopify…) or set the default rule.
- 2
Add folders to block, your sitemap URL and, if you want, block AI training bots.
- 3
Test any path in the tester – it uses the same rules as Google (longest match wins).
- 4
Download the file and upload it to the root of your site, so it opens at yourdomain.com/robots.txt.
Frequently asked questions
What is robots.txt?
A small text file at the root of your website that tells crawlers which parts of the site they may crawl. It must be at exactly /robots.txt.
Does Disallow remove a page from Google?
No. Disallow only stops crawling – a blocked page can still be indexed if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
Should I block AI bots?
It is your choice. Blocking bots such as GPTBot, CCBot or Google-Extended asks AI companies not to use your content for training. It does not affect your normal Google rankings.
Should I block CSS and JavaScript files?
No. Google needs CSS and JavaScript to render your pages. Blocking them can hurt how your pages are understood and ranked.
Other SEO tools you may like
SERP Snippet Preview
See how your title and meta description will look on Google – desktop and mobile, with pixel-width limits.
Meta Tag & Open Graph Generator
Create SEO, Open Graph and Twitter Card tags and preview how your link looks on WhatsApp, Facebook and X.
Schema Markup Generator
Generate JSON-LD structured data for FAQ, Local Business, Course, Article, Product and Breadcrumbs.
Image Compressor & WebP Converter
Shrink JPG, PNG and WebP images by up to 80% and convert them to WebP or AVIF – in your browser, nothing is uploaded.
Email Signature Generator
Design a professional email signature with logo, social icons and a call-to-action button – copy it into Gmail or Outlook in one click.
UTM Link Builder
Build campaign URLs with UTM tags for Google Analytics – with presets and a downloadable QR code.
Want to master SEO like a pro?
Learn SEO, Google Ads, Meta Ads and AI tools hands-on in our AI & Digital Marketing Course. Next batch starts 26 November 2026 · Only 15 seats per batch