Robots.txt Generator
Build a correct robots.txt file in a few clicks and test which URLs Google is allowed to crawl.
robots.txt
Test a URL
Tests the file above using Google’s rules: the most specific (longest) matching rule wins; Allow wins a tie.
How to use the Robots.txt Generator & Tester
- 1
Start from a preset (Allow all, WordPress, Shopify…) or set the default rule.
- 2
Add folders to block, your sitemap URL and, if you want, block AI training bots.
- 3
Test any path in the tester – it uses the same rules as Google (longest match wins).
- 4
Download the file and upload it to the root of your site, so it opens at yourdomain.com/robots.txt.
Frequently asked questions
What is robots.txt?
A small text file at the root of your website that tells crawlers which parts of the site they may crawl. It must be at exactly /robots.txt.
Does Disallow remove a page from Google?
No. Disallow only stops crawling – a blocked page can still be indexed if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
Should I block AI bots?
It is your choice. Blocking bots such as GPTBot, CCBot or Google-Extended asks AI companies not to use your content for training. It does not affect your normal Google rankings.
Should I block CSS and JavaScript files?
No. Google needs CSS and JavaScript to render your pages. Blocking them can hurt how your pages are understood and ranked.
More digital marketing tools
On-Page SEO Checker
Enter any URL and get an SEO score with 25+ checks – title, meta, headings, images, schema, speed and more.
Page Speed & Core Web Vitals
Test any page with Google’s Lighthouse – speed score, Core Web Vitals from real users and what to fix first.
Redirect & Status Checker
Follow every redirect hop (301, 302, 307…), find chains and loops, and test the http / www versions of your site.
Domain Age & WHOIS Checker
Find when a domain was registered, when it expires, the registrar, name servers and status – from the official registry.
SERP Snippet Preview
See how your title and meta description will look on Google – desktop and mobile, with pixel-width limits.
Schema Markup Generator
Generate JSON-LD structured data for FAQ, Local Business, Course, Article, Product and Breadcrumbs.
Want to master digital marketing like a pro?
Learn SEO, Google Ads, Meta Ads and AI tools hands-on in our AI & Digital Marketing Course. Next batch starts 26 November 2026 · Only 15 seats per batch