Web, SEO & Business
Choose which pages crawlers may explore, list your sitemap, and decide whether AI model-training crawlers may read your site. Download the result as robots.txt.
Free · no sign-up · runs in your browser · updated
Starting point
Example: /admin/ or /private/
Example: /admin/public/
Ignored by Google.
Pick a starting point: allow everything, block everything or WordPress. Add your own Allow and Disallow paths if needed.
Enter your sitemap address, and tick the AI crawlers you want to block.
Copy or download robots.txt, then upload it to the root folder of your site so it opens at yourdomain.com/robots.txt.
robots.txt asks well-behaved crawlers not to explore certain paths. It is not a security measure: bad crawlers can ignore it, and a blocked page can still appear in search results if another site links to it. To keep a page out of the index, use noindex, and for sensitive data use authentication.
One file applies per host (including each subdomain and protocol). It must sit at the root of the domain, and the file name is lowercase: robots.txt.
Some AI operators use different crawlers to train models (for example GPTBot, ClaudeBot, CCBot and Google-Extended) and to search for or cite content when a user asks a question. Blocking a training crawler is not the same as blocking a search crawler, so your site can still appear as a source in AI answers. Decide according to your content policy.
Not completely. A blocked page can still be indexed without its content if other pages link to it. To remove a page from search results, use a meta noindex tag and make sure the page is not blocked, so crawlers can read it.
It sets a pause between crawler requests. Google ignores Crawl-delay, but some other search engines, such as Bing, support it.
A Sitemap line helps crawlers find your list of pages without you submitting it manually. Use the full URL, including https://.
Not automatically. Those are model-training crawlers. Crawlers that fetch pages to answer a user’s question use different names and are controlled separately.
At the root of the domain, so it opens at https://yourdomain.com/robots.txt. A file in a subfolder is not read by crawlers.
Paste robots.txt, see which AI crawlers are allowed or blocked, then build the blocking rules.
Open toolCreate an llms.txt that follows the llmstxt.org specification from your pages, or check an existing file.
Open toolCreate title, description, canonical, Open Graph and Twitter Card tags.
Open toolSee roughly how your title and description look in Google results.
Open toolNeed more than tools?
These tools are only a small part of what Tipamedia can build. Tell us what you need and we will help you work out the next step.