Tools / Create / Web & Developer

Web & Developer tool

Create Robots.txt

Build starter robots.txt rules with user-agent sections, allow/disallow paths, crawl-delay, sitemap, and warnings.

Multi-section Review warnings

Build rules

Add one or more user-agent sections, then review the preview.

robots.txt preview

Copy this into a file named robots.txt after review.

 
 
 
 
 
  
 
 
 
More web & developer tools
 

Try more web tools for slugs, UUIDs, metadata, robots.txt rules, and developer helpers.

 
 
    Create URL Slugs        Create UUIDs        Write Meta Descriptions      
 
 
 
 
 
How this tool helps
 
Create Robots.txt
 
Create starter robots.txt files with user-agent sections, rules, crawl-delay, sitemap, and review warnings.
 
 
 
How to use it
 
     
  1.   1  Enter the details that match the task you are working on. 
  2.   2  Adjust the options to compare a few useful variations. 
  3.   3  Copy, download, or review the output before using it in a real project. 
    4.  
 
 
 
Useful for
 
     
  • Web & Developer tasks involving robots.txt
  • Web & Developer tasks involving seo
  • Web & Developer tasks involving developer
    •