SEO / THE EVERYDAY TOOLKIT
Robots.txt Generator.
Draft crawler directives and a sitemap URL for your website. This is a free utility from my portfolio toolkit, with local browser processing and no account required.
Robots rules guide compliant crawlers; they do not secure private files or guarantee deindexing. Review the draft before publishing it at the website root.
SIMPLE BY DESIGN
How to use Robots.txt Generator
- Enter your website URL and choose an allow or disallow-all crawl policy.
- For an allow policy, enter any paths to exclude, one per line starting with /.
- Review every directive before publishing the file at the website root.
Make crawler rules explicit before publishing them.
A robots.txt file describes crawl preferences to compliant bots. This generator starts from your website origin and produces either an allow policy with optional disallowed paths or a disallow-all policy. It also includes a sitemap location. Blocking a URL in robots.txt is not authentication and is not a reliable way to remove a known page from search. Protect private content with access controls and use the appropriate indexing directives for public pages.
A FEW THINGS TO KNOW
A little more clarity.
Does disallowing a path secure it?
No. Anyone can still request an unprotected URL, and some crawlers ignore robots.txt. Use server-side authorization for private content.
Is my input sent to a server?
No. This utility processes your input in your browser. URLs you paste are treated as data; the tool does not fetch them. Be careful with clipboard history on shared devices.
FROM THE JOURNAL
Robots.txt and Noindex Solve Different Problems
Choose between crawling controls and indexing directives without accidentally preventing a crawler from reading your instruction.