How Robots.txt Works
User-agent and disallow or allow rules control crawling behavior for specified bots. Sitemap locations can also be declared in the file.
A Simple Example
A site may block crawling of internal search-result paths while leaving public guides accessible.
Why Robots.txt Matters
Robots.txt helps manage crawler access and avoid wasting crawl activity on unsuitable URL patterns.
Common Misreading
Blocking crawling is not a reliable way to remove a URL from search. A blocked URL can still be indexed from external signals, and robots.txt does not protect private data.