How to use this check
Enter a public site URL and SitePulse will request only the same-origin /robots.txt file.
The probe derives the origin from the input URL and fetches /robots.txt after validating the host.
Choose an amount before the secure payment page opens.
Crawlability
Same-origin fileRobots.txt Checker
Fetch the origin robots.txt file and identify basic crawl directives and sitemap hints.
Pulse score--Same-origin file
Signal checklist
The result shows robots status, size, directives and sitemap hints.
Robots.txt Checker
Enter a public site URL and SitePulse will request only the same-origin /robots.txt file.
Result
The result shows robots status, size, directives and sitemap hints.
Support
Keep this free result available
Use the result first. Support and promotional areas stay separate from the useful result.
- Useful result first
- No signup required
- Promotions kept separate
Choose an amount before the secure payment page opens.
Guide and interpretation
How to interpret the result
A missing robots.txt is not always an error, but accidental Disallow rules can block important pages.
Example
A robots file with Sitemap: https://example.com/sitemap.xml helps crawlers discover submitted URLs.
Common issues and next steps
Staging Disallow rules can accidentally ship to production.
Review robots.txt after site changes and confirm important sections are crawlable before requesting indexing.
Methodology and limits
The free check does not emulate every crawler or evaluate every path rule.
FAQ
Will this crawl my site?
No. It fetches only robots.txt for the normalized origin.
Is Disallow always bad?
No. It is useful for areas that should not be crawled, but dangerous when applied broadly by mistake.