Security maturity
robots.txt check
robots.txt guides search engine crawlers. It does not hide anything, and the paths it lists are public.
Check this on your own domain now. It is free with a DomainGuard account and works on any domain. The check reads only what your domain publishes, and gives you a report you can send to whoever runs your website.
What we look at
We read /robots.txt and list the paths it asks crawlers to skip.
Why it matters to your business
People sometimes list private areas in robots.txt hoping to hide them, which publishes a map of exactly where to look. It is worth checking that nothing listed there is meant to be secret.
How to fix it
- Protect private areas with a login, not with robots.txt.
- Remove entries that point at sensitive paths.
If you would rather not touch DNS or server settings yourself, forward this page and your report to whoever manages your website, or ask us to do it.
Check your own site for this
A free outside-in check of your website, email and domain records, with what to fix first. No card and no password to choose.