Robots.txt

Click Robots.txt in the menu. Before fetching a URL, Compare Versions checks the site’s robots.txt. If those rules block the page, it waits instead of fetching. Related: Fetch status and Add webpages.

When a page is blocked

Blocked pages appear on the Robots.txt screen under Waiting for authorization. They are not fetched until you authorize them.

Authorizing an override

Click Authorize on a page that is waiting. It then appears under Authorized overrides, and checks can continue. The Terms of Service cover when an override is allowed. After that, Daily checks can run again for the page.

Revoking an override

Revoke an override to stop fetching that page again. Versions already saved are kept.

  • Fetch status What fetch statuses mean, and what happens when a site blocks or rate-limits checks.
  • What to monitor Examples of pages people track, and what a daily check is looking for.
  • Add webpages Paste up to 50 URLs at once, pick a folder, and start tracking changes.
  • Daily checks Every active URL is checked automatically once a day. Here is what each run does.
All help topics FAQ Create account