Skip to content
ReeBot

About ReeBotCrawler

If you found ReeBotCrawler/1.0 in your server logs, a ReeBot customer asked us to import pages from your site so their assistant can answer questions about them.

When it visits

  • Only when a ReeBot customer adds a website address to their assistant's knowledge, or asks to re-sync it. It doesn't browse the web on its own.
  • It stays on the site it was given, follows your sitemap and links on the same site, and stops at the page limit the customer chose.
  • It makes a few requests at a time, with timeouts and size limits, and never submits forms or signs in.

How to control it

ReeBotCrawler respects robots.txt. To stop it reading your whole site, add:

User-agent: ReeBotCrawler
Disallow: /

You can also disallow specific paths in the same way. Changes apply the next time a customer imports or re-syncs your site.

Problems or questions

Our customers may only import sites they own or are authorised to use. If you think the crawler is misbehaving, or that someone imported your site without permission, contact us with your domain and the dates you saw it, and we'll look into it.