Skip to content

Crawler policy

We operate an automated crawler that collects publicly published notices of distressed property sales in Ontario — municipal tax sales under the Municipal Act, 2001, power of sale listings, judicial sales, and estate dispositions.

How to identify our crawler

Our requests carry this User-Agent:

OntarioDistressedAssetFinder/1.0
  (+https://odaf.example.com/about/crawler;
   aggregates public distressed-property notices;
   contact: crawler@odaf.example.com)

What we respect

  • robots.txt. We fetch and parse it before any request to a host, and we do not fetch disallowed paths. Longest-match wins, and Allow beats Disallow at equal specificity.
  • Crawl-delay. If your robots.txt specifies one, we use it. Otherwise we default to at least 1.5 seconds between requests to the same host, and often longer.
  • Conditional requests. We send If-None-Match and If-Modified-Since. If your page has not changed, we take the 304 and do not re-download it.
  • Volume. We check each source three times a day and fetch a bounded number of detail pages per run. We do not crawl exhaustively.

How to block us

Add this to your robots.txt and we will stop on our next run:

User-agent: OntarioDistressedAssetFinder
Disallow: /

You can also email crawler@odaf.example.com and we will remove your site from our source list directly.

Corrections and removals

If a listing on our site is inaccurate, out of date, or should not be published, contact us and we will correct or remove it. We keep the publisher named on every listing precisely so that errors can be traced and fixed.