If you found this page in your access log, this is us

How we identify ourselves

We identify ourselves on every request we make, as ourselves. Every request carries this header:

User-Agent: Reqbeat/1.0 (+https://reqbeat.com/crawler)

Match on it in your access log to confirm a request came from us. This page describes crawler policy version 1.0.0.

In a hurry

To stop us completely, publish this at the root of your site:

User-agent: Reqbeat
Disallow: /

We re-read robots.txt every 24 hours, so we honor the change within 24 hours. To slow us down instead, or for anything robots.txt cannot express, email [email protected]. No lever requires telling us first.

What we collect, and from where

We collect public job advertisements and their index pages from public job boards and ATS pages we observe directly. We keep the ad plus its metadata: employer, role, location, first-seen date and the carrying board. We keep no personal data about applicants or employees. Every result is a real posting, tagged with the board it was detected on. We never infer intent. LinkedIn is not included.

Every request is a plain GET of a published endpoint. We do not sign in, submit forms, or render and execute pages to reach content a site does not serve.

What we publish from it, and the limits we hold ourselves to when we do, are on data ethics. How the numbers are computed is on the method page.

How we read robots.txt

We implement RFC 9309, not an approximation of it. Concretely:

• We fetch /robots.txt before anything else on a host.
• We obey the Reqbeat group if one is present, else the * group. Never both.
• The longest matching rule wins. An Allow beats a Disallow of equal length.
• * and $ wildcards work. Paths are case-sensitive.
• An empty Disallow means nothing is disallowed.
• A 4xx answer for robots.txt counts as "no robots.txt published".
• We honor Crawl-delay as a floor, capped at 60 seconds.

Where we deviate, and why

A page like this is worth reading only if it also says what we do not do. There is one departure. If robots.txt is unreachable (a 5xx answer or a timeout), we serve the last cached decision for the host. If no cached decision exists, we allow the fetch and log a warning. This departs from the SHOULD in RFC 9309 section 2.3.1.3.

How often we fetch

The corpus refreshes every 3 hours across 7 regions. We re-read robots.txt every 24 hours, and we honor Crawl-delay up to 60 seconds. If the pace is still a problem for your site, ask us to slow down and we will.

What this page covers

This page addresses site owners, not buyers. It covers the crawler that collects public job postings. If you want to use the data instead, start at the FAQ or the quickstart.

Contact

A person reads [email protected]. Include the domain you are asking about and what you would like changed. If you want us out of a source these rules do not cover, say so and we will act on it.