The Reqbeat crawler
If you found this page in your access log, this is us. Here is what we collect, how often we fetch, the rules we follow, the one place we knowingly deviate from them, and every way you have to slow us down or stop us.
We identify ourselves on every request we make as ourselves:
User-Agent: Reqbeat/1.0 (+https://reqbeat.com/crawler)
Policy version 1.0.0 · Last checked against the crawler's own source 2026-08-10 · Page generated September 14, 2026
In a hurry
To stop us completely, publish this at the root of your site and we will stop on our next read of it — within 24 hours:
User-agent: Reqbeat
Disallow: /To slow us down instead, or for anything robots.txt cannot express, see how to control us — or email [email protected].
What we collect, and from where
- •We read public job advertisements — the postings companies publish on their own applicant-tracking systems and on public job boards — and the index pages that list them.
- •What we keep is the advertisement and its metadata: the employer, the role, where it is, when it first appeared, and which board carried it. Those are what we measure hiring demand from, across companies and over time.
- •Requests we make as ourselves are plain GETs of endpoints the site already publishes. We do not sign in, we do not submit forms, and we do not render or execute anything to get at content the page does not serve.
- •We collect no personal data about applicants or employees, and we publish nothing about a named individual. There is no field in our published corpus that holds a person's name or contact details, and adding one fails our build.
What we publish from it, and the limits we hold ourselves to when we do, are on Data ethics; how the numbers are computed is on Method.
How we read robots.txt
We implement RFC 9309, not an approximation of it. Concretely:
- •We fetch your /robots.txt before we fetch anything else on your host, and we re-read it every 24 hours.
- •If you publish a group for the product token Reqbeat, we obey that group. If you do not, we obey the * group. Never both — that is what RFC 9309 asks for, and a rule you wrote for us replaces the general one rather than adding to it.
- •Within the group that applies to us, the longest matching rule wins, and an Allow beats a Disallow of the same length (RFC 9309 §2.2.2).
- •* and $ work as wildcards. Paths are matched case-sensitively, as the RFC requires — a rule written for /API/ does not govern /api/.
- •An empty `Disallow:` means nothing is disallowed, not everything.
- •If your /robots.txt answers 4xx we treat that as "no robots.txt published" and proceed, which is what RFC 9309 §2.3.1.4 asks for. A 401 or a 404 is not a refusal.
- •Every decision is logged against the source it governed. A fetch we skipped because you disallowed it is recorded as a policy decision, never as a fault on your site — so honouring your rules never costs you anything else.
Where we deviate, and why
A page like this is worth reading only if it also says what we do not do. These are the departures, in full — there are no others.
If your /robots.txt is unreachable — a 5xx, a timeout, a network failure — we serve the last decision we cached for your host, and if we hold none we allow the fetch and log a warning.
Departs from: RFC 9309 §2.3.1.3
The standard says a crawler SHOULD assume a complete disallow when robots.txt is unreachable. It is a SHOULD rather than a MUST, and we depart from it deliberately: a transient 500 on one vendor's robots endpoint would otherwise drop a large share of our data in a way indistinguishable from a quiet hiring week. We would rather over-fetch for a few minutes and say so here. A host that means to keep us out can do it with a robots.txt that answers — and we re-read yours on the next cycle either way.
We honour a published Crawl-delay as a floor on how often we touch your host, but we cap the value we will honour at 60 seconds.
Departs from: Crawl-delay — a de-facto extension, not part of RFC 9309, which defines no rate directive at all
The cap is there so a mistyped value cannot stall us against your host indefinitely. If you need us slower than one request every 60 seconds, a Disallow or an email will do it, and both are honoured without a ceiling.
How often we fetch
- •We pace per host, not per employer. Thousands of employers share one applicant-tracking vendor, so the only limit that means anything is the one on the vendor's host: every employer behind it queues on the same pacer instead of adding up into a burst.
- •We re-check one employer's postings somewhere between once an hour and once every 7 days. The interval is derived from how often that employer's postings actually change, so a board that rarely moves is fetched rarely — and where a cheap change marker is available we read that first and skip the full fetch when nothing has changed.
- •If you answer 429, we slow down without being asked twice: the interval between our requests to your host doubles, up to 60 seconds, and comes back down gradually once you are answering normally again.
How to rate-limit us, or opt out
Every lever below is one the crawler actually implements. Pick whichever suits you — none of them requires telling us first.
Publish a `User-agent: Reqbeat` group in your robots.txt.
We read that group in preference to `*`, and honour its rules on every request we make as ourselves.
Add `Crawl-delay: <seconds>` to that group.
It becomes a floor on how often we touch your host, up to the 60-second cap disclosed above.
Answer 429 Too Many Requests.
We back off automatically and recover slowly. You do not have to tell us anything else, and nothing needs to be published for this to work.
Block or rate-limit on the User-Agent at your edge.
The product token is `Reqbeat` and it is stable across versions, so a match on that alone keeps working.
Opt out entirely: `Disallow: /` under `User-agent: Reqbeat`.
We stop. We do not treat a disallow as an obstacle to route around, and we will not come back under a different name.
Email [email protected] with your domain and what you want changed.
This is the route for anything robots.txt cannot express, and the one to use for a source where the rules above do not apply.
What this page does and does not cover
- •This User-Agent marks the requests we make as ourselves: our own client, our own token, no proxy and no impersonation. Everything on this page describes those requests.
- •It is not our whole crawl. We also read public job boards and aggregators, and not every one of them is fetched this way. We keep a record, per source, of whether we make this claim about it — and where we do not, we do not pretend these rules apply.
- •That record is what the claim is worth. A source cannot be marked as one we fetch as ourselves until someone has measured its robots.txt against the path we actually request, and written down what they measured.
- •Where a robots.txt we consulted says no, we take no for an answer — including on vendors we would rather be able to read. Being told to stay out is this working, not this failing.
Contact
A person reads [email protected]. Include the domain you are asking about and what you would like changed. If you want us out of a source these rules do not cover, this is the address — say so and we will act on it.