About our crawler

Job radar keeps a searchable index of job postings that employers publish on their own careers sites. This page says what our crawler fetches, how it behaves on your servers, what it keeps, and how to stop it.

Contact: contact address to be added

What it fetches

• Only employers' own job boards, as published through their applicant tracking system (ATS): the listing of open jobs and, where the listing has no description, each posting's page.
• Nothing else: no job aggregators or resellers, no social networks, no candidate profiles, no pages behind a sign-in.

Today it reads 12,361 boards of 12,296 employers on these platforms:

How it identifies itself

Public job-board APIs

On the documented public APIs of boards-api.greenhouse.io, api.lever.co, api.eu.lever.co, api.ashbyhq.com, apply.workable.com, .jobs.personio.de, .jobs.personio.com, .recruitee.com, every request carries this User-Agent:
jobcollector/0.1 (private non-commercial job-search tool)

Other careers sites

Requests to other employer careers sites carry a standard browser User-Agent string (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36). That choice is under review (see below); it never changes how the crawler treats your robots.txt or your rate limits.

robots.txt token

jobcollector, on every host, whatever User-Agent the request carries.

How it behaves

• robots.txt is honoured. It is read before anything else on a host and kept for up to 24 hours. Disallow rules for jobcollector (or *) are obeyed and Crawl-delay is used as the minimum gap between requests. When robots.txt answers with a server error or can't be reached, nothing is fetched from that host; it is asked again after 1 hour. One documented exception: smartrecruiters.com, whose robots.txt admits only one named crawler; it is read at the same rate as every other host.
• Requests to one edge start at least 1 second apart (or your Crawl-delay, if longer). Hosts that share one backend count as one edge: every tenant of one Workday pod, of one Oracle Cloud region, and of the multi-tenant platforms .recruitee.com, .jobs.personio.de, .jobs.personio.com, .breezy.hr, .bamboohr.com, .teamtailor.com.
• A board is never read twice within 1 hour, and is read at least once every 24 hours: busier boards more often, quiet ones daily. A failed read is retried after 30 minutes, then less and less often.
• Backoff. A failed request is retried up to 3 times with growing, randomised waits from 1 second to 1 minute. Retry-After is honoured; one longer than 1 minute stops every request to that edge for its full time. A 429 that persists stops the edge for 15 minutes. These pauses survive restarts.
• No evasion. A bot challenge, CAPTCHA or access-denied page is never solved or retried: the edge is paused for 1 hour and the read is recorded as incomplete. An access-denied page that comes right after normal answers means we asked too fast: that edge is then read at half the pace, for good (down to one request every 16 seconds at most). No headless browsers, no proxy pools or rotating addresses, no sign-ins.
• Each request gives up after 30 seconds.

What it keeps

• Postings as published (title, company, location, description, dates, the link to apply) and what we extract from them (seniority, country, work mode, salary when stated), with when each was first seen, last confirmed open, and closed. People apply on your site: we link to your posting.
• Raw answers, as evidence that lets us check and repair what we extracted: listing pages for 14 days; a posting's page while the job is open, and for 90 days after it closes or changes.
• Postings stay in the index after they close, marked closed. Nothing about the people who visit your site is collected.

How to opt out

• robots.txt: User-agent: jobcollector with Disallow: / stops our requests to that host within 24 hours.
• Ask us: write to contact address to be added with your company name and careers site. We switch your boards off and record your company in our exclusion list so it is not added again, and, if you ask, remove the postings and raw answers we already hold.