The same role posted twice is one job

A company opens a role on its ATS. An aggregator picks it up. A job board syndicates it. We scrape all three boards, so one role arrives as three rows from three sources, with three ids and three apply URLs.

There is one job. Deciding that is deduplication, and it is one of two mechanisms in this API that reduce the number of rows you get back. The two are constantly mistaken for each other, so start here:

The one-line version: dedup removes a posting you would otherwise see twice. Collapse folds genuinely separate postings that happen to share a name.

Dedup is a flag, not a delete#

When mirrors of one posting are identified, the extras get a duplicate_of pointing at the row we kept. Both /v1/jobs and /v1/jobs/feed exclude flagged rows unconditionally. There is no parameter to turn it off and no way to ask for the mirrors.

The flag is the important design decision. Marking is reversible and non-destructive: a row identified as a duplicate is flagged, never deleted. If the matching turns out to be wrong, the flag clears and the row comes back.

That is why duplicate_of is a column and not a DELETE. Any dedup system is making a judgement call about identity, some of those calls are wrong, and a DELETE makes being wrong permanent. A flag makes it a bug you can fix.

The one place it leaks#

Because dedup is invisible and automatic, it costs you nothing to think about — with one exception, and it is about counting.

total counts distinct postings, not scraped rows. That is almost always what you wanted. But it means this query does not mean what it looks like:

# NOT "how many jobs exist on Greenhouse".
curl -G https://api.hyperjobs.io/v1/jobs \
  -H "Authorization: Bearer $HYPERJOBS_KEY" \
  --data-urlencode "source=greenhouse" \
  --data-urlencode "limit=1"

It counts postings where Greenhouse won the dedup — not every posting Greenhouse carried. A role that exists on both Greenhouse and LinkedIn is attributed to exactly one of them, and ?source= only finds it under the winner. Any per-source breakdown you build is a breakdown of surviving rows, and it is worth saying so in whatever you publish.

The magnitude is small: the coverage snapshot puts flagged duplicates at roughly 1% of rows. Read /v1/meta for the live figure rather than hardcoding a number that rots.

Collapse is a different question#

Some employers post one row per location. A retailer hiring baristas in 40 stores files 40 real, distinct postings: 40 ids, 40 apply URLs, 40 legitimately separate applications. None of them are duplicates. They are just repetitive.

collapse=true groups them:

curl -G https://api.hyperjobs.io/v1/jobs \
  -H "Authorization: Bearer $HYPERJOBS_KEY" \
  --data-urlencode "organization=starbucks" \
  --data-urlencode "collapse=true"

The grouping key is the company and the title:

GROUP BY COALESCE(company_id, lower(organization_name)), title

Three things change in the response. collapsed: true appears at the top level. Every job gains an openings count. And total changes meaning — it becomes the number of groups, not the number of postings. Same field, different unit.

{
  "collapsed": true,
  "total": 812,
  "limit": 50,
  "offset": 0,
  "data": [
    {
      "id": "greenhouse:4820193",
      "title": "Barista",
      "openings": 40,
      "company": { "slug": "starbucks", "name": "Starbucks" }
    }
  ]
}

That shape is illustrative — the counts are there to show you the structure, not to report anything.

The representative row is one real posting#

The corollary bites hardest with location filters. city=Denver with collapse=true finds groups whose representative is in Denver. A group that contains a Denver posting but is represented by the Seattle one will not match. The filter and the grouping do not compose the way you expect, because the representative carries its own location, not the group's.

Collapse also groups on the exact title string. "Barista" and "Barista (Part Time)" are two groups, not one. It is a GROUP BY, not a fuzzy match — which makes it predictable, and means openings is a floor on how repetitive an employer actually is.

When to reach for it#

Do collapse when the group is the unit you care about. Forty identical "Barista" cards is a broken result page; one card reading "Barista · 40 openings" is the correct presentation, in a single request. Same for "how many companies are hiring for X" — without collapse you are counting postings, and one multi-location employer dominates the count.

Don't collapse when the posting is the unit. You need every apply_url, or location matters, or you are syncing a mirror — and the feed has no collapse anyway, because sync is posting-level by definition.

Expiry, the third mechanism#

Not deduplication, but it is the other reason a row you expected is missing.

Postings are expired by board-diffing: every re-scrape compares the source board's current listing against what we hold, and anything that disappeared gets an expires_at. The employer taking the job down is the signal. We do not guess.

const isLive = (job) => !job.expires_at || new Date(job.expires_at) > new Date();

Which one is eating my rows?#

1. Is `collapsed: true` in the response?

   Then total counts groups, and every row stands in for openings others. Drop collapse to see postings.

2. Is the row expired?

   Fetch it with /v1/jobs/{id} and check expires_at. Present in the feed but missing from /v1/jobs is the signature.

3. Is the row a duplicate?

   If a posting exists on two boards and you only ever see one, dedup picked a winner. That is working correctly.

4. Or is it just a filter?

   Far more likely than any of the above. Bad values do not 400 — they are ignored or match nothing. See filtering for the bisection procedure.

The full reference is duplicates & collapsing.