# The same role posted twice is one job
> A role mirrored to Greenhouse and LinkedIn is one posting, not two. What deduplication has to decide, and what the API lets you see.
Source: https://hyperjobs.io/blog/collapsing-duplicate-postings
*HyperJobs Team · 23 June 2026 · 7 min read*
A company opens a role on its ATS. An aggregator picks it up. A job board
syndicates it. We scrape all three boards, so one role arrives as three rows from
three sources, with three ids and three apply URLs.
There is one job. Deciding that is deduplication, and it is one of two mechanisms
in this API that reduce the number of rows you get back. The two are constantly
mistaken for each other, so start here:
| | **Dedup** | **Collapse** |
| --- | --- | --- |
| What it merges | The **same** posting mirrored across boards | **Different** postings sharing company + title |
| Example | One Stripe job on Greenhouse *and* LinkedIn | 40 separate "Barista" postings, one per store |
| Trigger | Automatic, always on | Opt-in via `?collapse=true` |
| Can you disable it? | No | It's off by default |
| Effect | The mirror rows are **excluded** | Rows are **grouped**, with an `openings` count |
The one-line version: **dedup removes a posting you would otherwise see twice.
Collapse folds genuinely separate postings that happen to share a name.**
## Dedup is a flag, not a delete
When mirrors of one posting are identified, the extras get a `duplicate_of`
pointing at the row we kept. Both [`/v1/jobs`](/docs/api-reference/jobs-list) and
[`/v1/jobs/feed`](/docs/api-reference/jobs-feed) exclude flagged rows
unconditionally. There is no parameter to turn it off and no way to ask for the
mirrors.
The flag is the important design decision. Marking is reversible and
non-destructive: a row identified as a duplicate is flagged, never deleted. If
the matching turns out to be wrong, the flag clears and the row comes back.
That is why `duplicate_of` is a column and not a `DELETE`. Any dedup system is
making a judgement call about identity, some of those calls are wrong, and a
`DELETE` makes being wrong permanent. A flag makes it a bug you can fix.
## The one place it leaks
Because dedup is invisible and automatic, it costs you nothing to think about —
with one exception, and it is about counting.
`total` counts distinct postings, not scraped rows. That is almost always what
you wanted. But it means this query does not mean what it looks like:
```bash
# NOT "how many jobs exist on Greenhouse".
curl -G https://api.hyperjobs.io/v1/jobs \
-H "Authorization: Bearer $HYPERJOBS_KEY" \
--data-urlencode "source=greenhouse" \
--data-urlencode "limit=1"
```
It counts postings where **Greenhouse won the dedup** — not every posting
Greenhouse carried. A role that exists on both Greenhouse and LinkedIn is
attributed to exactly one of them, and `?source=` only finds it under the winner.
Any per-source breakdown you build is a breakdown of surviving rows, and it is
worth saying so in whatever you publish.
The magnitude is small: the [coverage snapshot](/docs/reference/coverage) puts
flagged duplicates at roughly 1% of rows. Read `/v1/meta` for the live figure
rather than hardcoding a number that rots.
This is also why `/v1/health` and `/v1/meta` disagree on job counts. `/v1/health`
counts every row including duplicates and expired postings; `/v1/meta` counts
non-duplicate ones. Neither is wrong — the gap between them is the duplicates.
## Collapse is a different question
Some employers post one row per location. A retailer hiring baristas in 40 stores
files 40 real, distinct postings: 40 ids, 40 apply URLs, 40 legitimately separate
applications. None of them are duplicates. They are just repetitive.
`collapse=true` groups them:
```bash
curl -G https://api.hyperjobs.io/v1/jobs \
-H "Authorization: Bearer $HYPERJOBS_KEY" \
--data-urlencode "organization=starbucks" \
--data-urlencode "collapse=true"
```
The grouping key is the company and the title:
```sql
GROUP BY COALESCE(company_id, lower(organization_name)), title
```
Three things change in the response. `collapsed: true` appears at the top level.
Every job gains an `openings` count. And `total` **changes meaning** — it becomes
the number of groups, not the number of postings. Same field, different unit.
```json
{
"collapsed": true,
"total": 812,
"limit": 50,
"offset": 0,
"data": [
{
"id": "greenhouse:4820193",
"title": "Barista",
"openings": 40,
"company": { "slug": "starbucks", "name": "Starbucks" }
}
]
}
```
That shape is illustrative — the counts are there to show you the structure, not
to report anything.
## The representative row is one real posting
Each group is represented by whichever posting is most recent. It is not a
merged or synthesised record — it is one real posting standing in for the
group.
Everything posting-specific about the other 39 is **not in the response and not
reachable from it**: their ids, their locations, and critically their
`apply_url`s. There is no parameter to expand a group.
If a user needs to apply to the Denver store specifically, a collapsed response
cannot serve them. Query without `collapse` and group client-side instead.
The corollary bites hardest with location filters. `city=Denver` with
`collapse=true` finds groups whose *representative* is in Denver. A group that
contains a Denver posting but is represented by the Seattle one will not match.
The filter and the grouping do not compose the way you expect, because the
representative carries its own location, not the group's.
Collapse also groups on the exact title string. "Barista" and "Barista (Part
Time)" are two groups, not one. It is a `GROUP BY`, not a fuzzy match — which
makes it predictable, and means `openings` is a floor on how repetitive an
employer actually is.
### When to reach for it
**Do** collapse when the group is the unit you care about. Forty identical
"Barista" cards is a broken result page; one card reading "Barista · 40 openings"
is the correct presentation, in a single request. Same for "how many companies
are hiring for X" — without collapse you are counting postings, and one
multi-location employer dominates the count.
**Don't** collapse when the posting is the unit. You need every `apply_url`, or
location matters, or you are syncing a mirror — and
[the feed](/docs/api-reference/jobs-feed) has no `collapse` anyway, because sync
is posting-level by definition.
## Expiry, the third mechanism
Not deduplication, but it is the other reason a row you expected is missing.
Postings are expired by board-diffing: every re-scrape compares the source
board's current listing against what we hold, and anything that disappeared gets
an `expires_at`. The employer taking the job down is the signal. We do not guess.
Both endpoints exclude duplicates. Only `/v1/jobs` excludes expired rows. So
the index and the feed return **different populations**, and a mirror that
bootstraps from one and polls the other will drift.
```js
const isLive = (job) => !job.expires_at || new Date(job.expires_at) > new Date();
```
It is deliberate. A sync consumer has to *see* a posting expire in order to
expire its own copy — if the feed hid it, the row would just stop appearing and
you would have no way to tell "dead" from "unchanged". Route those rows to a
delete. [Syncing the feed](/docs/guides/feed-sync) has the loop.
## Which one is eating my rows?
Then `total` counts groups, and every row stands in for `openings` others.
Drop `collapse` to see postings.
Fetch it with [`/v1/jobs/{id}`](/docs/api-reference/jobs-retrieve) and check
`expires_at`. Present in the feed but missing from `/v1/jobs` is the
signature.
If a posting exists on two boards and you only ever see one, dedup picked a
winner. That is working correctly.
Far more likely than any of the above. Bad values do not
[400](/docs/errors) — they are ignored or match nothing. See
[filtering](/docs/guides/filtering) for the bisection procedure.
The full reference is [duplicates & collapsing](/docs/guides/deduplication).