# Duplicates & collapsing > Two different mechanisms that get confused — automatic cross-source dedup, and opt-in company+title collapsing. Source: https://hyperjobs.io/docs/guides/deduplication There are two distinct things in this API that both reduce the number of rows you get back, and they are constantly mistaken for each other. They solve different problems, they trigger differently, and only one of them is optional. | | **Dedup** | **Collapse** | | --- | --- | --- | | What it merges | The **same** posting mirrored across boards | **Different** postings sharing company + title | | Example | One Stripe job on Greenhouse *and* LinkedIn | 40 separate "Barista" postings, one per store | | Trigger | Automatic, always on | Opt-in via `?collapse=true` | | Can you disable it? | No | It's off by default | | Effect | The mirror rows are **excluded** | Rows are **grouped**, with an `openings` count | The one-line version: **dedup removes a posting you'd otherwise see twice. Collapse folds genuinely separate postings that happen to share a name.** ## Dedup: the same posting, seen twice A company posts a job to its ATS. An aggregator picks it up. A job board syndicates it. We scrape all three, so the same underlying role arrives as three rows from three sources. When we identify those as mirrors of one posting, the extras are marked with a `duplicate_of` pointing at the row we kept. **6,567 rows** currently carry that mark, out of 603,335. Both [`/v1/jobs`](/docs/api-reference/jobs-list) and [`/v1/jobs/feed`](/docs/api-reference/jobs-feed) exclude them unconditionally. There's no parameter to turn it off and no way to ask for the mirrors — every query runs against the deduplicated population, so you never need to think about it. Marking is **reversible and non-destructive**. A row identified as a duplicate is flagged, never deleted. If our matching turns out to be wrong, the flag clears and the row returns — which is why `duplicate_of` is a column and not a `DELETE`. Because dedup is invisible and automatic, its only practical consequence is on counting: `total` is a count of distinct postings, not of scraped rows. That's almost always what you wanted, but it means "how many Greenhouse jobs are there" via `?source=greenhouse` counts postings where Greenhouse won the dedup, not every posting Greenhouse ever carried. ## Collapsing Some employers post one row per location. A retailer hiring baristas in 40 stores files 40 real, distinct postings — 40 different `id`s, 40 different `apply_url`s, 40 legitimately separate applications. None of them are duplicates. They're just repetitive. `collapse=true` groups them: ```bash curl -G https://api.hyperjobs.io/v1/jobs \ -H "Authorization: Bearer $HYPERJOBS_KEY" \ --data-urlencode "organization=starbucks" \ --data-urlencode "collapse=true" ``` The grouping key is the company and the title: ```sql GROUP BY COALESCE(company_id, lower(organization_name)), title ``` `collapse=title,organization` is accepted and does the same thing — it names the grouping key explicitly rather than changing it. ### What changes in the response Added at the top level, set to `true`. This is how you know the shape you're holding is grouped. Added to every job. How many postings were folded into this group. **Changes meaning.** It becomes the number of **groups**, not the number of postings. Same field, different unit. ```json { "collapsed": true, "total": 812, "limit": 50, "offset": 0, "data": [ { "id": "greenhouse:4820193", "title": "Barista", "openings": 40, "company": { "slug": "starbucks", "name": "Starbucks" } } ] } ``` Collapsed responses carry **no `next_cursor`** — the keyset cursor doesn't exist for grouped rows. Page them with `limit`/`offset` instead; see [pagination](/docs/pagination). Each group is represented by whichever posting is most recent (`ORDER BY MAX(COALESCE(posted_at, scraped_at)) DESC`). It is not a merged or synthesised record. It's one real posting standing in for the group. Everything posting-specific on the other members is **not in the response and not reachable from it**: their `id`s, their locations, and critically their `apply_url`s. `openings: 40` tells you 39 other postings exist. It does not give you any of them, and there's no parameter to expand a group. If a user needs to apply to the Denver store specifically, a collapsed response cannot serve them. Query without `collapse` and group client-side instead. ### When to collapse **Do** collapse when the group is the unit you care about: - **Job-board and search UIs.** Forty identical "Barista" cards is a broken result page. One card reading "Barista · 40 openings" is the correct presentation, and it's a single request. - **"How many companies are hiring for X."** Without collapse you're counting postings and a single multi-location employer dominates the count. - **Market-shape analysis.** Employer × role is usually the honest denominator; postings over-weight whoever files most granularly. **Don't** collapse when the posting is the unit you care about: - **You need every `apply_url`.** Collapsing hides them with no way back. - **Location matters.** The representative row carries its own location, not the group's. Filtering `city=Denver` with `collapse=true` finds groups whose *representative* is in Denver — a group with a Denver posting can be represented by the Seattle one and won't match. - **You're syncing a mirror.** [Feed sync](/docs/guides/feed-sync) is posting-level by definition, and the feed has no `collapse` anyway. Collapse groups on the *exact* title string. "Barista" and "Barista (Part Time)" are two groups, not one. It's a `GROUP BY`, not a fuzzy match — which makes it predictable, and means `openings` is a floor on how repetitive an employer actually is. ## Expiry, the third mechanism Not deduplication, but it's the other reason a row you expected isn't there. Postings are expired by **board-diffing**: every re-scrape compares the source board's current listing against what we hold, and anything that disappeared gets an `expires_at`. **8,482 rows** are currently expired. A posting can also carry an explicit expiry date from its source. [`/v1/jobs`](/docs/api-reference/jobs-list) and [`/v1/jobs/feed`](/docs/api-reference/jobs-feed) both exclude duplicates **and** expired postings. (A pre-release draft served expired rows on the feed but not on `/v1/jobs` — an asymmetry that would have made a mirror bootstrapped from one and polled from the other drift. It never shipped.) That leaves a sync consumer needing a way to *see* expiry — you can't expire your copy of a row that just stops appearing. That's its own endpoint: [`/v1/jobs/expired`](/docs/api-reference/jobs-expired) streams `{id, expired_at}` pairs for exactly this. [Syncing the feed](/docs/guides/feed-sync) replays those rows as deletes. ## Which one is eating my rows? Then `total` counts groups, not postings, and every row is standing in for `openings` others. Drop `collapse` to see postings. Fetch it directly with [`/v1/jobs/{id}`](/docs/api-reference/jobs-retrieve) and check `expires_at` — an expired posting stays retrievable by id while the row is still in the pool. Appearing in [`/v1/jobs/expired`](/docs/api-reference/jobs-expired) is the definitive signal. If a posting exists on two boards and you only ever see one, dedup picked a winner. That's working correctly — there's no opt-out, `?source=` will only find it under the source that won, and fetching a folded row's id returns a [404](/docs/errors#404-not-found). Far more likely than any of the above. A bad **enum** value is a [400](/docs/errors) that names the allowed values — but the open vocabularies (`skill`, `source`, `city`, …) still match nothing on an unknown value. See [filtering](/docs/guides/filtering). ## Next Everything that's applied to a query, including the two you can't turn off. The endpoint reference, `collapse` included.