Duplicates & collapsing

There are two distinct things in this API that both reduce the number of rows you get back, and they are constantly mistaken for each other. They solve different problems, they trigger differently, and only one of them is optional.

The one-line version: dedup removes a posting you'd otherwise see twice. Collapse folds genuinely separate postings that happen to share a name.

Dedup: the same posting, seen twice#

A company posts a job to its ATS. An aggregator picks it up. A job board syndicates it. We scrape all three, so the same underlying role arrives as three rows from three sources.

When we identify those as mirrors of one posting, the extras are marked with a duplicate_of pointing at the row we kept. 6,567 rows currently carry that mark, out of 603,335.

Both /v1/jobs and /v1/jobs/feed exclude them unconditionally. There's no parameter to turn it off and no way to ask for the mirrors — every query runs against the deduplicated population, so you never need to think about it.

Because dedup is invisible and automatic, its only practical consequence is on counting: total is a count of distinct postings, not of scraped rows. That's almost always what you wanted, but it means "how many Greenhouse jobs are there" via ?source=greenhouse counts postings where Greenhouse won the dedup, not every posting Greenhouse ever carried.

Collapsing#

Some employers post one row per location. A retailer hiring baristas in 40 stores files 40 real, distinct postings — 40 different ids, 40 different apply_urls, 40 legitimately separate applications. None of them are duplicates. They're just repetitive.

collapse=true groups them:

curl -G https://api.hyperjobs.io/v1/jobs \
  -H "Authorization: Bearer $HYPERJOBS_KEY" \
  --data-urlencode "organization=starbucks" \
  --data-urlencode "collapse=true"

The grouping key is the company and the title:

GROUP BY COALESCE(company_id, lower(organization_name)), title

collapse=title,organization is accepted and does the same thing — it names the grouping key explicitly rather than changing it.

What changes in the response#

{
  "collapsed": true,
  "total": 812,
  "limit": 50,
  "offset": 0,
  "data": [
    {
      "id": "greenhouse:4820193",
      "title": "Barista",
      "openings": 40,
      "company": { "slug": "starbucks", "name": "Starbucks" }
    }
  ]
}

When to collapse#

Do collapse when the group is the unit you care about:

• Job-board and search UIs. Forty identical "Barista" cards is a broken result page. One card reading "Barista · 40 openings" is the correct presentation, and it's a single request.
• "How many companies are hiring for X." Without collapse you're counting postings and a single multi-location employer dominates the count.
• Market-shape analysis. Employer × role is usually the honest denominator; postings over-weight whoever files most granularly.

Don't collapse when the posting is the unit you care about:

• You need every apply_url. Collapsing hides them with no way back.
• Location matters. The representative row carries its own location, not the group's. Filtering city=Denver with collapse=true finds groups whose representative is in Denver — a group with a Denver posting can be represented by the Seattle one and won't match.
• You're syncing a mirror. Feed sync is posting-level by definition, and the feed has no collapse anyway.

Expiry, the third mechanism#

Not deduplication, but it's the other reason a row you expected isn't there.

Postings are expired by board-diffing: every re-scrape compares the source board's current listing against what we hold, and anything that disappeared gets an expires_at. 8,482 rows are currently expired. A posting can also carry an explicit expiry date from its source.

Which one is eating my rows?#

1. Is `collapsed: true` in the response?

   Then total counts groups, not postings, and every row is standing in for openings others. Drop collapse to see postings.

2. Is the row expired?

   Fetch it directly with /v1/jobs/{id} and check expires_at — an expired posting stays retrievable by id while the row is still in the pool. Appearing in /v1/jobs/expired is the definitive signal.

3. Is the row a duplicate?

   If a posting exists on two boards and you only ever see one, dedup picked a winner. That's working correctly — there's no opt-out, ?source= will only find it under the source that won, and fetching a folded row's id returns a 404.

4. Or is it just a filter?

   Far more likely than any of the above. A bad enum value is a 400 that names the allowed values — but the open vocabularies (skill, source, city, …) still match nothing on an unknown value. See filtering.

Next#