Syncing the feed

Keeping a local copy of the dataset current is a two-phase job: take one full snapshot from /v1/jobs, then keep it fresh by polling three feeds — /v1/jobs/feed for new postings, /v1/jobs/expired for deletions, and /v1/jobs/modified for in-place changes. Each covers one kind of mutation, so together they replay everything that happened to the dataset since your last poll.

All three page the same way: follow next_cursor until it comes back null, and you have everything in the window. Rows arrive ascending by event time (index time, expiry time, modification time respectively), which makes checkpointing natural — the last row you processed is your new high-water mark.

Don't re-walk the index#

The obvious approach — page through /v1/jobs on a schedule and diff it against what you have — fails on cost alone. At limit=200, one full walk of ~597,000 postings is about 3,000 requests. Repeat it hourly and you're spending over 2.1 million requests a month — more than four times a Growth plan's entire quota, to re-download a dataset that changed by a fraction of a percent. And a diff can tell you that a row disappeared, but not whether it expired or you mis-paged.

Polling all three feeds every 15 minutes costs under 9,000 requests a month at steady state, fits inside a Starter plan with room to spare, and tells you explicitly what changed and how.

The shape#

1. Bootstrap once with /v1/jobs

   Page the index with cursor — keyset paging is a consistent walk, so unlike offset paging, nothing shifts under you mid-snapshot — and store every row. This is the only time you walk it. (A filtered mirror can bootstrap the same way with filters applied.)

2. Poll /v1/jobs/feed on an interval

   New postings since the window opened, ascending by when we indexed them. Page with cursor until next_cursor is null — busy windows span multiple pages, and none of the rows are lost.

3. Apply /v1/jobs/expired as deletions

   The purge feed: {id, expired_at} pairs for every posting that expired in the window. Delete (or mark dead) each id on your side. Expiry is driven by board-diffing — a posting that disappears from its source board on a re-scrape is marked expired.

4. Apply /v1/jobs/modified as upserts

   Full Job rows whose tracked fields changed — salary, title, apply URL, locations, and so on. modified_fields on each row says exactly what changed. Upsert by id.

Everything keys on id, and every window overlaps by design, so you re-see rows. Upserting makes that harmless.

Choosing since#

The rule: always poll on a window larger than your poll interval.

An exactly-matched window — polling every hour with since=1h — assumes your clock, our clock, and ingestion latency all agree perfectly. They don't. A posting that lands a few seconds after your window closes but is timestamped a few seconds before it falls into the gap between two polls and is never seen again. Overlap closes that gap, and because you upsert by id, the redundancy costs you nothing but a slightly larger response.

The feed windows on when we indexed the posting, not its posting date. A posting backdated by its source board still enters the feed the moment we first see it, so nothing slips behind your window by carrying an old posted_at.

Resumable checkpoints#

since is the right tool for a poller that never stops. For anything that can crash, deploy, or pause, use the absolute forms instead — created_gte on the feed, expired_gte on expired, modified_gte on modified — and persist a high-water mark. Because rows arrive ascending, the checkpoint is just the timestamp of the last row you processed:

let checkpoint = await store.maxCreatedAt(); // e.g. "2026-07-16T09:48:15.000Z"
 
for await (const job of drain("/v1/jobs/feed", { created_gte: checkpoint, limit: 1000 })) {
  store.upsert(job);
  checkpoint = job.created_at; // ascending order → last row = new high-water mark
}
 
await store.saveCheckpoint(checkpoint);

Restart after a week offline and the same code catches up from where it stopped — no window arithmetic, no gap.

A complete worker#

Runnable as-is on Node 18+ (node sync.mjs). Swap store for your database.

const BASE = "https://api.hyperjobs.io";
const KEY = process.env.HYPERJOBS_KEY;
 
const POLL_MS = 15 * 60_000;
const SINCE = "1h"; // must exceed POLL_MS — see "Choosing since"
 
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
 
// Replace with your database. Upsert on `id`: the overlapping window guarantees
// you re-see rows, and that redundancy is what makes the sync safe.
const store = {
  jobs: new Map(),
  upsert(job) {
    this.jobs.set(job.id, job);
  },
  expire(id) {
    this.jobs.delete(id);
  },
};
 
async function api(path, params) {
  const res = await fetch(`${BASE}${path}?${new URLSearchParams(params)}`, {
    headers: { Authorization: `Bearer ${KEY}` },
  });
  if (res.status === 429) {
    // Per-minute 429s carry Retry-After in seconds. (A monthly-quota 429
    // doesn't — but sleeping and retrying that one won't save you anyway.)
    await sleep(Number(res.headers.get("retry-after") ?? 5) * 1000);
    return api(path, params);
  }
  if (!res.ok) throw new Error(`${path} → ${res.status}`);
  return res.json();
}
 
// Every list endpoint pages the same way: follow next_cursor until null.
async function* drain(path, params) {
  let cursor = null;
  do {
    const page = await api(path, cursor ? { ...params, cursor } : params);
    yield* page.data;
    cursor = page.next_cursor;
  } while (cursor);
}
 
async function bootstrap() {
  // Keyset paging: a consistent walk of the index, no drift, no double-counting.
  for await (const job of drain("/v1/jobs", { limit: 200 })) store.upsert(job);
}
 
async function poll() {
  // 1. New postings, ascending by index time.
  for await (const job of drain("/v1/jobs/feed", { since: SINCE, limit: 1000 })) {
    store.upsert(job);
  }
 
  // 2. Deletions. The payload is just ids + timestamps, so the pages are big.
  for await (const row of drain("/v1/jobs/expired", { since: SINCE, limit: 5000 })) {
    store.expire(row.id);
  }
 
  // 3. In-place changes — full Job rows; modified_fields says what changed.
  for await (const job of drain("/v1/jobs/modified", { since: SINCE, limit: 1000 })) {
    store.upsert(job);
  }
}
 
await bootstrap();
console.log(`bootstrapped ${store.jobs.size} jobs`);
 
for (;;) {
  await poll();
  console.log(`polled → ${store.jobs.size} live`);
  await sleep(POLL_MS);
}

Notes on running this for real#

Don't cache the feed. There's no Cache-Control, ETag, or Last-Modified header on any endpoint, including the one that would benefit most. A CDN or a fetch cache won't help you; your local copy is the cache.

Filter after, not during. The feed takes no filters — no skill, no country, no source. If you only want a slice of the dataset, either pull the window and filter client-side, or poll /v1/jobs with filters plus created_gte checkpoints. There's no filtered feed.

Leave descriptions off. description_format omits descriptions by default, and a 1,000-row feed page with them turned on is the difference between a 20KB and a multi-megabyte response on every poll. Fetch them on demand with /v1/jobs/{id}.

Watch the rate-limit headers. Every authed response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, so a polite worker can pace itself before the 429 instead of after — see rate limits.

Divide the rate if you shard. Limits are per account, not per key, so four workers on a Growth plan get 75 requests/min each, not 300.

Next#