# Syncing the feed
> The incremental-sync pattern — one full snapshot, then poll the feed, expired, and modified endpoints.
Source: https://hyperjobs.io/docs/guides/feed-sync
Keeping a local copy of the dataset current is a two-phase job: take one full
snapshot from [`/v1/jobs`](/docs/api-reference/jobs-list), then keep it fresh by
polling three feeds — [`/v1/jobs/feed`](/docs/api-reference/jobs-feed) for new
postings, [`/v1/jobs/expired`](/docs/api-reference/jobs-expired) for deletions,
and [`/v1/jobs/modified`](/docs/api-reference/jobs-modified) for in-place
changes. Each covers one kind of mutation, so together they replay everything
that happened to the dataset since your last poll.
All three page the same way: follow `next_cursor` until it comes back `null`,
and you have everything in the window. Rows arrive **ascending by event time**
(index time, expiry time, modification time respectively), which makes
checkpointing natural — the last row you processed is your new high-water mark.
## Don't re-walk the index
The obvious approach — page through `/v1/jobs` on a schedule and diff it against
what you have — fails on cost alone. At `limit=200`, one full walk of ~597,000
postings is about 3,000 requests. Repeat it hourly and you're spending over 2.1
million requests a month — more than four times a Growth plan's entire
[quota](/docs/rate-limits), to re-download a dataset that changed by a fraction
of a percent. And a diff can tell you *that* a row disappeared, but not whether
it expired or you mis-paged.
Polling all three feeds every 15 minutes costs under 9,000 requests a month at
steady state, fits inside a Starter plan with room to spare, and tells you
explicitly what changed and how.
## The shape
Page the index with `cursor` — keyset paging is a consistent walk, so
unlike offset paging, nothing shifts under you mid-snapshot — and
store every row. This is the only time you walk it. (A filtered mirror can
bootstrap the same way with [filters](/docs/guides/filtering) applied.)
New postings since the window opened, ascending by when we indexed them.
Page with `cursor` until `next_cursor` is `null` — busy windows span
multiple pages, and none of the rows are lost.
The purge feed: `{id, expired_at}` pairs for every posting that expired in
the window. Delete (or mark dead) each `id` on your side. Expiry is driven
by board-diffing — a posting that disappears from its source board on a
re-scrape is marked expired.
Full Job rows whose tracked fields changed — salary, title, apply URL,
locations, and so on. `modified_fields` on each row says exactly what
changed. Upsert by `id`.
Everything keys on `id`, and every window overlaps by design, so you re-see
rows. Upserting makes that harmless.
## Choosing `since`
The rule: **always poll on a window larger than your poll interval.**
An exactly-matched window — polling every hour with `since=1h` — assumes your
clock, our clock, and ingestion latency all agree perfectly. They don't. A
posting that lands a few seconds after your window closes but is timestamped a
few seconds before it falls into the gap between two polls and is never seen
again. Overlap closes that gap, and because you upsert by `id`, the redundancy
costs you nothing but a slightly larger response.
| Poll interval | `since` | Overlap |
| --- | --- | --- |
| 5 min | `15m` | 10 min |
| 15 min | `1h` | 45 min |
| 1 h | `6h` | 5 h |
| 6 h | `1d` | 18 h |
| 1 d | `7d` | 6 d |
`m` means **minutes** — `since=15m` is a quarter of an hour, and sub-hourly
polling no longer has to over-fetch a full hour. The units are `m`, `h`, `d`;
anything else, `since=banana` included, is a
[400](/docs/errors) rather than a silently applied 24-hour default. See
[the feed reference](/docs/api-reference/jobs-feed#query-parameters).
The feed windows on **when we indexed the posting**, not its posting date. A
posting backdated by its source board still enters the feed the moment we first
see it, so nothing slips behind your window by carrying an old `posted_at`.
## Resumable checkpoints
`since` is the right tool for a poller that never stops. For anything that can
crash, deploy, or pause, use the absolute forms instead — `created_gte` on the
feed, `expired_gte` on expired, `modified_gte` on modified — and persist a
high-water mark. Because rows arrive ascending, the checkpoint is just the
timestamp of the last row you processed:
```js
let checkpoint = await store.maxCreatedAt(); // e.g. "2026-07-16T09:48:15.000Z"
for await (const job of drain("/v1/jobs/feed", { created_gte: checkpoint, limit: 1000 })) {
store.upsert(job);
checkpoint = job.created_at; // ascending order → last row = new high-water mark
}
await store.saveCheckpoint(checkpoint);
```
Restart after a week offline and the same code catches up from where it
stopped — no window arithmetic, no gap.
## A complete worker
Runnable as-is on Node 18+ (`node sync.mjs`). Swap `store` for your database.
```js title="sync.mjs"
const BASE = "https://api.hyperjobs.io";
const KEY = process.env.HYPERJOBS_KEY;
const POLL_MS = 15 * 60_000;
const SINCE = "1h"; // must exceed POLL_MS — see "Choosing since"
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// Replace with your database. Upsert on `id`: the overlapping window guarantees
// you re-see rows, and that redundancy is what makes the sync safe.
const store = {
jobs: new Map(),
upsert(job) {
this.jobs.set(job.id, job);
},
expire(id) {
this.jobs.delete(id);
},
};
async function api(path, params) {
const res = await fetch(`${BASE}${path}?${new URLSearchParams(params)}`, {
headers: { Authorization: `Bearer ${KEY}` },
});
if (res.status === 429) {
// Per-minute 429s carry Retry-After in seconds. (A monthly-quota 429
// doesn't — but sleeping and retrying that one won't save you anyway.)
await sleep(Number(res.headers.get("retry-after") ?? 5) * 1000);
return api(path, params);
}
if (!res.ok) throw new Error(`${path} → ${res.status}`);
return res.json();
}
// Every list endpoint pages the same way: follow next_cursor until null.
async function* drain(path, params) {
let cursor = null;
do {
const page = await api(path, cursor ? { ...params, cursor } : params);
yield* page.data;
cursor = page.next_cursor;
} while (cursor);
}
async function bootstrap() {
// Keyset paging: a consistent walk of the index, no drift, no double-counting.
for await (const job of drain("/v1/jobs", { limit: 200 })) store.upsert(job);
}
async function poll() {
// 1. New postings, ascending by index time.
for await (const job of drain("/v1/jobs/feed", { since: SINCE, limit: 1000 })) {
store.upsert(job);
}
// 2. Deletions. The payload is just ids + timestamps, so the pages are big.
for await (const row of drain("/v1/jobs/expired", { since: SINCE, limit: 5000 })) {
store.expire(row.id);
}
// 3. In-place changes — full Job rows; modified_fields says what changed.
for await (const job of drain("/v1/jobs/modified", { since: SINCE, limit: 1000 })) {
store.upsert(job);
}
}
await bootstrap();
console.log(`bootstrapped ${store.jobs.size} jobs`);
for (;;) {
await poll();
console.log(`polled → ${store.jobs.size} live`);
await sleep(POLL_MS);
}
```
The feed excludes expired postings and duplicates, the same as `/v1/jobs` —
each endpoint carries one kind of mutation (inserts, deletes, updates), so
nothing here inspects `expires_at` by hand. The feed never carries dead rows,
and `count === limit` doesn't mean data loss — it means there's another page,
and `next_cursor` takes you there.
## Notes on running this for real
**Don't cache the feed.** There's no `Cache-Control`, `ETag`, or `Last-Modified`
header on any endpoint, including the one that would benefit most. A CDN or a
`fetch` cache won't help you; your local copy *is* the cache.
**Filter after, not during.** The feed takes no filters — no `skill`, no
`country`, no `source`. If you only want a slice of the dataset, either pull
the window and filter client-side, or poll [`/v1/jobs`](/docs/api-reference/jobs-list)
with [filters](/docs/guides/filtering) plus `created_gte` checkpoints. There's
no filtered feed.
**Leave descriptions off.** `description_format` omits descriptions by default,
and a 1,000-row feed page with them turned on is the difference between a 20KB
and a multi-megabyte response on every poll. Fetch them on demand with
[`/v1/jobs/{id}`](/docs/api-reference/jobs-retrieve).
**Watch the rate-limit headers.** Every authed response carries
`X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`, so a
polite worker can pace itself before the 429 instead of after — see
[rate limits](/docs/rate-limits).
**Divide the rate if you shard.** Limits are per account, not per key, so four
workers on a Growth plan get 75 requests/min each, not 300.
## Next
The endpoint reference — every parameter and response field.
The purge feed — ids and expiry timestamps, in bulk.
The update feed — changed rows and their `modified_fields`.
Budgeting a poll interval against your plan.