Consuming a Managed data feed

This guide is for the engineering team on the receiving end — any system that reads a Managed data feed into its own store: a data lake, a data warehouse (Snowflake, BigQuery, Databricks), or a bespoke integration. It covers the pattern: a one-time snapshot to seed, then incremental polling to stay current. You pull on your own schedule with your own tooling; in Live mode we store nothing and your system holds the history.

New to the connector itself? Start with Managed data feed connector.

Would you rather we push each change to an endpoint you run, instead of pulling? See the Custom API endpoint.

What we give you

• A feed web address (base): https://<host>/api/feed/<your-slug>
• A consumer key (a bearer token)
• Which custody mode the feed runs in: Live (zero retention) or Cached

Authenticating

Every request carries the consumer key:

Authorization: Bearer <consumer-key>

If your tooling genuinely cannot send headers we can enable ?api_key=<consumer-key> on the feed — but the header is preferred, because keys in URLs end up in logs.

The two-step pattern

1. Snapshot once (initial load).

GET /api/feed/<slug>

Returns the full current table as { "data": [ ... ], "meta": { ... } }. Load every row in data into your landing zone.

• In Cached mode you can page a large snapshot with ?after=<record_key>&limit=<n>; keep going until meta.next_after is null.
• In Live mode the snapshot is the whole table in one response (no paging) — see Sizing below.

2. Poll for changes (keep current).

GET /api/feed/<slug>/changes

Returns only what changed, each record tagged "op": "upsert" or "op": "tombstone" (a removal). Upsert into your store on _record_key; treat tombstones as soft-deletes.

How you track your position depends on the custody mode:

The record shape

Each item in data is your mapped fields plus a small envelope:

Current state vs full history: by default the record key is the product code, so you hold one current row per product. If you need every WFX version as its own row (full history), ask us to set the feed's record key to the WFX record id.

Handling responses

Always honour retry_after on a 429.

Cadence (Live mode)

A Live feed serves at most once per its configured interval (default 15 minutes, set per dataset). Back-to-back reads get a 429 with retry_after. Match your polling job to that interval. Cached feeds have no cool-down — only a generous overall rate limit.

Sizing (Live mode)

Live mode reads WFX in full on each call and refuses to return a partial table — if the feed is too large it returns 422 rather than silently truncating. If you hit that, either scope the pipeline (fewer fields, or a product-category filter) or switch the feed to Cached, which holds an encrypted copy and serves instantly.

Examples

A minimal end-to-end with curl:

# 1. Initial load (snapshot)
curl -s -H "Authorization: Bearer $KEY" \
  "https://<host>/api/feed/<slug>" | jq '.data'

# 2a. Incremental — Cached: persist the cursor each time
curl -s -H "Authorization: Bearer $KEY" \
  "https://<host>/api/feed/<slug>/changes?cursor=$SAVED_CURSOR"
#     then store meta.next_cursor for next time

# 2b. Incremental — Live: no cursor, just poll on your interval
curl -s -H "Authorization: Bearer $KEY" \
  "https://<host>/api/feed/<slug>/changes"

Most tooling can drive this directly:

• Data lake — a scheduled notebook (Databricks) or a small script that snapshots into the raw zone, then appends /changes on each run.
• Warehouse — a REST source in Azure Data Factory, or a Fivetran / Airbyte custom connector, staging then merging on _record_key.
• Anything else — any scheduled job that can make an authenticated HTTP GET and store JSON.

There is also a /status endpoint (GET /api/feed/<slug>/status) for health and monitoring.

One thing to confirm with us

If your feed is fed by more than one pipeline (multiple datasets), use the dataset-specific address we give you rather than the bare slug — a bare Live address covering several datasets is intentionally rejected. We will tell you which applies to you.