The integration is working, the rows come back, and now the real question: where do they go, and will the answer still be right in six months?
Most feeds do not die dramatically. They drift. A column that used to be populated goes mostly null and nobody notices because nobody queries it. A site is dropped from the schedule during a cleanup and the dashboard quietly covers one fewer competitor. The table has a last_updated column that stopped updating. All three are invisible if the only thing you check is whether the pipeline ran.
Decide the cadence from the decision, not the data
The common instinct is to collect as often as the provider allows. That is backwards. The right frequency is set by how often somebody acts on the number.
If your repricing job runs once a day at six in the morning, collecting hourly buys you nothing except twenty-three discarded datasets and a bill. If a buyer checks a dashboard on Monday mornings, daily collection is already generous. Hourly makes sense for a narrow set of volatile, high-value SKUs, and almost never for the whole catalogue.
Set the collection to finish comfortably before the thing that consumes it starts. Comfortably means with enough margin that a slow run does not mean a stale decision — and a slow run is normal, because run duration is a property of how the sites behaved that day.
Three ways to get the rows out, and when each fits
Most platforms offer all three. They are not interchangeable.
| Method | Good for | The catch |
|---|---|---|
| Scheduled export to a file | Warehouse loads, BI tools, anything batch | You need a loader on the other end, and somebody has to notice when a file does not arrive |
| Pull the results over the API | Full control, custom transforms, backfills | Pagination and retry logic are now yours, and so is the scheduling |
| Webhook on completion | Triggering a downstream job the moment data is ready | Deliveries get lost. Pair with a poll, as in lesson four. |
Land it raw, then transform
Write what the API returned into a landing table, unmodified, with a collection timestamp and the run id. Then build your cleaned view on top of it. This is standard advice and it is standard for a reason: the day somebody asks why Tuesday's price looks wrong, you want to be able to see what actually arrived on Tuesday rather than what your transform made of it.
It also makes a transform bug recoverable. If the cleaned table is the only copy and your parser mangled a currency for a fortnight, that fortnight is gone. If the landing table is intact you re-run the transform and the fortnight comes back.
Keep the source URL on every row through every layer. It is the only thing that lets a human verify a number by opening a page, and it will be asked for.
Columns worth having that people leave out
Each of these exists to answer a question somebody will eventually ask.
- collected_at — when the value was read, not when the row was written. These differ and the difference is the age of your data.
- run_id — so a suspicious batch can be traced to one run and compared against its neighbours
- source_url — the exact page, so a number can be verified by a human in one click
- raw_price_text — what the page actually said, before parsing. Invaluable the first time a decimal separator goes wrong.
- A row per observation rather than an updated row per product, so that price history exists by construction rather than being added later at great expense
Monitor the shape of the data, not the health of the job
A green pipeline run tells you the code did not throw. It says nothing about whether the data is right, and the failures that matter are almost all of the second kind.
Three checks catch most of it. Row count against the trailing average, per site, with an alert on a meaningful drop — a feed halving is a far more common failure than a feed stopping. Null rate per column against its own baseline, because a column going empty is a redesign signal. And a freshness check that actually compares collected_at to now, rather than trusting that the scheduler ran.
The reason to do this per site rather than in aggregate is that aggregate numbers hide exactly the failures you care about. Forty sites holding steady and one going to zero is a two and a half per cent drop in the total, which no sensible threshold will catch, and it is also one competitor having completely vanished from your pricing decisions.
Worked example: what update-in-place destroys
Four observations of one SKU, stored two ways. The left-hand table is what almost everyone builds first, because it matches how a price feels — a current value. The right-hand one answers the question that always gets asked. The cost of the right-hand version is 365 rows a year per SKU per competitor, which for 1,200 SKUs and five competitors is about 2.2 million rows: unremarkable for any database built this decade, and more than any spreadsheet should be asked to hold.
| Observation | Update-in-place | Append-only |
|---|---|---|
| Monday, €229.00 | one row: €229.00 | row 1 |
| Tuesday, €229.00 | one row: €229.00 | row 2 |
| Wednesday, €189.00 | one row: €189.00 | row 3 |
| Thursday, €189.00 | one row: €189.00 | row 4 |
| "When did they drop it, and from what?" | unanswerable | Wednesday, from €229.00 |
| "How long has the promotion run?" | unanswerable | two days so far |
| Rows after a year of daily runs, one SKU | 1 | 365 |
What usually goes wrong
Six months in, a feed is either still trusted or quietly worked around. These are the decisions that determine which.
- Updating in place. The cheapest decision on day one and the most regretted, because the history cannot be reconstructed afterwards.
- Transforming before landing. If the raw response is discarded, a bug in the transform costs you the data instead of an afternoon of reprocessing.
- Setting cadence from what the provider allows. Set it from how often somebody actually changes a price; everything above that is pages you pay for and nobody reads.
- Monitoring the job instead of the shape. Row count and null rate per site against a trailing baseline catch the degradations. A green job catches almost nothing.
- Monitoring in aggregate. One competitor out of five dropping to zero moves the total by a fifth, which sits comfortably under any threshold you would have set.
- Leaving the coverage gaps undocumented. The feed does not include third-party marketplace sellers, or a market nobody configured, or anything behind a login — and the first person surprised by that will be the one presenting from it.
Write down what the feed does not cover
Every feed has holes: sites that block, pages that do not publish a price, categories that were never in scope. These are fine. What is not fine is that they live in one engineer's head.
Keep a short document next to the pipeline that lists which sites are in scope, which are excluded and why, and which fields are known to be unreliable where. It takes twenty minutes to write and it prevents the single most damaging thing a data feed can do, which is to be quietly interpreted as complete by somebody who was not there when it was built.
That is the course. The next one is about the thing that breaks this feed most often: the sites themselves changing, and occasionally deciding they would rather not be read.
Course four: keeping scrapers alive