A correct, well-matched, well-monitored price feed that lands somewhere nobody looks has the same business value as no feed at all. This lesson is short because the principle is simple: deliver into the tool where the pricing decision is already being made, not into a new tool you are hoping people will adopt.
In practice that is almost always a spreadsheet, and there is no shame in it.
Four routes
| Route | Good for | The catch |
|---|---|---|
| Spreadsheet | A pricing manager who already works in one | Breaks at a few hundred thousand rows; versions multiply |
| Scheduled CSV or Excel drop | Feeding an existing internal job | Somebody has to own the filename convention and the failure case |
| REST API | Your own application or an automation tool | Requires a developer once; then it is the lowest-maintenance option |
| Warehouse table | Teams already running BI | Highest setup cost, and only pays off if the dashboards get opened |
The column contract
Whatever the route, the single most valuable property of a feed is that its columns do not move. Downstream jobs — a spreadsheet formula, an ETL step, a repricing script — are written against column names and positions, and they break silently when those change. A column that disappears usually produces a blank rather than an error, and a blank price reads as free.
So pin the contract. Every run returns the same columns, in the same order, with the same names, whether or not a particular page had a value for them. An empty cell is a valid answer and must be distinguishable from a missing column. Adding a new column at the end is safe; renaming or reordering is not, and should be treated as a versioned change with a deprecation window, however informal.
Fields the consumer of the feed will ask for within a week
Include them from the start; retrofitting them means reprocessing history.
- The observation timestamp, not just the date — so a reader can tell a morning sample from an evening one
- The competitor name as a clean key, not as a URL to be parsed
- Your own SKU alongside their identifier, so no downstream join is needed
- The match tier, so a reader can see which rows are EAN-certain and which are a judgement call
- The source URL, so a surprising number can be checked by clicking it
Worked example: a column contract in full
The whole contract for a morning sheet — every column, in a fixed order, emitted on every run whether or not there is a value for it. The right-hand column is the part that matters: it is what everything downstream is entitled to assume. Guarantees five and six exist because a spreadsheet treats an empty cell as zero in most arithmetic, and a zero price reads as free.
| # | Column | Guarantee |
|---|---|---|
| 1 | captured_at | Always present, UTC, ISO 8601. Never the time the sheet was opened. |
| 2 | competitor_key | A stable slug, never the display name. Renaming a shop must not break a formula. |
| 3 | your_sku | Your identifier, not theirs. This is the join key for everything downstream. |
| 4 | match_tier | 1 to 4, present even when the match is certain. |
| 5 | list_price | Empty if the page showed no struck-through price. Empty, not zero. |
| 6 | sale_price | Empty if there was no promotion. Empty, not a copy of list_price. |
| 7 | currency | ISO code, on every row, including rows from your home market. |
| 8 | availability_text | Verbatim from the page, uninterpreted. |
| 9 | source_url | The page the row came from, so an argument can be settled in one click. |
What usually goes wrong
Delivery is the stage where a technically correct feed becomes a feed nobody uses.
- Emitting a column only when it has data. The header row shifts, every formula in the sheet moves one column left, and the error stays silent for a week.
- Building a dashboard first. It gets opened on launch day and at the quarterly review, while the pricing decision carries on happening in a spreadsheet nobody told you about.
- Delivering the competitor's identifier and not yours. Every consumer then has to redo the join, and each of them will do it slightly differently.
- Using zero as the empty value. A zero price is a 100% undercut, and most repricing rules will act on it before anyone notices.
- Overwriting yesterday's file. The first time somebody asks "when did they drop it?", the only honest answer is "sometime in the last month".
- Emailing the whole catalogue every morning. It stops being opened in week three. Send the rows that moved and link to the full sheet.
Ship one view, not a platform
The strongest first deliverable we have seen is a single sheet, refreshed every morning, with one row per in-scope product and one column per competitor, cells coloured by whether you are above or below. No navigation, no filters, no login. It is unglamorous and people open it.
The elaborate version — the dashboard with the drill-downs and the trend charts — is the right second deliverable, once the daily sheet has proven somebody cares. Built first, it is usually the thing that makes the project look expensive and optional at the same time.
Last lesson: turning a column of competitor prices into a decision that does not destroy your margin.
Lesson 8: from price data to repricing rules