Most data collection projects that die in legal review do not die because the answer was no. They die because the question was unanswerable, so the safe response was no.
"Are we allowed to scrape competitor websites?" has no good answer. A lawyer hearing it has to imagine the worst version of what you might mean and advise against that. One page of specifics changes the entire conversation, and writing it takes about an hour.
The one-page brief
Eight sections, a few sentences each. Written before you start, not after someone asks.
- 1
Purpose, in one sentence
What business decision this data informs. "Weekly price position against eleven named competitors on the four hundred SKUs we actively compete on." Specific, bounded, obviously ordinary.
- 2
Sources, named
The actual domains. Not "competitor websites" — the list. Vagueness here reads as evasion even when it is laziness.
- 3
Fields, exhaustively
Every column you will store. This is where you demonstrate minimisation, and where a reviewer can see at a glance that there is no personal data in it.
- 4
Access method
Public pages, no login, nothing circumvented. State it plainly, because it is the question they most want answered.
- 5
Terms review
What each source's terms say about automated access, and the date you read them. A table with one row per source.
- 6
Rate and frequency
Requests per minute, concurrency, how often the run happens, and how that compares to the source's own traffic. Numbers, not adjectives.
- 7
Personal data position
Either "none collected, here is the field list" — much the better answer — or your lawful basis and retention period.
- 8
Who sees the output
Internal only, a specific team, or something customer-facing. Republishing changes the analysis completely and is the thing they will ask about if you do not say.
Three framings that guarantee a no
| What people say | What it sounds like | Say this instead |
|---|---|---|
| "It's all public data" | You have not thought about personal data or terms | "No login, no personal data, here is the field list" |
| "Everyone does it" | You have no position of your own | "This is standard market research; here is our conduct policy" |
| "They can't tell it's us" | You are relying on not being caught | "We identify ourselves and publish a contact address" |
Expect conditions, not a verdict
A good review rarely produces a clean yes. It produces a yes with conditions, and the conditions are usually reasonable and cheap: drop two fields, halve the frequency, exclude one source whose terms are explicit, add a retention limit, re-check the terms annually.
Treat that as the success case. A conditional approval is a documented, bounded authorisation that protects you and the project, and it is far more valuable than an unconditional shrug that nobody wrote down.
If the answer is a flat no, ask which of the eight sections caused it. Frequently it is one — a single source, or one field — and the project survives without it.
The position we operate under
Worth stating, because a vendor who will not tell you where their line is has not thought about it, and you should ask every vendor this question.
We collect from public pages only. No logins, no credential sharing, no circumvention of access controls. We rate-limit by default and back off on 429 and 503. We do not build datasets about individuals — no seller names, no review text, no contact details. We honour removal requests from site operators. And we tell customers plainly when a source cannot be collected rather than filling the gap with a plausible number, because a coverage gap is a known quantity and an invented row is not.
That last one is a data integrity commitment rather than a legal one, but it belongs in the same list. The projects that cause problems downstream are rarely the ones with a documented gap. They are the ones where somebody decided a gap looked bad.
Worked example: the same project, briefed badly and briefed well
Two versions of one request about the same five competitors. The left column is what legal teams usually receive. The right column is the same project described in terms somebody can actually sign off, line for line.
| What they usually get | What they can answer | Why the difference matters |
|---|---|---|
| "Can we scrape competitor websites?" | "We want to read 1,200 public product pages across five named sites, once a day, off-peak, at roughly 1 request per second." | The first has no boundary, so the safe answer is no. The second has five facts to check. |
| "We'd collect pricing data." | "Eight fields: title, GTIN, MPN, price, currency, availability, shipping cost, URL. No descriptions, no images, no reviews, no seller names." | A named field list is the single strongest thing in the brief. It proves the collection cannot substitute for the source. |
| "For analysis." | "Input to internal repricing. Not republished, not resold, not shown to customers or in marketing." | Use decides more of the answer than technique does. |
| "It's all public." | "Three sites need no account. Two show trade prices only behind a login, and we have excluded those two pending your view." | Flagging the hard case yourself is what makes the rest credible. |
| (no date) | "Terms of each site reviewed on 14 March. We re-review every six months and on any redesign." | An undated review is an assertion. A dated one is a control. |
| (no exit) | "We stop within one business day of any request from the site operator, and here is the mailbox that receives it." | Reversibility turns a permanent decision into a revocable one. |
What usually goes wrong
A no from legal is usually a response to the brief, not to the project.
- Asking the abstract question. "Is scraping legal" has no answer that helps anyone, and the only safe response to an unbounded question is no.
- Asking after the crawler is already running, which turns an approval into an incident review.
- Hiding the login-gated sites in the middle of the list instead of naming them as the open question.
- Promising a field list and then letting it grow, so the thing that was approved and the thing that runs drift apart within a quarter.
- Treating the answer as permanent. Conditions expire, sites get redesigned, terms change, and a review with no date on it stops being evidence of anything.
- Leaving no owner. If no named person re-reviews on a schedule, the brief is a document rather than a control.
The closing checklist
If you can tick all of these, you are in ordinary territory and you have the paperwork to show it.
- Purpose written in one sentence, and it does not sound evasive
- Public pages only, no account, nothing circumvented
- Field list is the minimum the purpose needs, and contains no personal data
- Terms reviewed per source, with dates recorded
- Rate and frequency stated as numbers, and justified against how often the data changes
- Identifiable user agent with a working contact address
- Someone qualified has read the brief and signed off, with any conditions written down
- A review date in the calendar — terms change, catalogues change, and so does your purpose
That is the end of the course. If you want to see what collected data actually looks like before committing to anything, every retailer page publishes real measured output — including the ones where we say plainly that nothing readable came back.
See real run output by retailer