The legal and ethical side, without the hand-waving
Most writing on this subject is either a confident "it's public data, you're fine" or a lawyer's refusal to say anything useful. Neither helps you decide whether to start. This course sets out the distinctions that actually matter — public versus logged-in, factual versus personal, considerate versus costly — so you can make a defensible call and write it down. It is written by practitioners and it is not legal advice.
What you will be able to do
- Separate the three distinct questions people collapse into "is scraping legal"
- Tell the difference between public data, terms-bound data and data behind a login, and why that line matters more than any other
- Recognise when personal data rules apply to a dataset you thought was about products
- Set rate limits and identification that you would be comfortable defending in writing
- Produce a one-page brief that gets a useful answer from your legal team instead of a reflexive no
The five lessons
Lesson two contains the distinction that decides most real cases. Lesson five is the deliverable — if you only read one, read that.
- 01Three questions hiding inside one"Is scraping legal" bundles access, copying and use into a single question. Separating them is most of the work.10 min read Read lesson 1 →
- 02Public data, terms of service, and what changes at the loginWhy a terms page you never agreed to is weaker than people think, why the one you did agree to is stronger, and where that leaves mobile app APIs.11 min read Read lesson 2 →
- 03Personal data, and why product scraping quietly becomes itPublic does not mean unregulated. The categories that catch people out, and the simplest way to stay clear of the whole problem.11 min read Read lesson 3 →
- 04Rate limits, robots.txt, and being easy to live withThe conduct half. What robots.txt is for, what rate to actually use, and why identifying yourself is the most underrated decision available.11 min read Read lesson 4 →
- 05What to put in front of your legal teamA one-page brief that gets a real answer, the three mistakes that guarantee a no, and the position we operate under ourselves.12 min read Read lesson 5 →
Who this is for
Written for
- Anyone who has been asked "are we allowed to do that?" and does not have a good answer ready
- Developers and analysts who want to make a defensible decision rather than an optimistic one
- Teams preparing to put a data collection project in front of legal, procurement or a customer's security review
Not written for
- Anyone needing an authoritative legal opinion. This is background so that the conversation with a qualified lawyer is a short one.
- Readers looking for a jurisdiction-by-jurisdiction reference. The principles here travel; the specifics do not.
Before you start
The questions that come up in the first meeting, every time.
No, and it cannot be. It is a practitioner's map of the questions that matter, written so that when you do speak to a qualified lawyer in your jurisdiction, you arrive with a specific description rather than "can we scrape?". That conversation is much shorter and much cheaper when the facts are already written down.
Start at lesson one
"Is scraping legal" bundles access, copying and use into a single question. Separating them is most of the work.