[{"data":1,"prerenderedAt":223},["ShallowReactive",2],{"learn-lesson-web-scraping-legal-and-ethical-personal-data-and-gdpr":3},{"course":4,"lesson":66,"index":196,"outline":197,"prev":221,"next":222},{"slug":5,"order":6,"level":7,"time":8,"card_text":9,"seo":10,"hero":16,"outcomes":26,"who":34,"syllabus":45,"faq":48,"lessonCount":65},"web-scraping-legal-and-ethical",6,"No legal background assumed","5 lessons, about 55 minutes","The question that stops projects: are we allowed to do this? Public data versus terms of service, what changes the moment you log in, where personal data rules bite, what good conduct actually looks like, and how to write the one page your legal team needs.",{"title":11,"description":12,"keywords":13,"og_title":14,"og_description":15},"Web Scraping Legal and Ethical: A Free 5-Lesson Course","Is web scraping legal? Public data, terms of service, logins, personal data and GDPR, robots.txt and rate limits, and how to brief your legal team. Written for practitioners, not lawyers. Free, ungated.","is web scraping legal, web scraping gdpr, terms of service scraping, robots txt, public data scraping, web scraping ethics, scraping personal data, scraping compliance","A free course on the legal and ethical side of web scraping","Five written lessons on what you can collect, what changes when you log in, where personal data rules apply, and how to answer your legal team.",{"badge":17,"title":18,"subtitle":19,"cta_primary":20,"cta_secondary":23},"Course six","The legal and ethical side, without the hand-waving","Most writing on this subject is either a confident \"it's public data, you're fine\" or a lawyer's refusal to say anything useful. Neither helps you decide whether to start. This course sets out the distinctions that actually matter — public versus logged-in, factual versus personal, considerate versus costly — so you can make a defensible call and write it down. It is written by practitioners and it is not legal advice.",{"label":21,"url":22},"Start with lesson one","/learn/web-scraping-legal-and-ethical/is-web-scraping-legal",{"label":24,"url":25},"See what we collect and publish","/custom-scrapers",{"title":27,"items":28},"What you will be able to do",[29,30,31,32,33],"Separate the three distinct questions people collapse into \"is scraping legal\"","Tell the difference between public data, terms-bound data and data behind a login, and why that line matters more than any other","Recognise when personal data rules apply to a dataset you thought was about products","Set rate limits and identification that you would be comfortable defending in writing","Produce a one-page brief that gets a useful answer from your legal team instead of a reflexive no",{"title":35,"for_title":36,"for":37,"not_title":41,"not_for":42},"Who this is for","Written for",[38,39,40],"Anyone who has been asked \"are we allowed to do that?\" and does not have a good answer ready","Developers and analysts who want to make a defensible decision rather than an optimistic one","Teams preparing to put a data collection project in front of legal, procurement or a customer's security review","Not written for",[43,44],"Anyone needing an authoritative legal opinion. This is background so that the conversation with a qualified lawyer is a short one.","Readers looking for a jurisdiction-by-jurisdiction reference. The principles here travel; the specifics do not.",{"title":46,"intro":47},"The five lessons","Lesson two contains the distinction that decides most real cases. Lesson five is the deliverable — if you only read one, read that.",{"badge":49,"title":50,"description":51,"items":52},"FAQ","Before you start","The questions that come up in the first meeting, every time.",[53,56,59,62],{"title":54,"description":55},"Is this legal advice?","No, and it cannot be. It is a practitioner's map of the questions that matter, written so that when you do speak to a qualified lawyer in your jurisdiction, you arrive with a specific description rather than \"can we scrape?\". That conversation is much shorter and much cheaper when the facts are already written down.",{"title":57,"description":58},"So is web scraping legal or not?","The question is too coarse to have an answer. Collecting publicly posted prices, respectfully, for market analysis sits in a very different place from harvesting personal profiles from behind a login. Lesson one breaks the question into the three separate ones it actually contains.",{"title":60,"description":61},"Does robots.txt have legal force?","It is a convention rather than a contract, and treating it as either irrelevant or binding both miss the point. Lesson four covers what it is for and why ignoring it is a bad idea regardless of what a court would say about it.",{"title":63,"description":64},"What does Scrapewise itself refuse to do?","We do not collect from behind logins, we do not build personal profile datasets, and we rate-limit by default. Lesson five includes the acceptable-use position we actually operate under, because a vendor who will not tell you where their line is has not thought about it.",5,{"slug":67,"nav_title":68,"title":69,"summary":70,"time":71,"needs_account":72,"seo":73,"blocks":77,"takeaways":187,"next_step":192},"personal-data-and-gdpr","Personal data","Personal data, and why product scraping quietly becomes it","Public does not mean unregulated. The categories that catch people out, and the simplest way to stay clear of the whole problem.","11 min",false,{"title":74,"description":75,"keywords":76},"Web Scraping and GDPR: When Product Data Becomes Personal Data","Why publicly available personal data is still regulated, the fields that turn a product dataset into a personal one — seller names, reviews, marketplace listings — and how to avoid the problem entirely.","web scraping gdpr, scraping personal data, gdpr public data, scraping reviews gdpr, marketplace seller data, data minimisation",[78,83,115,120,128,135,167,177],{"type":79,"paragraphs":80},"prose",[81,82],"The most common surprise in this whole area: publicly available personal data is still personal data. The fact that someone posted their name on a public page does not take it outside data protection law, and the intuition that \"public means fair game\" is simply wrong under the GDPR and its equivalents.","This matters for product scraping specifically, because personal data arrives in product datasets by accident far more often than by design.",{"type":84,"title":85,"headers":86,"rows":90},"table","Fields that look commercial and are not",[87,88,89],"Field","Why it is personal data","Safer option",[91,95,99,103,107,111],[92,93,94],"Marketplace seller name","Very often an individual or a sole trader","Keep a hashed seller id, or drop it",[96,97,98],"Review text and author","Written by an identifiable person, and sometimes revealing","Keep the rating count and average; drop the text",[100,101,102],"Q&A on a product page","Same — identifiable individuals","Drop entirely",[104,105,106],"Seller contact details","Directly identifying","Never collect",[108,109,110],"\"Sold by\" on a retailer listing","Corporate when it is a company, personal when it is not","Check which, or drop",[112,113,114],"Price, stock, SKU, title","Not personal data","Collect freely",{"type":116,"variant":117,"title":118,"text":119},"callout","warning","The accidental dataset","A price monitoring project aimed squarely at products ends up holding thousands of individual sellers' names and trading histories because \"sold by\" was one of the columns. Nobody decided to build a dataset about people. One field did it, and now the project is inside a regime it was never scoped for. This is the single most common way teams end up non-compliant without any bad intent.",{"type":79,"title":121,"paragraphs":122},"What the rules actually ask of you",[123,124,125,126,127],"Simplified considerably, and for the EU and UK regime specifically, there are four obligations that bite on collected data.","You need a lawful basis. For commercial research the usual candidate is legitimate interests, which requires you to balance your interest against the individual's rights and — importantly — to document that you did.","You have to tell people. There is a transparency obligation when you collect data about someone from a source other than them. There is a carve-out where notifying everyone would be disproportionate, but it is not automatic and it depends on you having thought about it.","You must minimise. Collect what the purpose needs and no more. This is also, conveniently, the rule that makes most of the problem disappear.","And you have to be able to respond. Individuals can ask what you hold and ask you to delete it, which in practice means you need to be able to find a named person in your dataset at all.",{"type":79,"title":129,"paragraphs":130},"The simplest answer is usually the right one",[131,132,133,134],"Do not collect it.","A price monitoring system does not need seller names to work. It does not need review text. Dropping those fields at the point of extraction — not filtering them out later, but never writing them down — moves the entire project out of the personal data regime and removes a category of risk, a category of obligation and a category of conversation.","Where a seller identity genuinely matters for the analysis, a stable hash preserves the ability to say \"this is the same seller as last week\" without holding anybody's name. That is sufficient for almost every commercial question people actually ask of this data.","It is also simply less to defend. A dataset that provably contains no personal data is the shortest possible answer to a security review, and security reviews are where these projects most often stall.",{"type":84,"title":136,"intro":137,"headers":138,"rows":143},"Worked example: auditing a product feed, column by column","A marketplace price feed looks entirely commercial until you list the columns and ask one question of each: could this, alone or combined with the rest, identify a living person? Here is the same feed before and after that audit.",[139,140,141,142],"Column","Identifies a person?","Needed for repricing?","Decision",[144,149,152,154,158,161,164],[145,146,147,148],"product_title","No","Yes, for matching","Keep",[150,146,151,148],"price, currency, in_stock","Yes",[153,146,147,148],"gtin, mpn",[155,156,146,157],"seller_name","Often yes on a marketplace, where a large share of sellers trade under their own name","Drop",[159,160,146,157],"seller_address","Yes, frequently a home address",[162,163,146,157],"review_author, review_text","Yes, and review text is a direct opinion attached to a named person",[165,166,146,157],"q_and_a_username","Yes, pseudonymous but linkable",{"type":168,"title":169,"intro":170,"items":171},"list","What usually goes wrong","Nobody sets out to build a personal dataset. It assembles itself from columns that each looked harmless.",[172,173,174,175,176],"Scraping the whole page because it was easier than selecting fields, then discovering a year later that the warehouse holds seller names and review text nobody ever used.","Treating pseudonyms as anonymous. A stable username plus a purchase history plus a town is identifying in practice, whatever it looks like in isolation.","Keeping seller_name \"for debugging\" and never removing it, so a temporary convenience becomes a permanent category change.","Assuming public means unregulated. Publication by the person does not remove the obligations on whoever builds a new collection from it.","Having no deletion path, so the first time someone asks what you hold about them there is no honest answer and no mechanism to act on it.",{"type":168,"title":178,"intro":179,"items":180},"When you do need personal data, the minimum discipline","Some research genuinely requires it. If that is you, these are not optional.",[181,182,183,184,185,186],"Write the lawful basis down before collecting, with the balancing reasoning, not after","Keep the narrowest field set that answers the question","Set a retention period and actually enforce it — indefinite retention is very hard to justify","Be able to locate and delete an individual's records on request","Never collect special category data — health, politics, religion, sexuality, biometrics — without specific advice","Get a real review from someone qualified. This is the part of the course where the stakes justify the fee.",[188,189,190,191],"Public personal data is still personal data. \"It was on a public page\" is not a defence.","Product datasets acquire personal data by accident — seller names, review text, Q&A. Usually one column does it.","Minimisation is the lever: never write the field down and the whole regime stops applying.","Where seller identity matters, hash it. You keep the continuity and hold nobody's name.",{"text":193,"label":194,"url":195},"Next: conduct rather than law — rate limits, robots.txt, and being the kind of client nobody has to block.","Lesson 4: rate limits, robots.txt and good conduct","/learn/web-scraping-legal-and-ethical/rate-limits-robots-and-being-a-good-citizen",2,[198,204,209,210,215],{"slug":199,"navTitle":200,"title":201,"summary":202,"time":203,"needsAccount":72},"is-web-scraping-legal","Is it legal?","Three questions hiding inside one","\"Is scraping legal\" bundles access, copying and use into a single question. Separating them is most of the work.","10 min",{"slug":205,"navTitle":206,"title":207,"summary":208,"time":71,"needsAccount":72},"public-data-terms-of-service-and-logins","Terms and logins","Public data, terms of service, and what changes at the login","Why a terms page you never agreed to is weaker than people think, why the one you did agree to is stronger, and where that leaves mobile app APIs.",{"slug":67,"navTitle":68,"title":69,"summary":70,"time":71,"needsAccount":72},{"slug":211,"navTitle":212,"title":213,"summary":214,"time":71,"needsAccount":72},"rate-limits-robots-and-being-a-good-citizen","Conduct and rate limits","Rate limits, robots.txt, and being easy to live with","The conduct half. What robots.txt is for, what rate to actually use, and why identifying yourself is the most underrated decision available.",{"slug":216,"navTitle":217,"title":218,"summary":219,"time":220,"needsAccount":72},"what-to-put-in-front-of-your-legal-team","Briefing legal","What to put in front of your legal team","A one-page brief that gets a real answer, the three mistakes that guarantee a no, and the position we operate under ourselves.","12 min",{"slug":205,"navTitle":206,"title":207,"summary":208,"time":71,"needsAccount":72},{"slug":211,"navTitle":212,"title":213,"summary":214,"time":71,"needsAccount":72},1791047867451]