[{"data":1,"prerenderedAt":222},["ShallowReactive",2],{"learn-lesson-product-data-api-authentication-and-your-first-call":3},{"course":4,"lesson":67,"index":189,"outline":190,"prev":220,"next":221},{"slug":5,"order":6,"level":7,"time":8,"card_text":9,"seo":10,"hero":16,"outcomes":26,"who":35,"syllabus":46,"faq":49,"lessonCount":66},"product-data-api",3,"Comfortable with HTTP and JSON","6 lessons, about 70 minutes","Every retailer gets asked for an API and most of them never ship one, so you end up calling somebody else's. What a product data API actually returns, how to declare the fields you want, why long runs are asynchronous, and how to retry without paying twice.",{"title":11,"description":12,"keywords":13,"og_title":14,"og_description":15},"Product Data API: A Free 6-Lesson Course for Developers","How to pull product, price and stock data over an API. Authentication, declaring fields, asynchronous runs, pagination, retries without double billing, and putting the feed into your stack. Free, ungated.","product data api, price api, ecommerce api, web scraping api, api key authentication, api documentation, scraper api tutorial, rest api product data","A free developer course on pulling product data over an API","Six written lessons: when an API beats a scraper, auth, declaring fields, async runs, error handling, and shipping the feed into your stack.",{"badge":17,"title":18,"subtitle":19,"cta_primary":20,"cta_secondary":23},"Course three","Pull product data over an API","Search volume for \"\u003Cretailer> API documentation\" is enormous and the documentation mostly does not exist. Amazon, Walmart, Target, Home Depot — developers keep looking for a product endpoint that was never published, or that was published and then locked behind a partner agreement. So you end up calling a web data API instead: something that takes a URL and gives you back the fields. This course is about doing that properly, from the first authenticated request to a feed your warehouse can depend on.",{"label":21,"url":22},"Start with lesson one","/learn/product-data-api/when-an-api-beats-a-scraper",{"label":24,"url":25},"See what a run returns","/custom-scrapers",{"title":27,"items":28},"What you will be able to do",[29,30,31,32,33,34],"Decide between an official retailer API, a web data API and writing your own scraper, with reasons you can defend in review","Make an authenticated request and read the response without guessing what the fields mean","Declare a schema so the extractor returns the fields you need rather than the ones it felt like returning","Work with asynchronous runs: start, poll, and handle a job that finishes half-done","Retry a failed request without being billed twice for the same page","Land the result in a warehouse table that does not quietly drift out of date",{"title":36,"for_title":37,"for":38,"not_title":42,"not_for":43},"Who this is for","Written for",[39,40,41],"Developers who have been told to \"get the competitor prices\" and have discovered the retailer has no public API","Data engineers deciding whether to own the collection layer or buy it","Backend teams wiring a third-party feed into an existing pipeline and wanting to know where it will break","Not written for",[44,45],"Readers looking for a no-code setup — course one covers the same ground without a terminal","Anyone wanting a specific vendor's endpoint reference. This is the shape of the problem; the reference lives in that vendor's docs.",{"title":47,"intro":48},"The six lessons","Lessons one and two are short and assume nothing but curl. From three onwards you will get more out of it with a key in your hand.",{"badge":50,"title":51,"description":52,"items":53},"FAQ","Before you start","The questions developers ask in the first ten minutes.",[54,57,60,63],{"title":55,"description":56},"Does this teach a specific vendor's API?","No, deliberately. Endpoint names change and a course that hardcodes them rots. What does not change is the shape: authenticate, declare what you want, start a run, poll it, handle partial results, retry safely. Learn the shape and any vendor's reference becomes a lookup rather than a tutorial.",{"title":58,"description":59},"Why is it not just a GET that returns the price?","Because fetching a page takes seconds and fetching ten thousand takes hours. Any API that hides that behind a synchronous request is either very slow or quietly returning you a cached number. Lesson four is about why the asynchronous shape exists and how to work with it instead of around it.",{"title":61,"description":62},"Do I need a Scrapewise account?","Lessons three and six include a Scrapewise-specific walkthrough and are labelled on the page. The other four are method and apply to whatever you are calling. A new account starts with five free requests and no card if you want to follow along.",{"title":64,"description":65},"Can I just scrape it myself in Python?","Often yes, and lesson one is honest about when that is the right call. The threshold is not technical skill — it is how many distinct sites you need and how much you mind being the person who gets paged when one of them redesigns. One site, one developer, no deadline: write it yourself.",6,{"slug":68,"nav_title":69,"title":70,"summary":71,"time":72,"needs_account":73,"seo":74,"blocks":78,"takeaways":180,"next_step":185},"authentication-and-your-first-call","Auth and the first call","Authentication, keys, and your first real request","Bearer tokens versus query-string keys, where to keep the secret, and how to read the first response you get back.","10 min",false,{"title":75,"description":76,"keywords":77},"API Key Authentication for Data APIs: Bearer Tokens Explained","How API key authentication works for product data APIs, bearer tokens versus query-string keys, key scoping and rotation, and reading your first response.","api key, bearer token, api authentication, api access, rest api key, api documentation, authorization header",[79,85,92,101,106,112,145,165,175],{"type":80,"paragraphs":81},"prose",[82,83,84],"Almost every data API authenticates one of two ways, and the difference matters more than it looks.","The common one is a bearer token in a header. You send Authorization: Bearer followed by the key, the server checks it, and the key never appears in the URL. The other is a key in the query string, which exists because it is trivially easy to test in a browser address bar and is therefore popular in quickstarts.","Prefer the header every time it is offered. A key in a query string ends up in server access logs, in browser history, in the Referer header of any outbound link, in your error-tracking tool's breadcrumb trail, and in the screenshot somebody pastes into a ticket. A key in a header ends up in none of those by default.",{"type":86,"title":87,"intro":88,"language":89,"code":90,"caption":91},"code","The shape of a first request","Nothing vendor-specific here. Replace the base URL and the path with whatever your provider's reference gives you.","bash","curl -s \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  \"$API_BASE/scraper/list\"","Read the key from the environment, never from the command line directly — your shell history is a plaintext file and it is backed up.",{"type":93,"title":94,"intro":95,"items":96},"list","Where the key should live","In rough order of how much trouble each one saves you later.",[97,98,99,100],"In a secret manager your deployment reads at boot, if you have one","In an environment variable set by your orchestrator, if you do not","In a gitignored .env file for local development only, with a .env.example that documents the names but holds no values","Never in the repository, including in a test fixture, including in a commented-out line, including in a notebook you were going to delete",{"type":102,"variant":103,"title":104,"text":105},"callout","warning","Scope and rotate before you need to","If the provider lets you mint more than one key, mint one per environment and one per integration. The entire value of that is on the day you have to revoke one: a single shared key means revoking it takes down everything at once, and you will discover which systems used it by finding out what broke. Rotating is also the only cheap way to recover from a key that has leaked into a log you did not know existed.",{"type":80,"title":107,"paragraphs":108},"Reading the first response properly",[109,110,111],"Two habits here pay for themselves within a week.","First, before you write any parsing code, print the whole response once and read it. Not the field you came for — all of it. Data APIs routinely return metadata alongside the payload: a run identifier, a count, a timestamp, a flag saying the result was truncated. The truncation flag in particular is the kind of thing you find out about by missing it.","Second, check the status code and the body separately. A great many APIs return 200 with an error object inside, because the HTTP request succeeded even though the thing you asked for did not. If your client only branches on the status code, those failures become empty rows rather than exceptions.",{"type":113,"title":114,"intro":115,"headers":116,"rows":120},"table","Status codes worth branching on","Different responses, genuinely different handling. Treating them all as \"error\" is why retries get expensive.",[117,118,119],"Code","What it usually means","What to do",[121,125,129,133,137,141],[122,123,124],"200 with an error body","The request was valid, the operation was not","Read the body. Do not retry — it will fail identically.",[126,127,128],"400","Your payload is malformed or a field is unknown","Fix the request. Retrying is pointless.",[130,131,132],"401 / 403","Key missing, wrong, revoked, or out of scope","Stop and alert. A retry loop on a dead key is just noise.",[134,135,136],"404","Wrong path, or an object that no longer exists","Check the reference before assuming the route is gone.",[138,139,140],"429","You are over a rate limit","Back off, honour Retry-After if present, and reduce concurrency.",[142,143,144],"5xx","Their problem, possibly transient","Retry with exponential backoff and a cap. See lesson five.",{"type":113,"title":146,"intro":147,"headers":148,"rows":152},"Worked example: three responses that are all 200","Branching on the status code alone is the specific mistake this lesson exists to prevent. All three of these came back 200 OK. One of them is a success.",[149,150,151],"Body","What it actually means","What a status-only check does with it",[153,157,161],[154,155,156],"{ \"run_id\": \"r_8812\", \"rows\": [ … 412 rows … ] }","A success","The right thing, by luck",[158,159,160],"{ \"run_id\": \"r_8813\", \"rows\": [] }","The run completed and found nothing — a site change, a dead URL list, or a genuinely empty result","Writes an empty table over yesterday's good one",[162,163,164],"{ \"error\": \"quota_exceeded\", \"retry_after\": 3600 }","You are out of credit","Records a success, stores no rows, and tells nobody",{"type":93,"title":166,"intro":167,"items":168},"What usually goes wrong","Authentication and the first call are where habits get set that are painful to change a year later.",[169,170,171,172,173,174],"Branching on the status and not the body. Two of the three rows above are failures wearing a 200.","Putting the key in the query string. It lands in server logs, in browser history, in referrer headers and in every screenshot of the address bar — none of which rotate when you do.","One key for everything. When it leaks, and eventually one does, revoking it takes down production, the staging job and somebody's notebook at the same moment.","Not establishing the billable unit on day one. Requests, pages fetched and rows returned are three different numbers, and a quote will use whichever is smallest.","Testing with a key that has more scope than production will have. The call works in development and 403s on deploy, which is the worst possible time to find out.","Retrying a 401. A second attempt changes nothing about a wrong credential, and some providers count the attempts against you.",{"type":80,"title":176,"paragraphs":177},"One thing to verify on day one",[178,179],"Find out, explicitly, what counts as a billable unit. Is it a request you make, a page the provider fetches, or a row you receive? These are not the same number, and the gap between them is where surprise invoices come from.","A run that fetches two thousand pages and returns twelve hundred rows has been billed for two thousand on most pricing models, because the fetch is the cost. If you are budgeting on rows you will be wrong by whatever your failure rate is — and your failure rate is a property of the sites you chose, not of the vendor.",[181,182,183,184],"Use the Authorization header, not a query-string key. Query strings leak into logs, history and referrers.","One key per environment and per integration, so that revoking one does not take everything down.","A 200 can contain an error. Branch on the body as well as the status.","Establish what a billable unit is on day one: requests, fetched pages and returned rows are three different numbers.",{"text":186,"label":187,"url":188},"Next: the step most people skip, and the reason their results come back missing half the fields.","Lesson 3: declaring the fields you want","/learn/product-data-api/declare-the-fields-you-want",1,[191,196,197,204,210,215],{"slug":192,"navTitle":193,"title":194,"summary":195,"time":72,"needsAccount":73},"when-an-api-beats-a-scraper","API, scraper or dataset","When an API beats writing your own scraper","Three ways to get product data, the honest cost of each, and the specific question that decides between them.",{"slug":68,"navTitle":69,"title":70,"summary":71,"time":72,"needsAccount":73},{"slug":198,"navTitle":199,"title":200,"summary":201,"time":202,"needsAccount":203},"declare-the-fields-you-want","Declaring the fields","Declaring a schema, and why your fields came back empty","An extractor returns what you asked for, and most people ask badly. How to declare fields, why types matter, and the one mistake that silently drops a column.","12 min",true,{"slug":205,"navTitle":206,"title":207,"summary":208,"time":209,"needsAccount":73},"asynchronous-runs-and-polling","Async runs and polling","Asynchronous runs, polling, and partial results","Why collection APIs hand back a job rather than data, how to poll without hammering, and what to do with a run that finished eighty per cent done.","11 min",{"slug":211,"navTitle":212,"title":213,"summary":214,"time":209,"needsAccount":73},"errors-retries-and-double-billing","Errors and retries","Errors, retries, and not paying twice","Which failures are worth retrying, how idempotency keys stop a retry becoming a second invoice, and the error class that means stop rather than try harder.",{"slug":216,"navTitle":217,"title":218,"summary":219,"time":202,"needsAccount":203},"put-the-feed-into-your-stack","Into your stack","Putting the feed into your stack without it drifting","Scheduling, loading, and the schema decisions that determine whether a price feed is still trustworthy in six months.",{"slug":192,"navTitle":193,"title":194,"summary":195,"time":72,"needsAccount":73},{"slug":198,"navTitle":199,"title":200,"summary":201,"time":202,"needsAccount":203},1791047867140]