Cloudflare Turnstile and Web Scraping: What Actually Clears It (2026)

Cloudflare Turnstile and Web Scraping: What Actually Clears It (2026)

Key takeaways

  • Turnstile is not a CAPTCHA you solve. By the time the widget renders, Cloudflare has already scored your TLS handshake, your IP and your browser internals — the checkbox only reports the verdict.
  • That is why CAPTCHA solving services mostly do not help on Turnstile. They answer a challenge; Turnstile is grading the client that asked for one.
  • Camoufox on residential IPs passed 91% of Cloudflare standard and 78% of Enterprise in our April 2026 run — the highest of six approaches, and the slowest.
  • The measured cost of a cleared challenge is about 42.5 seconds, roughly 85 challenged pages per hour per worker. That number, not the pass rate, is what breaks price-monitoring schedules.
  • The real leverage is the cf_clearance cookie: clear once in a browser on a sticky session, then run the rest of the crawl over a TLS-impersonating HTTP client until it expires.

Turnstile is not a puzzle you solve. It is a verdict being displayed to you. By the time the widget paints, Cloudflare has already fingerprinted your TLS handshake, checked your IP's reputation and probed the browser for automation traces. The checkbox is the receipt.

That single framing explains most of the frustration around it: teams buy a CAPTCHA solver, wire it in, and nothing changes. The solver answers challenges. Turnstile is grading the client that asked for one.

Choose Before You Read

Your situation Do this
You bought a solver and it changed nothing Expected — here is why
You are getting challenged on every single page Stop re-clearing. Reuse the cookie
You are on datacentre proxies Fix that first. It caps every row in the table below
You need to know if it is even Turnstile Thirty-second check
Your crawl schedule is slipping, not failing The 42 seconds is your problem, not the pass rate
You want the rows, not a challenge pipeline Skip to the last section

Turnstile Is a Score, Not a Puzzle

Turnstile is Cloudflare's replacement for the interactive CAPTCHA. It runs a short battery of browser checks — proof-of-work, API surface probing, interaction timing — and hands the result to the same bot-scoring engine that sits in front of the rest of the Cloudflare edge.

That engine assigns every request a bot score from 1 to 99, where 1 means almost certainly automated and 99 means almost certainly a person. The widget does not compute that score from scratch. It contributes to a verdict that was already most of the way formed from:

  1. The TLS handshake — your JA3/JA4 fingerprint, read before any JavaScript runs
  2. IP reputation — ASN, abuse history, whether the range belongs to a cloud provider
  3. HTTP/2 frame ordering — settings, window sizes, header order, all of which differ between a real browser and a library
  4. Browser internals — the properties and quirks that a patched or instrumented runtime cannot fully hide

Three of those four are decided before the widget exists. This is why the managed challenge sometimes passes you through in under a second with no interaction at all, and sometimes loops forever on the same code: nothing about your script changed, your score did.

The practical consequence: you do not get past Turnstile by getting better at Turnstile. You get past it by presenting a client that scores well enough never to be seriously challenged.

The clearest expression of that idea is thermoptic, a proxy that describes its own job as cloaking requests "as the Chrome browser across all layers of the stack." It does not answer a challenge at any point; its Turnstile tutorial and the Hacker News thread about it are both worth reading for what they do not contain.

How to Tell Turnstile From a Managed Challenge

Worth thirty seconds, because the fixes differ and the symptoms look identical from a failing scraper's logs.

  • A cf-turnstile-response field in a form, or a challenges.cloudflare.com iframe — the Turnstile widget proper, usually guarding a login or a signup
  • An interstitial page with cf_clearance set on exit — the managed challenge, which is what most catalogue and price pages run
  • __cf_bm with no interstitial at all — bot management is on, but you are passing. Do not touch anything
  • A cf-ray header with a 403 — you are being refused outright, which is an IP-reputation problem, not a challenge problem

If instead you see _abck and sensor_data, that is Akamai Bot Manager and the correct fix is almost the opposite one. _px cookies mean PerimeterX. __ddg means DataDome.

Michael Mintz, who maintains SeleniumBase, demonstrating the widget being cleared by a driver that simply scores well enough to be allowed to click. Note what is absent from that clip: there is no puzzle being answered.

What Cleared It in Our April 2026 Run

Six approaches, 500 requests each, spread over six hours against production Cloudflare-protected sites. Full method in bypassing Cloudflare, Akamai and PerimeterX; the Cloudflare slice is below.

Approach Cloudflare Std Cloudflare Enterprise
Requests library + datacentre proxies 19% 4%
curl-cffi + datacentre proxies 61% 38%
Camoufox + datacentre proxies 72% 44%
curl-cffi + residential proxies 79% 52%
Patchright + residential proxies 83% 69%
Camoufox + residential proxies 91% 78%

Two things to read out of this.

The proxy tier moves the numbers more than the tool does. Camoufox loses nineteen points by being put behind datacentre IPs — more than the gap between Camoufox and a plain HTTP client on good IPs. If you are on shared datacentre proxies, every hour spent on stealth libraries is being spent on the wrong layer.

Firefox beats Chromium here, and only here. Camoufox patches a Firefox build at the C++ level, and Cloudflare's fingerprinting treats Firefox TLS signatures differently from the Chromium family that nearly every automation tool descends from. On Akamai the ranking inverts and a browser-free HTTP client wins outright. There is no tool that is simply best.

The 42 Seconds Is the Real Cost

Pass rate is the number everyone quotes and the wrong one to plan against.

A cleared Turnstile challenge in Camoufox takes an average of 42.49 seconds — that is the figure behind the Camoufox vs Patchright comparison, and it works out to roughly 85 challenged pages per hour per worker.

Run that against a real brief. Four competitors, 5,000 SKUs each, refreshed daily is 20,000 pages. If even a fifth of them get challenged, that is 4,000 challenges, which is 47 worker-hours. You do not have a bypass problem at that point. You have a fleet-sizing problem, and it arrived without anything failing.

This is the failure mode that does not page anyone: the scraper is not broken, it is late. Prices land in the afternoon instead of before the morning repricing run, and the only visible symptom is somebody noticing the dashboard looks stale.

The highest-leverage change available on Cloudflare, and the one most teams skip because it is not a library.

When a challenge is cleared, Cloudflare issues a cf_clearance cookie. It is bound to your IP and your user agent, and for its lifetime it carries the verdict you just paid 42 seconds for. Clearing a challenge per page throws that away every time.

The pattern that works:

  1. Clear one challenge in a real stealth browser, on a sticky residential session
  2. Export cf_clearance and __cf_bm
  3. Run the rest of the crawl through a TLS-impersonating HTTP client such as curl_cffi, on the same IP and the same user-agent string
  4. Fall back to the browser only when a request starts getting challenged again
from curl_cffi import requests

# cf_clearance is bound to the IP and the UA that earned it.
# Change either one and it is worthless.
session = requests.Session(impersonate="chrome124", proxies=STICKY_PROXY)
session.cookies.update({"cf_clearance": CLEARANCE, "__cf_bm": BM})
session.headers["User-Agent"] = UA_THAT_EARNED_THE_COOKIE

r = session.get(url)
if "challenges.cloudflare.com" in r.text or r.status_code == 403:
    refresh_clearance_in_browser()   # pay the 42 seconds, once

The two failures that make people conclude this does not work: rotating the proxy per request, which invalidates the cookie immediately, and sending a different user agent than the browser that earned it. Both are defaults in most scraping stacks. The mechanics of the impersonation layer are in curl_cffi and TLS fingerprint impersonation.

What CAPTCHA Solvers Can and Cannot Do Here

Solver services do have a place, and it is narrower than their marketing suggests.

They work when a human-answerable challenge is genuinely presented — reCAPTCHA v2 image grids, hCaptcha, an explicit interactive Turnstile widget on a form you are submitting once.

They do not work when your client is being scored down before the widget loads. You can buy a valid token and still be refused, because the token is checked against the session that requested it. A good token presented by a badly-scoring client is not a pass; it is evidence.

So the order of operations matters: fix the TLS fingerprint and the IP tier first, and only then consider whether anything is left for a solver to do. Usually there is not. If you are comparing services anyway, we tested four of them.

What This Costs to Keep Alive

Approach Monthly infra at 10k pages/day Engineering maintenance
curl-cffi + residential + cookie reuse ~$140–200 4–6 hrs/month
Patchright + residential proxies ~$380–520 8–12 hrs/month
Camoufox + residential proxies ~$440–600 10–15 hrs/month

Cloudflare ships changes continuously and does not announce them. A pass rate that held for three months can drop twenty points in a week, and the first signal is usually a quieter one than an error: fewer rows than yesterday.

If what you need is competitor prices and stock levels arriving on a schedule rather than a challenge pipeline to keep alive, the Turnstile question is the wrong one to be answering. ScrapeWise runs the browsers, the proxies and the challenge handling and delivers structured rows. No plan and no compute units: a balance you top up, and a published charge per delivered page — EUR 0.15 per 1,000 for plain requests, EUR 0.75 with rendering, EUR 1.50 through the residential tier and EUR 3.75 for render plus residential. New accounts get 5 free requests and the balance never expires.

You can also call ready-made scraping API endpoints directly, or see how this feeds competitor price tracking.

So What Should You Run?

  • Camoufox on residential IPs — the target is Cloudflare Enterprise and the pages are high-value. 78% at Enterprise, and plan for 85 challenged pages per hour per worker.
  • Patchright on residential IPs — you are on Node.js or unwilling to adopt Firefox. Three points behind on standard, nine behind on Enterprise, meaningfully faster.
  • Browser once, then curl_cffi on the cookie — the highest throughput per euro available on Cloudflare, and the one most teams have not tried.
  • A solver — only for a genuine interactive widget on a form, after the fingerprint and the IP tier are already right.
  • None of the above — you want delivered rows on a schedule and would rather not maintain a challenge pipeline.

See pricing.

Paste any URL — ScrapeWise handles the anti-bot

Managed infrastructure that adapts when sites change. No proxies, no code, no per-request fees.

Not ready to sign up? See 12 real Amazon rows, 972 columns →

97% accuracy on Amazon benchmarks · no credit card · book a 15-min call →

FAQ

Frequently asked questions

What Cloudflare Turnstile actually measures, the measured pass rates for stealth browsers against plain HTTP clients, and why cookie reuse beats buying a solver

Not by solving it, which is the framing that causes most of the wasted effort. Turnstile runs a short battery of browser checks and feeds the result into Cloudflare's bot score, which was already mostly determined by your TLS handshake, your IP reputation and your HTTP/2 frame ordering before the widget loaded. You get past it by presenting a client that scores well enough never to be seriously challenged. In our April 2026 run of 500 requests per approach, Camoufox on residential proxies passed 91% of Cloudflare standard tiers and 78% of Enterprise, the highest of six approaches tested.

Because solvers answer challenges and Turnstile is grading the client that asked for one. A solver service can return a valid token and you can still be refused, because the token is validated against the session that requested it. A good token presented by a badly scoring client is not a pass, it is evidence. Solvers earn their place on genuinely interactive challenges such as reCAPTCHA v2 image grids or a one-off Turnstile widget on a form you submit manually. For catalogue scraping, fix the TLS fingerprint and the proxy tier first and there is usually nothing left for a solver to do.

When a Cloudflare challenge is cleared, the edge issues a cf_clearance cookie that carries that verdict for its lifetime. It is bound to the IP address and the user-agent string that earned it. The highest-leverage pattern available on Cloudflare is to clear one challenge in a stealth browser on a sticky residential session, export cf_clearance and __cf_bm, then run the rest of the crawl through a TLS-impersonating HTTP client such as curl_cffi on the same IP and the same user agent. The two mistakes that make teams conclude this does not work are rotating the proxy per request, which invalidates the cookie immediately, and sending a different user agent than the browser that earned it.

About 42.49 seconds on average in Camoufox, which works out to roughly 85 challenged pages per hour per worker. That number matters more than the pass rate for anyone running a schedule. Four competitors at 5,000 SKUs each refreshed daily is 20,000 pages; if a fifth of them are challenged that is 4,000 challenges and 47 worker-hours a day. The resulting failure mode is quiet. The scraper is not broken, it is late, and prices land after the morning repricing run instead of before it.

Camoufox, by eight to nine points. On residential proxies we measured Camoufox at 91% on Cloudflare standard and 78% on Enterprise, against Patchright at 83% and 69%. The reason is that Camoufox patches a Firefox build at the C++ engine level and Cloudflare's fingerprinting treats Firefox TLS signatures differently from the Chromium family that almost every automation tool descends from. That advantage is vendor-specific: on Akamai Bot Manager the ranking inverts and a browser-free HTTP client with TLS impersonation beat both stealth browsers at 87%.

Look at what the response contains. A cf-turnstile-response form field or a challenges.cloudflare.com iframe is the Turnstile widget proper, usually guarding a login or a signup. An interstitial page that sets cf_clearance on exit is the managed challenge, which is what most catalogue and price pages run. A __cf_bm cookie with no interstitial means bot management is active and you are passing, so change nothing. A cf-ray header with a 403 is an outright refusal, which is an IP reputation problem rather than a challenge problem and will not be fixed by any stealth library.

They cap your ceiling on every tier. In our benchmark Camoufox lost nineteen points simply by being moved from residential to datacentre IPs, dropping from 91% to 72% on standard Cloudflare and from 78% to 44% on Enterprise. That nineteen-point penalty is larger than the gap between Camoufox and a plain HTTP client on good IPs. If you are on shared datacentre proxies, every hour spent evaluating stealth libraries is being spent on the wrong layer, and the proxy tier should be fixed before anything else is changed.

At around 10,000 pages a day, a curl-cffi stack on residential proxies with cookie reuse runs roughly 140 to 200 dollars a month in infrastructure plus four to six engineering hours; Patchright on residential proxies is roughly 380 to 520 dollars and eight to twelve hours; Camoufox is roughly 440 to 600 dollars and ten to fifteen hours. The maintenance column is the one teams underestimate. Cloudflare ships changes continuously and does not announce them, so a pass rate that held for three months can drop twenty points in a week, and the first signal is usually fewer rows rather than an error.