Handling hCaptcha in a Python Scraping Pipeline in 2026

Most Python scrapers hit hCaptcha the same way. The pipeline runs fine for a few hundred pages, then a login, search or checkout step returns a page with a challenge instead of data, and every request after it fails the same way. This article walks through a pattern for that step: detect the challenge, get a token, submit it, and keep the pipeline running. The examples use requests for plain HTTP and Playwright for pages that need a browser.

1. Detect the challenge instead of parsing garbage

The first job is noticing that you got a challenge page. A scraper that doesn't check will parse the challenge HTML as if it were content and fill your dataset with empty rows.

An hCaptcha widget leaves clear traces in the HTML: the script from js.hcaptcha.com, a container with the class h-captcha, and a data-sitekey attribute. A small helper catches all three:

import re

SITEKEY_RE = re.compile(r'data-sitekey="([0-9a-f-]{36})"')

def find_hcaptcha(html: str) -> str | None:
    """Return the sitekey if the page carries an hCaptcha widget, else None."""
    if "hcaptcha.com" not in html and "h-captcha" not in html:
        return None
    m = SITEKEY_RE.search(html)
    return m.group(1) if m else None

The sitekey is a UUID that identifies the site's hCaptcha configuration. Together with the page URL, it is all a solver needs for a normal widget. Some sites, usually on hCaptcha Enterprise, also pass an rqdata value to the widget in their own JavaScript. If your target does, extract it as well and send it with the request.

2. Get a token

The widget's real output is a token. When a person passes the challenge, the widget writes it into a hidden form field named h-captcha-response, and the site's server checks it with hCaptcha. Your scraper never has to click anything; it only needs a valid token in that field.

A captcha solver service returns that token from the sitekey and URL. The examples here use NoneCap's Python client (pip install nonecap), which waits for the result for you:

import os
from nonecap import NoneCap

nc = NoneCap(api_key=os.environ["NONECAP_KEY"])

def get_token(sitekey: str, url: str):
    solve = nc.solve(type="hcaptcha", sitekey=sitekey, url=url)
    return solve  # solve.token, solve.user_agent, solve.id

For an Enterprise sitekey, use type="hcaptcha_enterprise" and pass rqdata= when the page provides one.

3. Submit it the way the page would

With plain HTTP, post the form with the token in h-captcha-response. Two details matter more than anything else:

• Send the returned User-Agent. The solve comes back with the user_agent it was made for. Some sites refuse a token that arrives from a different browser version, so use that value for the submit request.
• Use the token right away. A token is single-use and meant to be submitted shortly after it is made. Request it just before the step that needs it, not at the start of a long run.

import requests

session = requests.Session()

def submit_login(url: str, form: dict) -> requests.Response:
    page = session.get(url)
    sitekey = find_hcaptcha(page.text)
    if sitekey is None:
        return session.post(url, data=form)

    solve = get_token(sitekey, url)
    return session.post(
        url,
        data={**form, "h-captcha-response": solve.token},
        headers={"User-Agent": solve.user_agent},
    )

4. When the page needs a browser

Some targets only render the form with JavaScript. In Playwright, read the sitekey from the live DOM, then write the token into the hidden field before you submit:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://target.example/login")

    sitekey = page.get_attribute("[data-sitekey]", "data-sitekey")
    solve = get_token(sitekey, page.url)

    page.evaluate(
        "t => document.querySelectorAll('[name=h-captcha-response]').forEach(el => el.value = t)",
        solve.token,
    )
    page.click("button[type=submit]")

If the site reads the token through a JavaScript callback rather than the form field, find the callback name in the widget's data-callback attribute and call it with the token.

5. Make failures boring

At volume, some solves will fail and some tokens will be refused by the target. Treat both as normal events with a clear next step, not as crashes:

from nonecap import SolveFailedError, RateLimitError
import time

def get_token_with_retry(sitekey: str, url: str, attempts: int = 3):
    for _ in range(attempts):
        try:
            return get_token(sitekey, url)
        except SolveFailedError as err:
            if not err.retryable:
                raise
        except RateLimitError as err:
            time.sleep(err.retry_after or 1)
    raise RuntimeError("no token after retries")

Then tell the solver what the target did with the token. Only your code sees whether the site accepted it, and reporting it lets the service react to problems on your sitekey:

solve = get_token(sitekey, url)
resp = session.post(url, data={**form, "h-captcha-response": solve.token},
                    headers={"User-Agent": solve.user_agent})
nc.feedback.report(solve.id, outcome="accepted" if resp.ok else "rejected")

6. Scale out with async

A pipeline that waits for one token at a time leaves most of its time idle. The async client runs many solves at once; cap the concurrency with a semaphore so you stay inside your account's limits:

import asyncio
from nonecap import AsyncNoneCap

async def solve_many(jobs: list[tuple[str, str]], limit: int = 10):
    sem = asyncio.Semaphore(limit)
    async with AsyncNoneCap(api_key=os.environ["NONECAP_KEY"]) as nc:
        async def one(sitekey, url):
            async with sem:
                return await nc.solve(type="hcaptcha", sitekey=sitekey, url=url)
        return await asyncio.gather(*(one(s, u) for s, u in jobs), return_exceptions=True)

Wrapping up

The pattern is the same whatever you scrape: check every response for a challenge, get a token only when you need one, submit it with the matching User-Agent, and treat failures as data. hCaptcha support across solver services has changed a lot in the past year, so test any service on your own sitekey before you build on it.

Daniel Carlsson is the founder of NoneCap, a captcha solver service that focuses on hCaptcha.