01 · Reality check
What Claude actually does well here
Good at
- Reading and analysing pages whose URLs you paste into the conversation
- Structuring messy scraped data you already have into clean tables
- Writing the scraper code for you to run in your own environment
- Extracting fields from documents and pages you upload
- Explaining a site's structure so you can target your own scraper
Not the right tool for
- Crawling: it cannot follow links to find pages you have not given it
- Constructing URLs, which Anthropic explicitly disallows
- JavaScript-rendered pages through the web fetch tool
- Bypassing robots.txt, logins or captchas, all of which it refuses
- Bulk collection of any kind, which is simply not a capability
02 · The method
Step by step
- 1
Separate collection from analysis
Scraping is two jobs: getting the data, and making sense of it. Claude cannot do the first and is very good at the second. Plan around that split rather than fighting it.
- 2
For a handful of pages, paste the URLs
The web fetch tool can retrieve URLs that appear in your conversation. So giving Claude ten specific links works; asking it to find and follow links does not.
- 3
For bulk collection, have Claude write the scraper
Ask it to write a script in Python using standard libraries, which you then run yourself. This is the practical route to real volume, and it plays to Claude's coding strength.
- 4
Bring the data back for analysis
Once you have the raw output, upload it. Claude handles files up to 500 MB and can run code to clean, structure and analyse it, producing spreadsheets and charts.
- 5
Consider Claude in Chrome for interactive pages
For pages behind a login you legitimately hold, Claude in Chrome drives a real browser and therefore sees rendered content. It is browser automation, not scraping, and it is still beta in the browser.
- 6
Check you are allowed to collect it
Terms of service, robots.txt and data protection law all apply to you even when a tool would technically permit the request. This is your responsibility, not the model's.
03 · Use this now
Copy-paste prompt to get a scraper written
I need to collect [describe the data] from [describe the site or page structure]. You cannot fetch or crawl this yourself, so write me a script I will run in my own environment instead. Use Python with standard, well-maintained libraries. Requirements: respect robots.txt, include a realistic delay between requests, handle pagination, handle failures and retries gracefully, and write the output to CSV with these fields: [list them]. Explain how to run it and what to change if the page structure differs from what you assumed. Then tell me what could break and how I would notice.
04 · Avoid these
Common mistakes
- Assuming a better prompt will unlock crawling. The restriction is by design, not a prompting problem.
- Expecting the web fetch tool to handle JavaScript-rendered pages, which it does not.
- Asking it to scrape sites behind logins or captchas, which it refuses.
- Ignoring terms of service and data protection law, which apply to you regardless of the tool.
- Confusing Claude in Chrome, which drives a real browser, with the web fetch tool, which does not render pages.
05 · Questions
Frequently asked questions
Can Claude scrape a website?
No. Anthropic's documentation states the web fetch tool can only fetch URLs that have previously appeared in the conversation, and that Claude is not allowed to dynamically construct URLs. Together those rules make link-following crawls impossible. It also does not render JavaScript pages and it enforces robots.txt.
Can Claude read a web page at all?
Yes, if you give it the URL. Web search and URL fetching are available on all plans including Free, though the feature is not on by default and is toggled from the chat input. Paste a link and Claude can retrieve and analyse that page. What it cannot do is discover pages you did not give it.
How do I scrape data if Claude cannot?
Use Claude for the parts it is good at: have it write a scraper you run yourself, then bring the collected data back for cleaning and analysis. It handles large files and can run code to structure results into spreadsheets and charts. Collection happens in your environment; analysis happens in Claude.
Can Claude scrape LinkedIn?
No, and you should not try. Beyond Claude's technical restrictions, scraping LinkedIn violates its terms of service and can involve personal data subject to privacy law. Claude in Chrome also explicitly refuses to scrape facial images and requires confirmation for sensitive actions.
Does Claude in Chrome change this?
Partly, and it is worth being precise. Claude in Chrome drives a real browser, so it sees pages as rendered, including ones you are logged into. That is browser automation on pages you visit rather than bulk scraping, it requires a paid plan, it is still beta in the browser, and it refuses captcha bypass and sensitive data entry.