No-code web scraping tools
These suit marketers, researchers and analysts who want data without writing a script.
Maxun lets you train a “robot” by clicking the parts of a page you want, then runs it on a schedule and returns the results as a table or an API. It is the closest thing here to a commercial point-and-click scraper. It runs on your own server with Docker and uses the AGPL-3.0 licence.
Automa is a browser extension in which you connect blocks to record clicks, typing and data extraction, then replay them. It works well for small, repeated jobs in your own browser, such as copying a table from a page each morning. Its licence is listed as “Other”.
changedetection.io watches web pages and sends an alert by email, Slack or other channels when the content, a price or the stock status changes. Pick it when you need to know that something changed rather than collect large amounts of data.
Python, Node.js and Go frameworks for larger crawls
A crawler follows links from page to page, and a scraper pulls the data out of each page. These frameworks do both and handle queues, retries and storage for you, so they suit developers collecting thousands of pages.
Scrapy is the long-standing Python framework for fast, large-scale crawling. It is a strong candidate for the best open source web scraper for big jobs, and it has a wide range of plugins. On its own it does not run JavaScript, so pair it with a browser tool for pages that build their content in the browser.
Crawlee is a Node.js library for building reliable crawlers, with proxy rotation and a choice of plain HTTP requests or a full browser. Crawlee for Python brings the same approach to Python. Choose Crawlee when you want one tool that moves easily between fast HTTP crawling and browser crawling.
Colly is a Go library for writing fast crawlers and scrapers, with rate limits, caching and parallel requests built in. It uses the Apache-2.0 licence. Pick it if your team writes Go and wants web scraping tools in the same language as the rest of its code.
Scrapling is a Python library whose selectors try to keep working after a site changes its layout, which cuts down on broken scrapers. It suits long-running projects that watch the same sites for months.
Trafilatura pulls the main text, metadata and comments out of web pages and leaves menus and adverts behind. It is a strong choice for researchers building text collections, and you can use it from Python or the command line.
Browser automation for pages that need JavaScript
Many modern sites only show their content after scripts run in the browser. These web scraping tools control a real browser, so they see the page the way a visitor does.
Playwright controls Chromium, Firefox and WebKit from one API and works with Python, Node.js, Java and .NET. It was built for testing, and it works just as well for scraping dynamic pages. Start here if you need a browser and have no strong preference.
Selenium is the long-running browser automation framework, with drivers for every major browser. Like Playwright, it was built for testing and is also used to scrape dynamic pages. It uses the Apache-2.0 licence. Choose it when your team already has Selenium test code or needs a browser that Playwright does not drive.
Puppeteer is a Node.js library for controlling Chrome or Firefox, often used for scraping, screenshots and PDFs. It suits JavaScript developers who mainly target Chrome.
Lightpanda is a new headless browser (one that runs without a visible window) written in the Zig language, designed to use less memory than headless Chrome. It speaks the Chrome DevTools Protocol, the interface that tools use to control Chrome, so Puppeteer and Playwright scripts can connect to it. Try it when running many browsers at once gets expensive. It uses the AGPL-3.0 licence.
Stagehand mixes ordinary Playwright code with plain-language steps such as “click the sign-up button” or “extract the price”. It suits sites where writing exact selectors is slow or fragile.
Nanobrowser is a Chrome extension that runs AI agents inside your own logged-in browser, using your own model API key. Use it for one-off research tasks rather than repeatable data collection.
AI web scraper tools that prepare pages for language models
These tools turn messy web pages into clean text or structured data that a large language model (LLM) can use, which is what most AI projects need.
Crawl4AI is an open source web scraper and crawler that outputs clean Markdown and supports structured extraction. It runs entirely on your own machine, which makes it a good free web scraper option for feeding documents to an AI app.
Firecrawl turns websites into Markdown or structured data and can crawl whole sites or search the web. You can use the hosted service or run the open-source version yourself. It uses the AGPL-3.0 licence.
ScrapeGraphAI is a Python AI web scraper that you steer with a plain-language prompt, such as “list every product name and price”. It saves time on one-off jobs, but every page costs a model call, so it is slower and pricier than a normal scraper at scale.
Jina Reader converts a single web address into text that a language model can read easily. It is the quickest option when you only need a handful of pages.
Ready-made scrapers for one job
Google Maps Scraper is a command-line tool with a web interface that collects business listings, contact details and reviews from Google Maps into a spreadsheet file or a database. It suits people building local lead lists. Google’s terms limit automated collection from Maps, so read them before you use it.
Legal and anti-bot limits to know before you scrape
Scraping public pages is common, but these web scraping tools do not make every use lawful. Check the site’s terms of use and its robots.txt file, which tells automated visitors which pages they may visit. Be careful with personal data, because privacy laws such as the EU’s General Data Protection Regulation (GDPR) apply even when the data is public. Keep your request rate low so you do not slow the site down, and prefer an official API whenever one exists. This is general guidance, not legal advice.
Many sites also use anti-bot systems. If a site blocks you or shows a CAPTCHA, treat that as a clear signal to stop or to ask the owner for access.
How to choose between these web scraping tools
The GitHub page of each tool above shows recent activity and setup steps, so check both before you commit. These pointers narrow the choice quickly.
- You do not code: start with Maxun, or Automa for small jobs in your own browser.
- You want alerts when a page changes: use changedetection.io.
- You write Python and need scale: use Scrapy, and add Playwright for pages that need JavaScript.
- You write JavaScript: use Crawlee, which can drive Playwright or Puppeteer for you.
- You write Go: use Colly.
- You are feeding an AI app: use Crawl4AI or Firecrawl for whole sites, and Jina Reader for single pages.
If you want a web crawler that is open source and free to run at any size, Scrapy or Crawlee will cover most needs. All of these web scraping tools are free to download, but servers and proxies still cost money at scale, so start small and measure before you grow.