All Repos HereSaved (0)

Open-source repos for web scraping

60 repos tagged Web scraping, sorted by GitHub stars. Add filters from the other category types to narrow the list.

60repos tagged Web scraping
Instead of Octoparse or Bright DataKit: Web scraping, 20 compared
60 repos
Sort
Data from GitHub, checked

  1. yt-dlp/yt-dlp

    Feature-rich CLI to download video and audio from YouTube and thousands of other sites.

    Repurposing, YouTube, Marketing, Creators

    205k starsUnlicenseUpdated 30 Oct 2026

  2. firecrawl/firecrawl

    API that turns websites into clean Markdown or structured data for LLMs, with crawl and search.

    RAG and search, Web, Marketing, Engineering

    198k starsAGPL-3.0Updated 3 Nov 2026

  3. browser-use/browser-use

    Python library that lets AI agents control a web browser to fill forms, click and extract data.

    AI agents, Workflow automation, Web, Engineering, Operations

    120k starsMITUpdated 2 Nov 2026

  4. microsoft/playwright

    Library for automating Chromium, Firefox and WebKit, used for testing and dynamic-page scraping.

    CI/CD, Web, Engineering

    99.1k starsApache-2.0Updated 3 Nov 2026

  5. puppeteer/puppeteer

    Node.js API for controlling headless Chrome or Firefox for scraping, screenshots and PDFs.

    Web, Engineering

    95.8k starsApache-2.0Updated 2 Nov 2026

  6. D4Vinci/Scrapling

    Adaptive Python scraping library whose selectors keep working after site layout changes.

    Web, Engineering, Data and analytics

    94.4k starsBSD-3-ClauseUpdated 3 Nov 2026

  7. unclecode/crawl4ai

    Open-source crawler that outputs LLM-ready Markdown and supports structured extraction.

    RAG and search, Web, Engineering, Data and analytics

    86.8k starsApache-2.0Updated 3 Nov 2026

  8. scrapy/scrapy

    Long-standing Python framework for writing fast, large-scale web crawlers and data extractors.

    Web, Engineering, Data and analytics

    65.2k starsBSD-3-ClauseUpdated 3 Nov 2026

  9. huginn/huginn

    Runs agents that watch websites, feeds and APIs and act on changes, like a self-hosted IFTTT.

    Workflow automation, Social listening, Docker, Web, Operations, Marketing, Self hosters

    50.3k starsMITUpdated 2 Nov 2026

  10. DIYgod/RSSHub

    Generates RSS feeds for thousands of sites and social accounts that do not publish one themselves.

    Social listening, Docker, Web, Marketing, Media and publishing, Self hosters, Newsletters

    46.8k starsAGPL-3.0Updated 3 Nov 2026

  11. lightpanda-io/browser

    Headless browser built for AI agents and scraping, lighter and faster than headless Chrome.

    AI agents, Web, Engineering

    41k starsAGPL-3.0Updated 3 Nov 2026

  12. microsoft/playwright-mcp

    MCP server that lets AI assistants drive a real browser through Playwright accessibility snapshots.

    AI agents, Web, Engineering, Operations

    39.1k starsApache-2.0Updated 2 Nov 2026

  13. dgtlmoon/changedetection.io

    Monitors web pages for content, price or restock changes and sends alerts to email, Slack and more.

    Social listening, Monitoring, Docker, Email, Marketing, Operations, E-commerce, Retail, Self hosters

    35.5k starsApache-2.0Updated 2 Nov 2026

  14. SeleniumHQ/selenium

    Browser automation framework with drivers for every major browser, used for testing and scraping dynamic pages.

    Web, Engineering, Data and analytics

    34.6k starsApache-2.0Updated 4 Nov 2026

  15. ScrapeGraphAI/Scrapegraph-ai

    Python scraper that uses LLMs to extract data from pages based on a plain-language prompt.

    Web, Data and analytics, Marketing

    32.5k starsMITUpdated 6 Oct 2026

  16. ArchiveBox/ArchiveBox

    Saves web pages you care about as HTML, PDF, screenshots and media so you keep a private copy if they vanish.

    Backups, Docker, Web, Data and analytics, Media and publishing, Self hosters, Data journalism

    28.9k starsMITUpdated 2 Nov 2026

  17. apify/crawlee

    Node.js library for building reliable crawlers and scrapers with proxy rotation and browser support.

    Web, Engineering, Data and analytics

    26.3k starsApache-2.0Updated 3 Nov 2026

  18. browserbase/stagehand

    Browser automation SDK mixing Playwright code with natural-language actions and extraction.

    AI agents, Web, Engineering

    25.9k starsMITUpdated 3 Nov 2026

  19. gocolly/colly

    Go library for writing fast crawlers and scrapers with rate limits, caching and parallel requests.

    Web, Engineering, Data and analytics

    25.6k starsApache-2.0Updated 16 Oct 2026

  20. Skyvern-AI/skyvern

    Uses LLMs and computer vision to automate browser tasks like form filling and portal downloads.

    AI agents, Workflow automation, Web, Operations, Finance and fintech, Logistics

    23.3k starsAGPL-3.0Updated 3 Nov 2026

  21. AutomaApp/automa

    Browser extension for recording and running click-and-type automations and scrapers without code.

    Workflow automation, Lead generation, Web, Operations, Marketing, Agencies, No-code

    21.7k starsOtherUpdated 2 Mar 2026

  22. Evil0ctal/Douyin_TikTok_Download_API

    Self-hostable API to fetch watermark-free TikTok and Douyin video and data.

    TikTok, Engineering, Creators, Short-form video

    21.1k starsApache-2.0Updated 2 Nov 2026

  23. laramies/theHarvester

    Collects emails, names and subdomains for a company domain from public search sources.

    Lead generation, Enrichment, Google Search, Sales, IT and security, B2B services

    18.2k starsGPL-2.0Updated 30 Oct 2026

  24. getmaxun/maxun

    No-code platform for training robots that scrape websites into tables or APIs on a schedule.

    Lead generation, Docker, Web, Marketing, Data and analytics, No-code

    18.1k starsAGPL-3.0Updated 28 Oct 2026

  25. projectdiscovery/katana

    Fast command-line crawler that maps URLs and endpoints of a site, often used in security recon.

    Security scanning, Web, IT and security, Engineering

    17.8k starsMITUpdated 3 Nov 2026

  26. JoeanAmier/TikTokDownloader

    Downloads TikTok and Douyin videos, accounts and comment data in bulk.

    Social listening, TikTok, Marketing, Data and analytics, Creators, Short-form video

    17.3k starsGPL-3.0Updated 29 Oct 2026

  27. apify/apify-mcp-server

    MCP server that lets assistants run thousands of Apify scrapers for maps, social and e-commerce data.

    Lead generation, AI agents, Web, Marketing, Sales

    14.5k starsMITUpdated 3 Nov 2026

  28. nanobrowser/nanobrowser

    Chrome extension that runs multi-agent web automation with your own LLM API key.

    AI agents, Web, Operations, Marketing

    14.2k starsApache-2.0Updated 24 Oct 2026

  29. browserless/browserless

    Runs headless Chrome in Docker behind an API so scrapers and automation scripts can share remote browsers.

    Docker, Web, Engineering, SaaS

    13.9k starsOtherUpdated 3 Nov 2026

  30. instaloader/instaloader

    Downloads Instagram posts, captions, comments and profile metadata for analysis or archiving.

    Social listening, Instagram, Marketing, Data and analytics, Creators, Influencer

    13.8k starsMITUpdated 2 Nov 2026

  31. crawlab-team/crawlab

    Web admin for scheduling, running and monitoring scraper jobs written in any language across several nodes.

    Monitoring, Docker, Web, Engineering, Data and analytics

    12.3k starsBSD-3-ClauseUpdated 10 Feb 2026

  32. jina-ai/reader

    Converts any URL into LLM-friendly text by prefixing it with a reader endpoint.

    RAG and search, Web, Engineering

    12.2k starsApache-2.0Updated 22 May 2026

  33. streamlink/streamlink

    CLI that pipes streams from Twitch, YouTube and others into a video player or file.

    YouTube, Marketing, Creators

    11.9k starsBSD-2-ClauseUpdated 3 Nov 2026

  34. mozilla/readability

    Firefox's reader-mode library that strips a page down to the main article text and title.

    Web, Engineering, Media and publishing

    11.5k starsApache-2.0Updated 4 Aug 2026

  35. apify/crawlee-python

    Python version of Crawlee for crawling sites with HTTP or headless browsers and storing results.

    Web, Engineering, Data and analytics

    9.6k starsApache-2.0Updated 2 Nov 2026

  36. alirezamika/autoscraper

    Python scraper that learns extraction rules from a sample of the values you want, with no selectors to write.

    Web, Engineering, Data and analytics

    8.1k starsMITUpdated 29 Jul 2026

  37. steel-dev/steel-browser

    Self-hostable browser API that gives AI agents and scripts managed sessions, proxies and page actions.

    AI agents, Docker, Web, Engineering, SaaS

    7.9k starsApache-2.0Updated 3 Nov 2026

  38. firecrawl/firecrawl-mcp-server

    MCP server that lets assistants scrape, crawl and search the web through Firecrawl.

    AI agents, Web, Marketing, Data and analytics

    7.8k starsMITUpdated 3 Nov 2026

  39. gosom/google-maps-scraper

    Go CLI and web UI that extracts Google Maps places, emails and reviews into CSV or a database.

    Lead generation, Google Search, Docker, Sales, Marketing, Local business, Agencies, Local SEO

    7.2k starsMITUpdated 21 Oct 2026

  40. go-rod/rod

    Go driver for the Chrome DevTools Protocol for scripting browsers, testing and scraping JavaScript-heavy sites.

    Web, Engineering, Data and analytics

    7.1k starsMITUpdated 11 Aug 2026

  41. adbar/trafilatura

    Extracts main text, metadata and comments from web pages for research and corpus building.

    Web, Data and analytics, Media and publishing, Data journalism

    7.1k starsApache-2.0Updated 30 Oct 2026

  42. mishushakov/llm-scraper

    TypeScript library that uses an LLM and a schema to pull structured data out of any web page.

    LLM apps, OpenAI, Web, Engineering, Data and analytics

    7k starsMITUpdated 26 Oct 2026

  43. joeyism/linkedin_scraper

    Python library that scrapes LinkedIn person and company profiles with Selenium.

    Lead generation, Enrichment, LinkedIn, Sales, B2B services, Recruiting

    4.6k starsGPL-3.0Updated 10 Apr 2026

  44. praw-dev/praw

    Python wrapper for the Reddit API, used to monitor subreddits, keywords and comments.

    Social listening, Reddit, Marketing, Data and analytics, Community building

    4.3k starsBSD-2-ClauseUpdated 29 Oct 2026

  45. omkarcloud/google-maps-scraper

    Scrapes Google Maps business listings into lead lists with phone, website, rating and social data.

    Lead generation, Enrichment, Google Search, Web, Sales, Marketing, Local business, Agencies, Local SEO

    3.9k starsMITUpdated 19 Oct 2026

  46. vladkens/twscrape

    Python library for collecting X/Twitter posts, profiles and search results through authorized accounts.

    Social listening, X (Twitter), Marketing, Data and analytics, Media and publishing

    2.9k starsMITUpdated 31 Oct 2026

  47. spider-rs/spider

    Rust crawler and scraping library built for high-volume page collection, with optional headless Chrome.

    Web, Engineering, Data and analytics

    2.8k starsMITUpdated 30 Oct 2026

  48. brightdata/brightdata-mcp

    MCP server for web search and scraping through Bright Data's unblocking and proxy network.

    AI agents, Web, Marketing, Data and analytics

    2.7k starsMITUpdated 2 Nov 2026

  49. AWeirdDev/flights

    Python scraper that pulls flight prices and schedules from Google Flights.

    Google Search, Data and analytics, Travel

    2.1k starsMITUpdated 19 Oct 2026

  50. orangecoding/fredy

    Self-hosted bot that watches European rental and property sites and alerts you to new listings.

    Workflow automation, Docker, Telegram, Operations, Real estate, Self hosters

    1.6k starsApache-2.0Updated 2 Nov 2026

  51. eliasdabbas/advertools

    Python library for SEO and ads work: crawling, sitemaps, robots.txt, SERP pulls, keyword generation and log analysis.

    SEO research, Site audit, Google Search, Google Ads, Marketing, Data and analytics, Agencies

    1.5k starsMITUpdated 30 Jun 2026

  52. okfn-brasil/querido-diario

    Scrapes and indexes Brazilian municipal official gazettes so the public can search them.

    RAG and search, Data and analytics, Government, Media and publishing, Data journalism

    1.4k starsMITUpdated 13 Oct 2026

  53. egbertbouman/youtube-comment-downloader

    Downloads YouTube comments without the API for audience research.

    Social listening, YouTube, Marketing, Data and analytics, Creators

    1.3k starsMITUpdated 15 Oct 2026

  54. PhialsBasement/LibreCrawl

    Self-hosted SEO crawler, a free alternative to Screaming Frog, with issue detection and CSV/JSON export.

    Site audit, Web, Docker, Marketing, Agencies, Self hosters

    1.1k starsMITUpdated 2 Nov 2026

  55. flathunters/flathunter

    Bot that scrapes German flat listing sites and sends matching offers to Telegram.

    Workflow automation, Telegram, Operations, Real estate

    1.1k starsAGPL-3.0Updated 19 Apr 2026

  56. freelawproject/courtlistener

    Code behind CourtListener, a searchable archive of US court opinions, dockets and oral arguments.

    RAG and search, Web, Legal and compliance, Legal, Nonprofit, Data journalism

    1.1k starsOtherUpdated 3 Nov 2026

  57. Cybrarist/Discount-Bandit

    Self-hosted price tracker that watches products across stores and alerts on drops.

    Web, Docker, Data and analytics, E-commerce, Retail, Self hosters, Affiliate

    762 starsUpdated 26 Oct 2026

  58. serpapi/google-search-results-python

    Python client for SerpApi that returns Google and other search results as structured JSON.

    SEO research, Rank tracking, Google Search, Marketing, Engineering, Agencies

    761 starsMITUpdated 18 Aug 2026

  59. ZacharyHampton/HomeHarvest

    Python package that scrapes US property listings from Realtor.com into a dataframe.

    Lead generation, Sales, Data and analytics, Real estate

    760 starsMITUpdated 26 Dec 2025

  60. mysociety/theyworkforyou

    Publishes parliamentary debates, votes and representative records in a searchable form.

    Civic participation, Web, Data and analytics, Government, Media and publishing, Data journalism

    263 starsOtherUpdated 2 Nov 2026