Download website to local directory (including all css, images, js, etc.)
-
Updated
Dec 9, 2024 - JavaScript
#
website-scraper
Here are 107 public repositories matching this topic...
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
-
Updated
Mar 10, 2024 - Go
🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON.
nodejs json crawler scraper spider linkedin scraping crawling expressjs linkedin-profile scrapers scraping-websites linkedin-bot website-scraper profile-data linkedin-scraper linkedin-crawler puppeteer linkedin-scraping linkedin-profile-scraper
-
Updated
Apr 5, 2024 - TypeScript
Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.
python osint selenium tor information-extraction websites web-scraping webcrawler webscraping information-gathering website-scraper reconnaissance darkweb selenium-webscraper osint-python webcra osint-tool darkweb-crawler
-
Updated
Nov 24, 2024 - Python
Plugin for website-scraper which returns html for dynamic websites using puppeteer
-
Updated
Dec 9, 2024 - JavaScript
CMS, WebSite, Application and Ecommerce Development Tool Using JavaScript
javascript wordpress cms development templates magento website-builder shopify website-scraper kooboo website-development web-application-platform
-
Updated
Dec 12, 2024 - C#
🕸 Generates RSS feeds of any website & serves to the web! Automatic scraping. Ready to use configs. Write your own. Rolling Docker releases for speedy updates.
ruby docker rss scraper builder feed roda rss-feed rss-aggregator serves rss-feed-scraper website-scraper webfeeds webfeed html2rss rolling-release html2rss-configs feed-configs
-
Updated
Nov 25, 2024 - Ruby
Wayback Machine Downloader. 🔥 Download your entire archived websites from the Internet Archive Wayback Machine.
console command-line-app csharp internet internet-archive command-line-tool console-application wayback-machine command-line-parser website-scraper console-app internet-wayback-machine wayback-machine-downloader
-
Updated
Aug 5, 2022 - C#
DPULSE - Tool for complex approach to domain OSINT
intelligence osint web-scraping cybersecurity pentesting pentest information-security data-gathering webscraping information-gathering website-scraper intelligence-gathering pentest-tool domain-analysis google-dorking osint-tool cybersecurity-education osint-tools infosectools cybersecurity-tool
-
Updated
Dec 12, 2024 - Python
A server to collect & archive websites that also supports video downloads
-
Updated
Feb 11, 2023 - TypeScript
ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.
crawler scraper telegram-bot proxy robots-txt website-scraper rag huggingface large-language-models llm retrieval-augmented-generation ollama
-
Updated
Feb 17, 2024 - Python
Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.
crawler scraper user-agent scraping beautiful-soup robots-txt beautifulsoup scrapper website-scraper scrapping-python website-crawler beautifulsoup4 crawling-python iprotation
-
Updated
Jul 14, 2024 - Python
🌱 goClone - clone websites in seconds
go golang crawler cloner scraping crawling cloning scrapper scraping-websites hacktoberfest website-scraper website-cloner wp2static goclone
-
Updated
Sep 9, 2024 - Go
Automatically curates and posts content to LinkedIn. It can optionally use web scraping to gather data, which is then fed to ChatGPT to craft engaging LinkedIn posts.
linkedin linkedin-bot website-scraper linkedin-scraper post-scheduler openai-api post-scheduling linkedin-post chatgpt-api chatgpt-bot linkedin-posts-automation
-
Updated
Apr 6, 2024 - Python
Plugin for website-scraper which returns html for dynamic websites using PhantomJS.
-
Updated
Dec 29, 2021 - JavaScript
Evaluate JavaScript on a URL through headless Chrome browser.
command-line scrapping cli-utilities evaluator scrapers web-browser scrapper eval web-crawling commandline-interface browser-automation cmdline headless-browsers data-scraping website-scraper datascraping webscrapping web-scrapping pupeteer headless-browser
-
Updated
Jun 18, 2021 - JavaScript
Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.
game crawler scraper marvel website-scraper website-crawler marvel-characters crawler-python marvel-snap
-
Updated
Jul 1, 2024 - Python
Now you can keep track of your followers from YouTube, Instagram and Twitter accounts - Followers scraper API on AWS serverless
aws instagram lambda scraper youtube typescript twitter aws-lambda webscraper instagram-scraper aws-serverless webscraping twitter-scraper website-scraper webscraper-api followers-scraper nodejs-lambda twittersc instagramscraper
-
Updated
Jan 23, 2023 - TypeScript
JSON collection of scraped file extensions, along with their description and type, from FileInfo.com
-
Updated
Nov 9, 2022 - Python
Improve this page
Add a description, image, and links to the website-scraper topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the website-scraper topic, visit your repo's landing page and select "manage topics."