Web Crawler
Full-Stack Project
offline
Swipe to see more
A fully Dockerized full-stack web crawler application that allows users to submit, queue, and analyze URLs. Built with a robust backend in Express and Prisma (MySQL) and a modern frontend using React and TailwindCSS, it features real-time crawling updates via WebSockets, a visual dashboard for insights, and CRUD capabilities for URLs. The app is production-ready with Prisma migrations and seed logic auto-executed via Docker.
Tech Stack
React
TypeScript
Tailwind
Node.js
React Icons
Recharts
Express.js
Prisma ORM
MySQL
Docker
WebSocket (Socket.io)
Zod
axios
cheerio
http-errors
Features
- ✔ URL Submission
- ✔ Queue Management
- ✔ Real-Time Crawling Updates
- ✔ Broken Link Detection
- ✔ Status Code & Meta Analysis
- ✔ Reanalyze URLs
- ✔ Bulk Deletion
- ✔ Stats Dashboard
- ✔ Dockerized Environment
- ✔ Prisma Migration & Seed
- ✔ Responsive UI
API Routes
URL Crawling Routes
GET
/api/urls/Get all crawled URLs
GET
/api/urls/queuedGet queued URLs
GET
/api/urls/statsGet crawl statistics
GET
/api/urls/:urlIdGet single URL details
POST
/api/urls/Crawl a URL immediately
POST
/api/urls/queueAdd a URL to the queue
PUT
/api/urls/crawl-queuedStart crawling all queued URLs
PUT
/api/urls/crawl-selectedStart crawling selected URLs
PUT
/api/urls/:urlId/reanalyzeReanalyze a specific URL
URL Deletion Routes
DELETE
/api/urls/:urlIdDelete a specific URL
DELETE
/api/urls/Bulk delete selected URLs