feat: Added crawler template.

This commit is contained in:
Spencer Brower
2025-09-12 12:15:27 -04:00
parent 313051a9bd
commit a95cf11b08
8 changed files with 3921 additions and 0 deletions

View File

@@ -0,0 +1,26 @@
# TopMarket Scraper
A web scraper built with Crawlee for JavaScript.
## Setup
1. Install dependencies:
```bash
npm install
```
2. Run the scraper:
```bash
npm start
```
## Configuration
Edit `src/main.js` to:
- Change the `startUrls` array to target your desired websites
- Modify the `requestHandler` to extract the data you need
- Adjust `maxRequestsPerCrawl` to control crawling limits
## Output
Scraped data is saved to the `storage/datasets/default` directory in JSON format.