Website Crawler for LLM Training

Extract content from any website for LLM training. I use this to help me with client coding or digital marketing projects. I also see myself grabbing content from a client website or other resources online and pasting it into a document and uploading to LLM, or my projects like AI Receptionist, Legal AI Writer. Hopefully this is useful for other people.

This project was inspired by DeepCrawl.dev and Firecrawler.dev

2 levels
1 2 3 4 5
50 pages
10 30 50 70 100

Comma-separated path prefixes. Only URLs under these paths will be crawled.

Comma-separated path prefixes. URLs under these paths will be skipped.

Crawls pages by following internal links up to the configured depth.

Carlos Arias - AI Engineer & Digital Marketing Strategist.

Initializing crawler...

Crawl Results

0 pages 0s