An Interest In:
Web News this Week
- April 27, 2024
- April 26, 2024
- April 25, 2024
- April 24, 2024
- April 23, 2024
- April 22, 2024
- April 21, 2024
Some of Our Sources
- BoingBoing
- Technology Review
- Smashing Apps
- Six Revisions
- TutsPlus - Design
- You The Designer
- Fudge Graphics
- Web Design Ledger
- 24 Ways
- Design Modo
Help Webnuz
Referal links:
July 30, 2022 11:06 am GMT
Original Link: https://dev.to/ductnn/simple-tool-crawl-urls-form-domain-7m2
Simple tool crawl urls form domain
cUrls is a simple tool crawl urls from domain using colly
library.
Installation
First, install golang
Then, clone from soure code and install:
git clone https://github.com/ductnn/cUrls.gitcd cUrlsgo getUsage
Run command:
go run curls.go > sub.txt# Enter domain you want to crawl.# Examplehttp://httpbin.org/Check results in file sub.txt:
Visiting http://httpbin.org/Link found: "
" -> https://github.com/requests/httpbinLink found: "the developer - Website" -> https://kennethreitz.orgLink found: "Send email to the developer" -> mailto:[email protected] found: "Flasgger" -> https://github.com/rochacbruno/flasggerLink found: "HTML form" -> /forms/postSo done !!! =))))
Show your support
Give a if you like this application
Contribution
Contributions are more than welcome in this project!
License
The MIT License (MIT). Please see LICENSE for more information.
Original Link: https://dev.to/ductnn/simple-tool-crawl-urls-form-domain-7m2
Share this article:
Tweet
View Full Article
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To