An Interest In:
Web News this Week
- April 28, 2024
- April 27, 2024
- April 26, 2024
- April 25, 2024
- April 24, 2024
- April 23, 2024
- April 22, 2024
April 26, 2023 08:46 am GMT
Original Link: https://dev.to/coderhxl/x-crawl-v7-has-been-released-54pk
x-crawl v7 has been released
x-crawl
x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files.
If you also like x-crawl, you can give x-crawl repository a star to support it, thank you for your support!
Features
- Asynchronous Synchronous - Just change the mode property to toggle asynchronous or synchronous crawling mode.
- Multiple purposes - It can crawl pages, crawl interfaces, crawl files and poll crawls to meet the needs of various scenarios.
- Flexible writing style - The same crawling API can be adapted to multiple configurations, and each configuration method is very unique.
- Interval Crawling - No interval, fixed interval and random interval to generate or avoid high concurrent crawling.
- Failed Retry - Avoid crawling failure due to short-term problems, and customize the number of retries.
- Proxy Rotation - Auto-rotate proxies with failure retry, custom error times and HTTP status codes.
- Device Fingerprinting - Zero configuration or custom configuration, avoid fingerprinting to identify and track us from different locations.
- Priority Queue - According to the priority of a single crawling target, it can be crawled ahead of other targets.
- Crawl SPA - Crawl SPA (Single Page Application) to generate pre-rendered content (aka "SSR" (Server Side Rendering)).
- Control Page - You can submit form, keyboard input, event operation, generate screenshots of the page, etc.
- Capture Record - Capture and record crawling, and use colored strings to remind in the terminal.
- TypeScript - Own types, implement complete types through generics.
Example
Take the automatic acquisition of some photos of experiences and homes around the world every day as an example:
// 1.Import module ES/CJSimport xCrawl from 'x-crawl'// 2.Create a crawler instanceconst myXCrawl = xCrawl({maxRetry: 3,intervalTime: { max: 3000, min: 2000 }})// 3.Set the crawling task/* Call the startPolling API to start the polling function, and the callback function will be called every other day*/myXCrawl.startPolling({ d: 1 }, async (count, stopPolling) => { // Call crawlPage API to crawl Page const res = await myXCrawl.crawlPage({ targets: [ 'https://www.airbnb.cn/s/experiences', 'https://www.airbnb.cn/s/plus_homes' ], viewport: { width: 1920, height: 1080 } }) // Store the image URL to targets const targets = [] const elSelectorMap = ['._fig15y', '._aov0j6'] for (const item of res) { const { id } = item const { page } = item.data // Wait for the page to load await new Promise((r) => setTimeout(r, 300)) // Gets the URL of the page image const urls = await page!.$$eval( `${elSelectorMap[id - 1]} img`, (imgEls) => { return imgEls.map((item) => item.src) } ) targets.push(...urls) // Close page page.close() } // Call the crawlFile API to crawl pictures myXCrawl.crawlFile({ targets, storeDir: './upload' })})
running result:
Note: Do not crawl at will, you can check the robots.txt protocol before crawling. This is just to demonstrate how to use x-crawl.
More
For more detailed documentation, please check: https://github.com/coder-hxl/x-crawl
Original Link: https://dev.to/coderhxl/x-crawl-v7-has-been-released-54pk
Share this article:
Tweet
View Full Article
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To