Performance Optimization of Web Crawlers via Parallel and Asynchronous Processing

• Min-Sun Kim,
• Sanghoon Jeon,

Vol. 19, No. 12, December 31, 2025
10.3837/tiis.2025.12.012, Download Paper (Free):

• Crawler Optimization
• Execution Efficiency
• Parallel and Asynchronous Processing
• Resource-Aware Crawling
• Web crawling

Abstract

With the rapid development of the Internet, a vast amount of data is being generated, making various types of information easily accessible. Consequently, big data analysis, which involves collecting, storing, processing, and predicting data, has become increasingly important. Web crawlers have gained attention as tools for extracting data from specific web pages. They are utilized in various fields, including price comparison shopping, Search Engine Optimization (SEO), and Rich Site Summary (RSS) aggregation. Different types of web crawlers rely on static or dynamic crawling methods. Notable web crawlers include Scrapy, Selenium, BeautifulSoup, and Playwright, which are designed to effectively handle either static or dynamic web pages. In this paper, we focus on improving the execution performance of these crawlers by applying two tuning techniques: parallel and asynchronous processing. To evaluate their performance, we used four key metrics: Time per Image (TPI), Images per Second (IPS), CPU utilization, and memory consumption. Through controlled experiments across various web page configurations, we demonstrate how each tuning method affects the execution efficiency and system resource usage of different crawler architectures. Our findings highlight the practical trade-offs between performance and resource efficiency, providing useful insights for applying crawler optimization strategies to real-world data collection tasks.

Statistics

Cite this article