Applications of Web Scraping

Though it’s becoming more common for web sites to expose their data using web services, the absence of a data source that is tailored to machines and offers all the data of a corresponding web site is still a common situation. In these instances, the web site itself must effectively become your data source, and web scraping can be employed to automate the consumption of the data it makes available. Additionally, web services are also used to transfer information into external data systems. In their absence, web scraping can also be used to integrate with such systems that don’t offer web services, but do offer a web-based interface for their users.

Another application of web scraping that is likely more well-known is the development of automated agents known as crawlers, which seek out resources for storage and analysis that will eventually comprise the search results they deliver to you. In the earliest days of the internet, this type of data was sought out manually by human beings, a slow and tedious process which limited how quickly a search engine could expand its offerings. Web scraping provided an alternative to allow computers to do the grunt work of finding new pages and extracting their content.

Lastly, web scraping is one way - not the only way or necessarily the recommended way, but certainly a way - to implement integration testing for web applications. Using its abilities to act as a client in extracting and transmitting data, a web scraping application can simulate the browser activity of a normal user. This can help to ensure that web application output complies with its expected response with respect to the application's requirements.