Appropriate Use of Web Scraping

Some data providers may only offer a web site while others may offer APIs that do not offer equivalent data in a conducive format or at all. Some web services may involve an extensive authentication process or be unavailable for public consumption. Unreliability of web service endpoints compared to web sites may also make them unfeasible to use. It is in situations like these that web scraping becomes a desirable alternative.

It is common practice to avoid making changes that break backward-compatibility with existing applications that use web service APIs, or to version them to allow application vendors time to transition to new revisions. As such, web services are significantly less prone to be altered than the markup structure of pages on a web site, especially without advance warning. This is especially true of sites that change frequently, which can drastically affect the stability of applications that employ web scraping.

In a nutshell, web scraping should be used as a last resort when no other feasible options for acquiring the needed data are available.