Ethics & Responsible Web Scraping

Automated requests use a site’s resources and may collect information about people. Before writing a scraper, check whether automated access and your intended use are permitted. Prefer a documented API when it provides the data you need, and read its access, licensing and rate-limit policies.

Check Access Rules

A site can publish crawler instructions at its root, for example https://example.org/robots.txt. The following illustrative file asks all crawlers to avoid /private/:

User-agent: *
Disallow: /private/
Allow: /public/

User-agent: * identifies rules for crawlers without a more specific matching group. Disallow and Allow specify paths. These are crawler instructions, not access authorization; the robots standard explicitly distinguishes them from security controls [1]. Check the site’s terms and applicable permissions as well. Do not bypass authentication, paywalls or other access restrictions.

Limit Requests and Identify Your Program

Follow the service’s published request limits. Cache results, fetch only what you need, and avoid repeated downloads while testing your parser. If the service responds with 429, stop and follow its retry instructions, including Retry-After when present. A fixed one-second pause is not a universal guarantee of acceptable traffic.

Use a meaningful User-Agent when the service requests one. Wikimedia, for example, asks API clients to identify their application and provide contact information [2]. Use your own real details rather than impersonating a browser or copying somebody else’s contact address.

Collect and Reuse Carefully

Public visibility does not imply unrestricted reuse. Avoid collecting unnecessary personal information, check content licenses, and retain the source information needed for attribution. Share only what your permissions and the applicable license allow.

We will first test the next project’s parser with local HTML. That lets us check our code repeatedly without sending repeated requests to a real website.

References

1. Koster, M. et al. (2022). RFC 9309: Robots Exclusion Protocol.
2. Wikimedia. API access policy.