Fetching and Parsing Online Data

Fetching retrieves content; parsing turns that content into a structure your program can inspect. JSON commonly becomes Python dictionaries and lists. HTML becomes a tree of elements that we can search. We will separate these steps so that parsing can be tested without a network connection.

Parse JSON

The standard-library json module converts JSON text into Python values. This complete example uses a small local response, so its result is reproducible:

import json

response_text = '{"id": 1, "title": "Learning Python", "userId": 2}'
post = json.loads(response_text)
print(post["id"])
print(post["title"])

Output

1
Learning Python

loads() reads JSON from a string. Here it returns a dictionary, so the keys "id" and "title" select the values to print. These are sample values, not a recording of a live service.

For a live JSONPlaceholder response, Requests can parse the body directly [1]:

import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/posts/1", timeout=10
)
response.raise_for_status()
post = response.json()
print("Post:", post["id"])
print("Author ID:", post["userId"])

Expected output — requires a successful live request

Post: 1
Author ID: 1

response.json() parses the response body. It can fail if the body is not valid JSON, even when the HTTP request succeeded. The selected endpoint returns one post; endpoints returning several records commonly produce a list that you can loop over. Check an API’s documentation for its actual structure and request limits.

Parse HTML

Beautiful Soup is a third-party library for searching HTML. Install it in the active environment with python -m pip install beautifulsoup4. The imported name is bs4.

This complete example uses a small fictional news page:

from bs4 import BeautifulSoup

html = """
<html><body>
  <h1>Latest News</h1>
  <ul>
    <li class="headline"><a href="/story1">Python Workshop Opens</a></li>
    <li class="headline"><a href="/story2">New Tools for Data Science</a></li>
    <li class="headline"><a href="/story3">Digital Humanities in Practice</a></li>
  </ul>
</body></html>
"""

soup = BeautifulSoup(html, "html.parser")
headlines = []
for item in soup.find_all("li", class_="headline"):
    link = item.find("a")
    if link is not None and link.has_attr("href"):
        headlines.append((link.get_text(strip=True), link["href"]))

for title, url in headlines:
    print(f"{title} -> {url}")

Output

Python Workshop Opens -> /story1
New Tools for Data Science -> /story2
Digital Humanities in Practice -> /story3

BeautifulSoup builds a tree using Python’s html.parser. find_all() returns the matching list items; class_ searches the HTML class attribute because class itself is a Python keyword. Within each item, find() looks for a link and can return None. The condition checks that a link and its href exist before reading them. get_text(strip=True) extracts text and trims whitespace; link["href"] reads the destination. The resulting pairs are ordinary Python data.

To parse a downloaded page, pass a successful response’s text to BeautifulSoup instead of the local html string. Search rules must match that page’s structure. Content loaded later by JavaScript may be absent from the downloaded HTML.

Before combining these steps into a scraper, we need to consider the site’s access rules and the people whose data it contains.

References

1. JSONPlaceholder. Guide.
2. Beautiful Soup. Searching the tree.