Project: Fetch & Parse Real Website Data

Build a small collection of article titles and introductory paragraphs. We will test extraction with local HTML, then connect the same function to live Wikipedia pages. This separates parsing errors from network failures and keeps repeated tests offline.

The project uses requests and beautifulsoup4, introduced in the previous sections. Install them in your active environment if necessary. Before fetching live pages, review Wikimedia’s current access and content-reuse policies; its documented APIs are often preferable for a larger application [1].

Extract an Article

The extraction function accepts HTML and returns a dictionary. It searches for the article heading and the first non-empty paragraph in the article content. Missing elements raise a clear error rather than causing an attribute-access error.

from bs4 import BeautifulSoup

def extract_article(html):
    soup = BeautifulSoup(html, "html.parser")
    heading = soup.find("h1")
    content = soup.select_one("#mw-content-text")
    if heading is None or content is None:
        raise ValueError("Article heading or content is missing")
    for paragraph in content.find_all("p"):
        intro = paragraph.get_text(" ", strip=True)
        if intro:
            return {"title": heading.get_text(" ", strip=True), "intro": intro}
    raise ValueError("Article has no non-empty paragraph")

Defining the function produces no output. select_one("#mw-content-text") finds the element with that HTML ID. Passing a space to get_text() separates text from adjacent tags. The search is a simple extraction rule, not a guarantee that every Wikipedia layout puts the introduction first; inspect and adapt it when the page structure changes.

Test with Local HTML

Run the function definition above, then this example. The paragraphs are short, original test fixtures, not copied Wikipedia introductions.

samples = [
    ("Python (programming language)", "Python programs can process text and numbers."),
    ("Artificial intelligence", "AI systems perform tasks using computational models."),
    ("Web scraping", "A scraper extracts selected information from web pages."),
]
articles = []
for title, intro in samples:
    html = f'<h1>{title}</h1><div id="mw-content-text"><p></p><p>{intro}</p></div>'
    articles.append(extract_article(html))

for article in articles:
    print(article["title"])
    print(article["intro"])

Output

Python (programming language)
Python programs can process text and numbers.
Artificial intelligence
AI systems perform tasks using computational models.
Web scraping
A scraper extracts selected information from web pages.

The empty first paragraph is deliberately skipped. Each successful extraction adds one dictionary to articles; the final loop reads its two fields. Change a fixture’s heading or paragraph and verify that the extracted value changes. Then remove the heading and check that the function raises its stated ValueError.

Connect the Parser to Live Pages

Keep extract_article above this code in the same script. It fetches three pages sequentially. Enter a real contact email or project URL when prompted; the program uses it to identify the client. Check the site’s current crawler instructions, access rules and request limits before running it.

import time
import requests

contact = input("Your contact email or project URL: ").strip()
if not contact:
    raise ValueError("Contact information is required")
headers = {"User-Agent": f"ArticleStudy/1.0 ({contact})"}
urls = [
    "https://en.wikipedia.org/wiki/Python_(programming_language)",
    "https://en.wikipedia.org/wiki/Artificial_intelligence",
    "https://en.wikipedia.org/wiki/Web_scraping",
]
articles = []
for url in urls:
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        article = extract_article(response.text)
        article["url"] = url
        articles.append(article)
    except (requests.RequestException, ValueError) as error:
        print("Stopped:", error)
        break
    time.sleep(2)

for article in articles:
    print(article["title"])
    print(article["intro"][:200])
    print(article["url"])

Output — depends on the live pages and connection

The program first asks for contact information. On success it prints a title, up to 200 introduction characters and a source URL for each retrieved page. On failure it prints Stopped: and the error, then displays any records already collected. No fixed live transcript is given because pages, access conditions and error messages can change.

The exception handler covers HTTP, connection and parsing failures. It stops after a failure rather than repeatedly requesting an unavailable resource. The two-second pause is an example delay, not a substitute for the service’s current limits. The saved URL supports source tracking; check licensing and attribution requirements before redistributing extracted text.

Apply: save the collected dictionaries as JSON using the file-handling techniques introduced earlier. Test saving with the local fixtures first. Your program should preserve the titles and introductions and handle a missing heading without inventing data.

Reference

1. Wikimedia. API access policy and REST API documentation.