Beautiful Soup Cheatsheet

Beautiful Soup is a Python library that turns messy HTML into a searchable parse tree. Pair it with requests (or another HTTP client) when the page's first response already contains the data you need.

Install & Fetch

Install the parser stack and download a page.

# pip install beautifulsoup4 requests lxml
import requests
from bs4 import BeautifulSoup

html = requests.get(
    "https://example.com",
    headers={"User-Agent": "MyBot/1.0 (+https://example.com/bot)"},
    timeout=30,
).text
soup = BeautifulSoup(html, "lxml")

Find Elements

CSS-like lookups for tags and classes.

soup.find("h1")
soup.find_all("a")
soup.select("article h2")
soup.select_one(".price")
soup.find("div", class_="card")
soup.find(attrs={"data-id": "42"})

Text & Attributes

Pull strings and href/src values.

el = soup.select_one("a.product")
el.get_text(strip=True)
el["href"]
el.get("href")
el.get("class", [])

Navigate the Tree

Move relative to a node you already found.

el.parent
el.find_parent("section")
el.find_next_sibling()
el.find_previous_sibling()
list(el.children)
el.descendants  # recursive generator

Clean & Export

Normalize text and build rows.

rows = []
for card in soup.select(".item"):
    rows.append({
        "title": card.select_one("h2").get_text(strip=True),
        "url": card.select_one("a")["href"],
    })

---

Beautiful Soup shines on static or server-rendered HTML. For pages that fill in with JavaScript, render first with Playwright, then pass page.content() into Beautiful Soup — or extract with Playwright locators directly.

For full documentation, see https://www.crummy.com/software/BeautifulSoup/bs4/doc/