← BACK TO THE SIMPLE VERSION
PROPER INSTRUCTIONS

Make a simple Python web scraper: proper instructions

This guide creates a one-page link scraper using Python’s standard library. It downloads public HTML, extracts anchor links, resolves relative addresses, removes duplicates and writes a CSV. The small scope matters: a one-page fetch is easier to inspect and avoids treating a website as an unlimited data source. Scrape only pages you are authorised to access. Check published terms, robots.txt and rate limits, and never bypass logins, CAPTCHAs, paywalls, technical blocks or other access controls.

Prerequisites

• Python 3.9 or later, a terminal and a text editor.

• A public HTML page you own or are expressly allowed to fetch for the first test.

• An internet connection and optionally a spreadsheet, although a text editor is best for first CSV inspection.

• An honest contact address if the target’s policy expects a User-Agent contact field.

Numbered actions

1. Create a directory named simple-scraper, then create links.py.

2. Paste this program. It makes one request only, has a 15-second timeout and accepts HTML rather than arbitrary downloaded data.

import csv,sys
from html.parser import HTMLParser
from urllib.request import Request,urlopen
from urllib.parse import urljoin,urlparse
from urllib.error import HTTPError,URLError
class P(HTMLParser):
 def __init__(self,b): super().__init__();self.b=b;self.rows=[];self.h=None;self.t=[]
 def handle_starttag(self,x,a):
  if x=="a" and dict(a).get("href"): self.h=urljoin(self.b,dict(a)["href"]);self.t=[]
 def handle_data(self,x):
  if self.h is not None:self.t.append(x)
 def handle_endtag(self,x):
  if x=="a" and self.h is not None:self.rows.append((self.h," ".join("".join(self.t).split())));self.h=None
if len(sys.argv)!=3:raise SystemExit("Usage: python3 links.py URL output.csv")
u,out=sys.argv[1:]
if urlparse(u).scheme not in ("http","https"):raise SystemExit("Use http:// or https://")
try:
 r=Request(u,headers={"User-Agent":"LearningLinkScraper/1.0 (contact: you@example.invalid)"})
 with urlopen(r,timeout=15) as q:
  if q.headers.get_content_type() not in ("text/html","application/xhtml+xml"):raise ValueError("Response is not HTML")
  html=q.read().decode(q.headers.get_content_charset() or "utf-8",errors="replace")
except HTTPError as e:raise SystemExit(f"HTTP {e.code}: {e.reason}")
except URLError as e:raise SystemExit(f"Network error: {e.reason}")
p=P(u);p.feed(html);seen=set();rows=[x for x in p.rows if x[0].startswith(("http://","https://")) and not(x[0] in seen or seen.add(x[0]))]
with open(out,"w",newline="",encoding="utf-8") as f:w=csv.writer(f);w.writerow(["url","text"]);w.writerows(rows)
print(f"Wrote {len(rows)} unique links")

3. Run python3 links.py https://example.org links.csv, replacing the address with an authorised public page. Use py instead where appropriate.

4. Open the CSV in a text editor. It has url,text as its first row; blank link text is valid for icon and image links.

5. Inspect the page source and compare several expected links with the CSV. The program sees the server’s HTML, not content added later by browser JavaScript.

6. Do not put the command in a fast loop. A crawler requires separate permission, domain scope, delay, retry, storage and deletion decisions.

Checking and testing

1. Make a local page with an absolute link, a relative link and a duplicate. Serve its directory with python3 -m http.server 8000.

2. Run python3 links.py http://localhost:8000 local.csv from another terminal. Confirm relative links become absolute localhost URLs and duplicates occur once.

3. Try a PNG address. The program should reject it as non-HTML rather than decode binary data.

4. Try an invalid host. It should display a concise network error, not leave a partial CSV.

Common mistakes

• Claiming an HTTP 200 response proves the desired data was present.

• Removing the timeout and allowing an incomplete server response to hang the tool.

• Impersonating a browser or putting fake contact information in the User-Agent.

• Assuming browser-visible JavaScript content is available in the original HTML.

• Extending a simple fetch to authenticated or rate-intensive collection without authorisation.

Rollback and safety

The CSV is the only persistent output, so deleting it is rollback. Stop the local test server with Ctrl+C. A safe next extension is filtering to the same host by comparing urlparse(link).netloc to the starting host. Do not add automatic submissions, account actions or unrestricted recursive traversal.

Interpret results responsibly

A CSV is a record of what the server returned to this request at that time, not a permanent copy of the page and not proof that every link is usable. Links can redirect, be geographically varied, be generated by scripts, or lead to a different host. Keep the source URL and the time of collection with any results you plan to analyse later. If you need reproducibility for your own permitted page, save a copy of the raw response separately and do not publish it if it contains material you lack permission to redistribute.

Respectful collection means keeping request volume low and stopping when a site returns an error, asks you to slow down, or changes its policy. A robots.txt file is a published crawler preference rather than blanket permission; terms and law may impose additional conditions. The safe default is one manual fetch, then review. If your task needs a large structured export, look first for an official download, documented API, RSS feed or contact route. Those routes are usually more stable and fairer to the site operator than scraping presentation HTML.

Improve reliability without expanding scope

Keep the URL validation and response content-type check. They prevent accidental use of local paths or binary files. Use UTF-8 for CSV so non-English link text is preserved. A spreadsheet can display long URLs in a misleading format, so retain the original CSV and inspect it in a text editor when validating. Never put credentials, cookies or tokens into this introductory script. If a public page needs consent banners or browser rendering before content appears, that is a reason to reconsider the data source, not an invitation to bypass the page’s controls.