Definition
A script that needs to talk to a web service first has to speak HTTP: open a connection, build a request, set the right headers, wait for a response, then decode whatever comes back. Each step is a place where a small mistake can slip in, especially when the result only shows up after fifteen lines of code. requests exists to erase that mechanics and hand you directly what you actually care about: the server's response.
It is not part of the standard library, so it has to be installed with pip, which downloads it from PyPI.
python -m pip install requestsOnce installed, a complete call to a web service fits in three lines.
import requests
response = requests.get("https://api.example.com/articles", timeout=10)
print(response.status_code)
print(response.json())Behind those three lines, several decisions have already been made on your behalf: encoding parameters into the address, following redirects, keeping cookies around, decompressing the body and decoding the text according to the header the server announced. That invisible work is what made this library the default way to call a web interface.
What it replaces
Python already knows how to speak HTTP on its own, through the urllib.request module of the standard library. Almost nobody uses it for daily work, so it is worth looking at why in the table below.
| Need | With urllib.request | With requests |
|---|---|---|
| Add filters to the address | Encode them by hand | The params argument |
| Send JSON | Serialise, encode to bytes, set the header | The json argument |
| Read the body as text | Read bytes, then decode them | The text attribute |
| Keep cookies around | Build an opener and its handler | The Session object |
| Authenticate | Assemble the header yourself | The auth argument |
Nothing in the middle column is genuinely impossible: it is simply fifteen lines where one is enough, with as many chances to get an encoding wrong. The official Python documentation itself points to this library for everyday needs, which is rare for a project that is not part of the language.
The two traps that cost an evening
The first catches almost everyone once: a 404 or 500 response raises no exception at all. It is tempting to assume an error code stops the program, the way a division by zero would. It does not: as far as the library is concerned, the server answered, so the call succeeded. The script carries on, treating an error page as valid data, sometimes until an unrelated crash three functions later. The fix is one line, placed right after the call.
response = requests.get(url, timeout=10)
response.raise_for_status() # raises on 4xx and 5xx
data = response.json()Without the timeout argument, there is no waiting limit at all. A server that accepts the connection and then stays silent blocks the program indefinitely, with no message and no error, until someone stops it by hand. Put a timeout on every call, including the ones aimed at your own server.
params, data, json: the trio people swap
Three arguments exist to send something to the server, and they are not interchangeable: mixing them up produces a request the server refuses, without always saying why.
| Argument | Where the content goes | When to use it |
|---|---|---|
| params | Into the address, after the question mark | Filters and pagination on a read |
| data | Into the body, in form format | A classic web form |
| json | Into the body, serialised, header set | A modern API, by far the common case |
The symptom is easy to recognise: a 400 or 422 response on a call that seemed correct. Nine times out of ten, the cause is right there, a dictionary sent through data while the service was expecting JSON.
When it is of no use
It does not replace a browser. It fetches exactly the document the server returns, nothing more: if the page builds part of its content in JavaScript, that content will not exist in the response. Hunting for text plainly visible on screen and nowhere to be found in what the library received is the rite of passage of every first data extraction project.
It blocks, too. Every call waits for its answer before handing control back, which suits a script querying ten addresses perfectly, and turns into a bottleneck the moment it has to query a thousand. Parallel work belongs to asyncio and to a client built for it, such as httpx.
Finally, it is a client, not a server. It calls web interfaces, it publishes none: that role belongs to Flask, Django or FastAPI.
Frequently asked questions
Should a Session be used rather than isolated calls?
As soon as the same host is queried more than once, yes. A Session reuses the network connection instead of reopening it on every call, and keeps cookies and shared headers around, which is easy to measure over a loop of a hundred requests. It is used as a context manager, inside a with block, so it closes cleanly even if a call fails.
Why does the import fail when the installation succeeded?
Because the installation and the execution do not necessarily go through the same interpreter. The command dropped the library somewhere, the script looks for it elsewhere, and the result is a ModuleNotFoundError about a library that is genuinely present on the machine. Running python -m pip install after activating the project virtual environment rules the problem out for good.
Is httpx the better choice today?
Not as a matter of principle. httpx keeps almost the same writing while adding asynchronous mode and HTTP/2, which matters for a service firing hundreds of simultaneous calls. On a script, a scheduled job or a project calling three interfaces one after the other, the difference is not measurable, and the better documented of the two keeps a real advantage the day something breaks.