Working with APIs

What an API call is

You send an HTTP request to a URL and get a response, usually JSON. The parts that matter: the method (GET to read, POST to send), the status code, and the body.

requests

Not in the standard library, and the one nearly everybody uses:

bash

python -m pip install requests

Python

import requests

response = requests.get("https://api.example.com/people/1")
print(response.status_code)

data = response.json()
print(data["name"])

Output, from a real run elsewhere

200
Ada

Always check the status

requests does not raise on a 404 or a 500. You get a response object saying it went wrong, and ignoring that means parsing an error page as if it were data:

Python

import requests

response = requests.get("https://api.example.com/missing")

if response.status_code == 200:
    print(response.json())
else:
    print("failed with", response.status_code)

Output, from a real run elsewhere

failed with 404

response.raise_for_status() turns a bad status into an exception, which is often what you want inside a try.

The ranges are worth memorising: 2xx worked, 3xx redirect, 4xx you got it wrong, 5xx they got it wrong.

Query parameters

Pass a dictionary rather than building the URL by hand, so encoding is handled:

Python

import requests

response = requests.get(
    "https://api.example.com/search",
    params={"q": "ada lovelace", "limit": 10},
)
print(response.url)

Output, from a real run elsewhere

https://api.example.com/search?q=ada+lovelace&limit=10

Headers and authentication

Python

import requests

response = requests.get(
    "https://api.example.com/me",
    headers={
        "Authorization": "Bearer " + token,
        "Accept": "application/json",
    },
    timeout=10,
)

Python

import os

token = os.environ.get("API_TOKEN")
if token is None:
    print("API_TOKEN is not set; refusing to continue")
else:
    print("token found")

Output

API_TOKEN is not set; refusing to continue

Always set a timeout

Without one, a hung server hangs your program indefinitely:

Python

import requests

try:
    response = requests.get("https://api.example.com/slow", timeout=5)
except requests.Timeout:
    print("gave up after five seconds")

Output, from a real run elsewhere

gave up after five seconds

Sending data

Python

import requests

response = requests.post(
    "https://api.example.com/people",
    json={"name": "Ada", "born": 1815},
    timeout=10,
)
print(response.status_code)

Output, from a real run elsewhere

Use json= and requests sets the content type and encodes the body. Use data= only for form-encoded submissions.

Handling the response

This part is ordinary Python, and it does run here:

Python

import json

body = '{"results": [{"name": "Ada", "born": 1815}, {"name": "Grace"}], "next": null}'
data = json.loads(body)

for person in data["results"]:
    born = person.get("born", "unknown")
    print(f"{person['name']}: {born}")

print("more pages?", data["next"] is not None)

Output

Ada: 1815
Grace: unknown
more pages? False

JSON null becomes Python None, so there is no next page. Use get() with a default for fields an API may omit, as with Grace's missing birth year.

Paging

Most APIs return results in pages. The shape is nearly always the same:

Python

import requests

url = "https://api.example.com/people"
names = []

while url:
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    payload = response.json()
    names.extend(person["name"] for person in payload["results"])
    url = payload.get("next")

print(len(names))

Be a good client

Send a User-Agent that identifies you
Respect rate limits, and back off when you see a 429
Cache what does not change
Read the terms before scraping anything

The standard library alternative

If one call is all you need, urllib avoids a dependency:

Python

import json
from urllib.request import urlopen

with urlopen("https://api.example.com/people/1") as response:
    data = json.load(response)

print(data["name"])

Output, from a real run elsewhere

Ada

It is clumsier for anything with headers or authentication, which is exactly why requests is so widely used.

What an API call is

requests

Always check the status

Query parameters

Headers and authentication

Always set a timeout

Sending data

Handling the response

Paging

Be a good client

The standard library alternative

Test yourself

What does requests do when the server returns a 404?

Why always pass a timeout?

Testing