Pandas Intro

DataFrames

Pandas is built on NumPy and adds the thing NumPy lacks: labelled columns of mixed types. A DataFrame is a table.

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({
    "name": ["Ada", "Grace", "Katherine"],
    "born": [1815, 1906, 1918],
    "field": ["maths", "computing", "maths"],
})

print(df)

Output

        name  born      field
0        Ada  1815      maths
1      Grace  1906  computing
2  Katherine  1918      maths

That leading column of numbers is the index. Every row has a label, and by default it is its position.

Looking at it

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({"name": ["Ada", "Grace"], "born": [1815, 1906]})

print(df.shape)
print(list(df.columns))
print(df["name"].dtype.kind, df["born"].dtype.kind)

Output

(2, 2)
['name', 'born']
O i

The kind letters are the portable way to ask: i is integer, f float, O object. A text column reports O here; recent pandas versions have their own string dtype, so print kind rather than the dtype itself if you need an answer that holds across versions.

df.head(), df.info() and df.describe() are the three commands you run first on any real dataset.

Columns and rows

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({
    "name": ["Ada", "Grace", "Katherine"],
    "born": [1815, 1906, 1918],
})

print(df["name"].tolist())
print(df.loc[1, "name"])
print(df.iloc[0].to_dict())

Output

['Ada', 'Grace', 'Katherine']
Grace
{'name': 'Ada', 'born': 1815}

loc uses labels, iloc uses positions. Mixing them up is the most common pandas confusion, and the names are the mnemonic: label and integer.

Filtering

The same boolean masking as NumPy, with column names:

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({
    "name": ["Ada", "Grace", "Katherine"],
    "born": [1815, 1906, 1918],
})

print(df[df["born"] > 1900]["name"].tolist())
print(df[(df["born"] > 1900) & (df["name"] != "Grace")]["name"].tolist())

Output

['Grace', 'Katherine']
['Katherine']

New columns

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({"name": ["Ada", "Grace"], "born": [1815, 1906]})
df["century"] = df["born"] // 100 + 1
print(df)

Output

    name  born  century
0    Ada  1815       19
1  Grace  1906       20

The arithmetic applies to the whole column at once, exactly as in NumPy.

Grouping

The operation that earns pandas its place:

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({
    "field": ["maths", "computing", "maths"],
    "born": [1815, 1906, 1918],
})

print(df.groupby("field")["born"].mean().to_dict())
print(df.groupby("field").size().to_dict())

Output

{'computing': 1906.0, 'maths': 1866.5}
{'computing': 1, 'maths': 2}

Split by a column, apply a summary to each group, combine the results. Writing that by hand is a dictionary, a loop and several chances to get it wrong.

Sorting and missing values

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({"name": ["Ada", "Grace", "Alan"], "born": [1815, None, 1912]})

print(df.sort_values("name")["name"].tolist())
print(df["born"].isna().sum())
print(df.dropna().shape)
print(df["born"].fillna(0).tolist())

Output

['Ada', 'Alan', 'Grace']
1
(2, 2)
[1815.0, 0.0, 1912.0]

Missing data is NaN, and pandas has an answer for it at every step — which is most of why it exists.

Reading and writing files

Python needs pandas, downloaded on first run

import pandas as pd

df = pd.DataFrame({"name": ["Ada"], "born": [1815]})
df.to_csv("people.csv", index=False)

again = pd.read_csv("people.csv")
print(again.to_dict("records"))

Output

[{'name': 'Ada', 'born': 1815}]

index=False stops the row labels being written as an extra unnamed column, which is the single most common annoyance when a CSV comes back wrong.

read_csv also reads a URL, and there are read_json, read_excel and read_sql alongside it.

DataFrames

Looking at it

Columns and rows

Filtering

New columns

Grouping

Sorting and missing values

Reading and writing files

Test yourself

What is the difference between loc and iloc?

Why use & rather than 'and' when filtering a DataFrame?

Matplotlib Intro