Quick intro to pandas¶
MCS 275 Spring 2021 - Emily Dumas¶
This is a quick tour of basic plotting with matplotlib. For more detail see
You can install pandas on your own machine (e.g. python3 -m pip install pandas). It is also available on Google Colab.
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
In [32]:
# Prettification options
# Nice colors
plt.style.use("Solarize_Light2")
# Big text options
params = {'axes.labelsize': 18,
'axes.titlesize':32,
'font.size': 20,
'legend.fontsize': 20,
'xtick.labelsize': 16,
'ytick.labelsize': 16}
# Global params like default sizes live in mpl not plt
mpl.rcParams.update(params)
Opening files¶
Excel spreadsheet of animal data¶
In [49]:
df = pd.read_excel("data/animals.xlsx")
df
Out[49]:
In [ ]:
# In columns of type float, pandas uses "NaN" (not a number)
# to indicate missing data. In this case, the worldwide
# population of capybaras is not known.
HTML table (UIC academic calendar)¶
In [52]:
# read_html takes a URL, string, or urllib response object
# and gives a list of DataFrames, one for each table in the
# document.
# Here, we ask for just the *first* table
# (corresponding to summer 2019)
pd.read_html(
"https://catalog.uic.edu/ucat/academic-calendar/"
)[0]
Out[52]:
CSV of electrical energy consumption in Europe¶
This is the example we'll work with for the rest of lecture.
In [8]:
df = pd.read_csv(
"data/europe-electricity/europe-electricity-2017-to-2019.csv",
parse_dates=True,
index_col="date" # refers to name in header row
)
In [10]:
type(df)
# DataFrame = table
Out[10]:
In [13]:
df["France"] # one column of the table
Out[13]:
In [14]:
type(df["France"])
# Series = column
Out[14]:
In [15]:
pd.Timestamp("2018-04-28")
Out[15]:
In [16]:
# DataFrame[column_name][index]
df["United Kingdom"][pd.Timestamp("2018-04-28")]
Out[16]:
In [17]:
t0 = pd.Timestamp("2018-04-28")
t1 = t0 + pd.Timedelta(days=7)
In [19]:
df["United Kingdom"][t0:t1]
# Gotcha: in pandas, slices in the index include end!
Out[19]:
In [20]:
df.loc[t0:t1] # all the rows from t0 to t1
Out[20]:
In [23]:
# all rows where Italy < 30k and Germany > 50k
df.loc[(df["Italy"] < 30000.0) & (df["Germany"] > 50000.0)]
Out[23]:
In [25]:
import json
with open("data/europe-electricity/europe-pop-millions-2019.json") as infile:
pop = json.load(infile)
In [26]:
pop
Out[26]:
In [27]:
# Convert original dataframe to one that has per capita
# consumption in each column
# Since this modifies the data, let's make sure we start
# with the original (total MW) data by reloading.
df = pd.read_csv(
"data/europe-electricity/europe-electricity-2017-to-2019.csv",
parse_dates=True,
index_col="date" # refers to name in header row
)
for c in ["Italy","Germany","France","United Kingdom"]:
df[c] /= pop[c]
# This shows dataframes are MUTABLE
In [46]:
# Now df contains figures in units of MW/Mperson
# i.e. W per capita
df
Out[46]:
In [ ]:
# Aside: It's very common to use dataframe mutations to "fix"
# columns. E.g. if a column contains temperature strings like
# "43.1C" you could use
df["temperature"] = np.array([ x[:-1] for x in df["temperature"]]).astype("float")
# to replace it with a column of floats
Plotting¶
In [43]:
# You *can* use pandas series as the x and y arrays for
# a matplotlib plot.
plt.plot(df.index,df["France"])
# Here, df.index is the column of dates, the x axis
# df["France"] is the the y axis
Out[43]:
In [42]:
ax = df.plot(figsize=(15,8)) # uses plt.plot behind the scenes
ax.set_title("Electrical energy consumption 2017-2019",pad=20)
ax.set_ylabel("W per capita")
Out[42]:
In [41]:
ax = df.rolling(7).mean().plot(figsize=(15,8)) # uses plt.plot behind the scenes
ax.set_title("Electrical energy consumption 2017-2019 (7-day moving average)",size=22,pad=20)
ax.set_ylabel("W per capita")
Out[41]:
