Data Analysis with pandas

Name: Data Analysis with pandas
Author: StemPy Lab

Real data is a table with named columns, different types and missing values, and pandas is the library for it. In this course you build tables from dictionaries and read them from CSV text, look at them with shape, dtypes and describe, and find and handle missing values. You choose rows with conditions, labels and places, add new columns with formulas, summarise groups with groupby, and join tables with merge. You draw scatter plots and bar charts straight from a table, and use a log-log graph to find a power law. A project analyses the planets of the Sun, finds Kepler's third law from the data, and predicts a period that was left out of the fit. The practice runs your code and tells you which results are wrong.

5 lessons, about 2 h 55 min5 simulations10 problems10 practice questionsPython in every lesson

Created by StemPy Lab

Course outline (5 lessons)

Lesson 1about 40 min
Series and DataFrames
pandas is the library for tables of data. A Series is a NumPy array with a label for each value, so it does arithmetic on all values, looks up a value by its label, and has methods such as mean and idxmax, and a comparison picks the labels that pass. A DataFrame is a table of columns of the same length, each with a name and a type, made for example from a dictionary of lists. df.shape gives the rows and columns, df.columns the names, df.dtypes the types, df[name] one column as a Series, df[list of names] a smaller table, df.head the first rows, df.iloc a row by its place, and df.describe the count, mean, spread and quartiles of every numerical column. Columns of text have the type str.
1 simulation2 practice questions4 Python programs2 problems1 Python lab1 question set
Read the lesson
Lesson 2about 35 min
Reading and selecting data
pd.read_csv reads a table from a file or, with io.StringIO, from a text, and index_col makes a column the index. isna and sum count the missing values, which are NaN, the mean ignores them, dropna drops the rows with a missing value, and fillna fills them, for example with the mean. Rows are chosen with a boolean condition inside square brackets, combined with & and | in brackets, with isin for a list of values, with sort_values for an order, and with query for a condition written as text. loc picks by label, and a range of labels includes both ends, iloc picks by place. Choosing and cleaning take most of the time in a real analysis.
1 simulation2 practice questions4 Python programs2 problems1 Python lab
Read the lesson
Lesson 3about 35 min
New columns, groups and joins
A new column is made by a formula on whole columns, such as the period squared divided by the distance cubed for Kepler's third law, whose value is about 1 for every planet when the period is in years and the distance in astronomical units, and NumPy functions work on columns. groupby splits the rows by the values of a column, summarises each group with mean, sum, max or agg, and combines the results in a table, and value_counts counts the rows of each value. pd.merge joins two tables on a common column: the default keeps only the names that are in both tables, and how equal outer keeps all of them with NaN where a table has nothing. These three steps, derive, group and join, are the core of data analysis.
1 simulation2 practice questions3 Python programs2 problems2 Python labs
Read the lesson
Lesson 4about 35 min
Graphs from tables
pandas draws graphs straight from a table with p.plot, p.plot.scatter, p.plot.bar and p.plot.hist, using matplotlib, so the code should import matplotlib.pyplot and call plt.show. The plot methods give the axes, and the axes methods from chapter 2 set the title, the labels, the scales and the grid. A scatter plot of the periods of the planets against their distances is a curve, and on a log-log graph it is a straight line, because a power law T equal to c times a to the power p has the logarithm log T equal to log c plus p times log a, and the slope of the line through the logarithms is the power: 1.5, as Kepler's third law says. A bar chart of a table or of a groupby result compares groups.
1 simulation2 practice questions3 Python programs2 problems1 Python lab
Read the lesson
Lesson 5about 30 min
Project: Kepler's third law from a table
A project that uses the whole chapter: a table of eight planets and Pluto, with the period of Neptune missing, is read, checked for missing values, cleaned by dropping the incomplete row, extended with a column for the ratio of the period squared to the distance cubed, which is about 1.0 with a spread of 0.016, and analysed with a straight line through the logarithms, whose slope is 1.5. The fitted law predicts the period of Neptune from its distance as 165.0 years, which is close to the true 164.8, and a graph on log-log scales shows all the points on a straight line. A good analysis reads, checks, cleans, derives, summarises, predicts and draws, and reports what it did at every step.
1 simulation2 practice questions1 Python program2 problems1 Python lab1 note
Read the lesson

Test Your Learning

Data Analysis with pandas: Predict the Output

10 questions · 15 min

Not started

Your teacher

StemPy Lab

← Simulation and Data

← Simulation and RandomnessNext →

More onPython for Science and Maths › Data Analysis with pandas

Discussion (0)

Loading…

Data Analysis with pandas

Course outline (5 lessons)

Series and DataFrames

Reading and selecting data

New columns, groups and joins

Graphs from tables

Project: Kepler's third law from a table

Test Your Learning

Your teacher

Discussion (0)