Simulation and Data with pandas: Quick Revision

A brief note on Simulation and Data with pandas. Read it before a test. Try each question at the end before you open its answer.

Random numbers

• rng = np.random.default_rng(seed) makes a generator. The same seed gives the same numbers, so a result can be checked.
• rng.random(n), rng.uniform(a, b, n), rng.normal(mean, sd, n), rng.choice(items) and rng.shuffle(items) cover the usual needs.
• rng.integers(1, 7, size=n) throws a die $n$ times. The stop, 7, is not included.
• The law of large numbers: with more trials, the share of an outcome comes closer to its probability.

Monte Carlo: estimating by chance

• Estimate a probability as the share of random trials in which the event happens.
• $\pi$ is about 4 times the share of random points of a unit square that land inside the quarter circle.
• The error falls only as $1/\sqrt{N}$. One hundred times more trials give ten times more accuracy.
• Repeat the whole experiment to see how large the error is.

Random walks and decay

• A random walk is the running total of random steps: np.cumsum(steps). For many walkers, use a row for each and cumsum(axis=1).
• With steps of $+1$ and $-1$, the mean position stays 0. After $n$ steps the mean squared distance is $n$, so the typical distance is $\sqrt{n}$.
• The final positions of many walkers form a bell curve. This is diffusion.
• Each atom decays at a random moment. rng.binomial(n, p) gives the number of decays among $n$ atoms in one step.
• A count of $N$ scatters by about $\sqrt{N}$, so a large sample looks smoother than a small one.

How sure is a measurement?

• The standard error of the mean of $n$ readings, with the sample deviation std(ddof=1):
  • $\text{SE} = \dfrac{s}{\sqrt{n}}$
• Four times as many readings halve the uncertainty.
• The bootstrap resamples the data with replacement many times. The spread of the new means is the uncertainty.
• Averaging does not remove a systematic error. Calibrate the instrument instead.
• Report a result with its uncertainty, such as $2.35 \pm 0.04$ s.

Tables with pandas

• A Series is an array with labels. A DataFrame is a table of Series that share an index.
• First look: df.shape, df.columns, df.dtypes, df.head() and df.describe().
• df["a"] is a column and df[["a", "b"]] a smaller table. df.iloc[2] is a row by place, and df.loc picks by label. A loc range includes both ends.
• pd.read_csv reads a file. df.isna().sum() counts the missing values; dropna and fillna deal with them. Say what you did.
• df[df["a"] > 2] picks rows. Join conditions with & and |, each in brackets. sort_values, isin and query help.
• A new column is a formula on whole columns: df["speed"] = df["dist"] / df["time"].
• df.groupby("team")["speed"].mean() splits the rows, summarises each group and combines the results. pd.merge(a, b, on="name") joins two tables.
• df.plot.scatter(x="a", y="b") draws from a table; import matplotlib.pyplot and call plt.show().
• A power law $y = c\,x^p$ is a straight line on a log-log graph, and its slope is $p$.

A worked example

A school records six runners. Which team ran faster on average, and who ran 400 m for the red team?

1. A new column. runs["distance_m"] / runs["time_s"] divides place by place, so every runner gets a speed. Anu's is $400 / 62.0 = 6.45$ m/s.
2. Groups. groupby("team") splits the six rows into two teams. ["speed_m_s"].mean() gives one mean for each: blue 5.97 m/s, red 6.05 m/s.
3. Picking rows. Each condition is in its own brackets and joined with &. Only Anu and Esha are red runners over 400 m.

So the red team was a little faster, by about 0.08 m/s.

Practice with new numbers

A program to try

How likely is a total of 15 or more with three dice? Of the 216 outcomes, 20 give 15 or more. The program throws the dice more and more times and compares the share with the exact answer.

Mistakes to avoid

• No seed when a result must be repeated. Each run then gives new numbers. Write default_rng(42) or any fixed seed.
• rng.integers(1, 6) for a die. The stop is not included, so 6 never comes. Write rng.integers(1, 7).
• Trusting a small simulation. With 100 trials the error can be several hundredths.
• std() without ddof=1 for a sample. It makes the uncertainty a little too small.
• A column that does not exist. df["mass"] stops with KeyError: 'mass'. Look at df.columns for the exact name.
• and between columns. df[df["a"] > 2 and df["b"] < 5] stops with ValueError: The truth value of a Series is ambiguous. Use & with brackets.
• Dropping rows without a word. Missing values hide information; say how many rows you removed.

Check yourself

1. Kiran tosses a coin 1,000 times with rng.integers(0, 2, size=1000). Why does the number of 1s change each time the program runs, and how can she make it the same every time?

2. A Monte Carlo estimate has an error of about 0.03 after 2,500 trials. About how many trials would bring the error down to 0.01?

3. Each step of a walker is 1 m forwards or 1 m backwards, by chance. After 400 steps, what is the mean position of many walkers, and what is a typical distance from the start?

4. In one time step, about 2,500 atoms of a sample decay. About how much does this count change from one run of the simulation to the next?

5. A DataFrame has the index 0, 1, 2, and so on. How many rows do df.loc[2:4] and df.iloc[2:4] give?

6. Say in words what df.groupby("house")["points"].sum() gives.