Project 2: a secret-message cracker

What you will be able to do

After this lesson you can

• break a Caesar code with letter counts,
• explain why a short secret can fool the rule,
• try all 26 shifts and score each one, and
• combine a dictionary, max, and % 26 in one tool.

The idea

In the course on text you hid a message with a Caesar code. Can you read a secret without knowing the shift? Yes, by counting letters. In English the letter e is the most common. So the most common letter in the secret probably stands for e.

If the commonest letter of the secret is l, and l is 7 places after e, then the shift is 7.

Step 1: what to expect

Before you run any code, think. We have a long secret of about 130 letters. If the rule is right, then:

• the most common letter is l, and it appears more than any other letter,
• the guessed shift is (place of l - place of e) % 26, which is $(11 - 4) \bmod 26 = 7$,
• moving every letter back by 7 gives an English message.

Step 2: the program

Predict first, then run it.

• count_letters is the counting pattern: a dictionary, with the letter as the key. It ignores spaces and marks with isalpha().
• shift_text is the function of the text course. A negative shift goes back.
• max(counts, key=counts.get) picks the letter with the biggest count.
• (ord(commonest) - ord("e")) % 26 is the shift. The % 26 keeps the answer from 0 to 25. Without it, a letter before e in the alphabet (such as a) would give a negative number, and % 26 turns it into the right shift.
• The last line decodes the message by moving every letter back.

The letter l appears 28 times in this secret, and the program finds the shift 7. The message is readable.

When the guess is wrong

The rule needs a long text. Look at a short one.

This is an important lesson in the use of statistics: a rule that works on many items can fail on few. Always ask how much data you have.

In a short message the bars wobble, and another letter may be tallest. Then trying every shift, and checking for real words, is safer.

Try every shift

When the text is short, we can try every shift (there are only 26) and look at the answers. But a computer cannot "look". It needs a score: how English does the answer look? A simple score: count how many of the words are common English words.

The method has a weakness: it needs a list of common words, and a message with no common words will fool it. A bigger set of words helps. But the method does not need the text to be long.

Which method to use?

A good tool can use both: first the quick rule, and if the answer scores badly, try all the shifts.

Extend it

Pick one or two, and do them without looking at a solution. Write the expected answers first.

1. The first answer, checked. After the commonest-letter guess, score the answer. If the score is low, try all the shifts.
2. A bigger word list. Add twenty more common words. Does it help?
3. Capital letters. Make the tools work for text with capitals and marks. What must change?
4. A different cipher. Try a Vigenere-like cipher with a keyword. Why is it harder to break?
5. Letter frequencies as a table. Show the 5 most common letters with their counts.

Common mistakes

• Using the rule on a short text. It fails often.
• Forgetting % 26. A shift outside 0 to 25.
• Counting spaces and marks as letters.
• A tie for the commonest letter. max picks the first. Is that right?
• Decoding in the wrong direction. To decode, use -shift.

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

1. Warm-up. Open the lab Guess the shift. You write the function, and the checker tries it on long texts.
2. Build. Open the lab Crack the code by trying every shift and the problem The shift from the letter.
3. Stretch. Open the problem Which letter is the most common? and decide what to do for a tie.

Remember these

• In a long English text, e is the most common letter.
• The shift is the place of the commonest letter, minus the place of e, with % 26.
• A short text may break the rule. Then try all 26 shifts and score them.
• A rule that works on a lot of data can fail on a little.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. The most common letter of a long secret is x. What shift does the rule guess?

2. Why can a short secret fool the rule?

3. Why do we use a set for the common words?