Science is better with Bayes.

My goal here is to explain how to approach science in every-day life through Bayes' theorem.  I promise it'll be fun.

(Made you look.)

One of the (several) problems with falsificationism (Popper's approach to science I laid out in a previous post) is that it doesn't give a useful account of degrees of certainty.  It encourages this idea that either you know a thing is true, you know it's false, or you're completely in the dark about it and can't make any rational decisions based on it.  In reality, if you're 90% certain about something, you should mostly act as though it's true, but give yourself a little wiggle room in case you turn out to be wrong.  We're almost never 100% certain about things, and that's perfectly fine.  We can still do good science and make rational decisions while working with probabilities, especially if we take a little advice from Bayes.

Remember back to when you were a little kid and you were just starting to doubt the existence of the tooth fairy.  It was a difficult question, because if there's no tooth fairy then your parents are liars.  And that's bad.  But you can't shake the feeling that this tooth fairy business doesn't quite match up with your understanding of the way the world works.  So you say to the world, "Stand back.  I'm going to try science."

You start with a question.  You want to know how it is that money appears under your pillow whenever you lose a tooth.  The theory you want to test is that the tooth fairy flies into your room, carefully reaches under your pillow, takes the tooth, and leaves money.  So your theory seems to predict that you ought to be able to catch her on camera.  Your test consists of leaving your freshly liberated tooth under your pillow, pointing your webcam at your bed, setting it to record all night, going to sleep, and watching the video the next day.  Your hypothesis is that there will be a fairy somewhere in the video.  Good old capital "S", capital "M" Scientific Method, as usual.

Suppose you get exactly the result you hypothesized.  Sure enough, three hours into the video you see a light from outside, the window opens, and a small shiny woman with wings floats in.  She reaches under your pillow for the tooth, replaces it with money, and then leaves.  The intuitive response to this result is to become wholeheartedly certain that the tooth fairy exists.  Popper's falsificationism tells us it's going to take a whole lot more tests before we should be really certain that the tooth fairy exists, because even though this is a legitimately scientific theory, confirmation isn't nearly as strong as falsification.  But it doesn't tell us how sure we *should* be.  Just that we shouldn't be completely sure.  Should we be 20% sure?  50% sure?  90% sure?

How we should act when we're 20% sure vs. 90% sure is very different indeed.  If you're only 20% sure the tooth fairy exists even though your parents insist she does, you should probably have an important talk with them about honesty, whether they themselves actually believe in her, and maybe skepticism if they really do.  If you're 90% sure, you might want to set up your computer to sound an alarm when it registers a certain amount of light so you can wake up and ask her to let you visit fairyland.  So how do you know how much certainty is rational?

Have no fear.  Bayes is here.

First, you're going to have to guesstimate your certainty about a few things.  You should definitely do this before you even run the experiment.  If you want to be really hardcore about it, convince other people, and generally run things with the rigor of a professional scientist, guesstimating isn't quite going to do the trick.  But every-day science like this is necessarily messy, and that doesn't mean you shouldn't do it.  It's perfectly fine and useful to be somewhere in the ballpark of correct.  So here are the numbers you need.

• How certain are you that you really will catch her on film if she exists?  You reason that she probably is visible.  Otherwise she wouldn't have to come at night.  And fairies are supposed to glow or something, right?  You can't be invisible if you glow.  On the other hand, you don't really know how magic things interact with the rest of the world, so maybe she's like a vampire and simply can't be caught on film.  Let's call it 80% certainty, or 0.8.
• How certain are you about the existence of the tooth fairy in the first place, before the experiment?  Since you were definitely becoming a tooth fairy doubter, but still thought it was pretty up-in-the-air, you figure you were about 40% certain that there's a tooth fairy.  You can express that as the decimal 0.4.
• How likely is it that you'll see a fairy on the recording even if the tooth fairy doesn't actually exist?  It seems really unlikely.  But you can imagine other things that would cause this.  You mentioned to your older brother earlier that you were doubting the tooth fairy, so maybe he'll find out about your plan and play a prank with his film school buddies.  Or maybe there will be some fluke that causes damage to the file so it looks like there's a glowy person shaped thing in the recording that really is only in the recording.  So it's imaginable, but unlikely.  Let's say 5% sure something like that could happen.  0.05.
• Finally, how likely is it that there's no tooth fairy?  Well this one's easy.  You already decided you're 40% sure there's a tooth fairy, so you must be 60% sure there isn't one.  0.6.

Bayes' theorem is all about finding out how much the evidence should change your beliefs, and whether it should change them at all.  It weighs all those factors we just estimated against each other and comes up with a degree of certainty that actually makes sense when you put them together.  Human brains are really bad at weighing probabilities rationally.  They just aren't built to do it.  But that's ok, because we have powerful statistical tools like this to help us out--provided we know how to use them.

If you want to know the nitty gritties of what's really going on inside Bayes theorem, check out Eliezer Yudkowsky's "excruciatingly gentle introduction to Bayes' theorem".  He's already got that covered (beautifully).  I just want to show you how it ends up working in real life.  So let's run the numbers.

  

We're looking for the probability that there's a tooth fairy after accounting for having (apparently) caught her on camera.  That's P(A|B), read "probability of A given B", where A is "there's a tooth fairy" and B is "she's in the recording", so "probability that there's a tooth fairy given that she's in the recording". 

In the numerator, we start with P(B|A), which is how likely it is that we really will see her on camera if she exists--probability "she's in the recording" given "there's a tooth fairy".  And that's 0.8.  Next, we multiply that by how sure we were that there's a tooth fairy before we caught her on film, simply probability "there's a tooth fairy".  And that's 0.4, for a total of 0.32 on top.

For the denominator, we start with a value we already have.  "P(B|A) P(A)" is what we just worked out to be 0.32.  So that's on one side of the addition sign.  Next, we want the probability that we'd see the tooth fairy in the recording even if the tooth fairy didn't actually exist.  The squiggly ~ symbol means "not"; P(B|~A) is probability "she's in the recording" given "she doesn't exist".  And that's 0.05.  Then we multiply that by P(~A), the probability that there isn't a tooth fairy, which is 0.6, for a total of 0.03 on the other side of the addition sign.  Add that up, and it's 0.35 on the bottom.

Finally, divide the top by the bottom: 0.32 divided by 0.35 equals 0.914ish.  What does that mean?  It means that if you started out thinking it's a bit less likely that there's a tooth fairy then that there isn't one, and then you caught her on camera, you should change your beliefs so that you're just a little over 90% certain that there's a tooth fairy.

In other words, you're growing up into an excellent rationalist who just made a groundbreaking discovery.  Go show the world your tooth fairy video, and see about having tea with the faeries.

Everything's better with science, and science is better with Bayes.

**************************************************************************

Problem Set: No, really, run the numbers.

1) Your power is out. It's storming. Use Bayes' theorem to decide how sure you are that a line is down.

2) A person you're attracted to smiles at you. Are they into you too?

3) (For this one, intuit the answer first. Make your best guess before applying the theorem, and WRITE IT DOWN. It's ok if you're way off. Just about all of us are. That's the point. Human brains aren't built for this kind of problem. I just don't want you falling prey to hindsight bias.) 1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?