Quiz 5 — Reasoning from Data

The Guide

Two things move together. Now what?

This is the quiz that pays for the whole series. Almost every hard question on the science test is some version of it: here is a result, here is what somebody says it means, and do those two things actually match.

Start with the difference between a finding and an explanation. A finding is what was measured. An explanation is a story about why. The measurement can be perfectly accurate and the story completely wrong, and telling them apart is most of the work.

When two things rise together, there are four possibilities

The first causes the second. More fertilizer, more tomatoes.
The second causes the first. People in good health take up swimming, rather than swimming producing good health.
Something else causes both. A bigger town has more coffee shops and more car crashes, and neither one caused the other.
Coincidence. With enough measurements, some pairs will move together for no reason at all.

Data showing that two things move together cannot, by itself, tell you which of the four you are looking at. That is not a technicality. It is the single most common way a true measurement gets turned into a false claim, in advertising, in the news, and on this test.

What makes cause believable. Four things, roughly in order of strength. An experiment where people were assigned to the treatment rather than choosing it. A known mechanism — some account of how the one thing could produce the other. The effect appearing when the cause is added and disappearing when it is removed. And other researchers finding the same thing separately.

Three questions for any claim. Does the evidence reach that far — one town, twenty households, two weeks? Does the claim say what the data says, or something stronger? And is there another explanation that fits the same numbers just as well? A claim that survives all three is worth believing.

Watch the strength words. Proves and shows are stronger than suggests and is consistent with. Wrong answers on this test are very often correct sentences with one word upgraded. Real scientists write suggests far more often than people expect, and that caution is not weakness — it is accuracy.

The missing comparison. When a study reports what happened to one group, ask what happened to everyone else. A treatment that helped 80 percent of patients means nothing until you know how many got better without it.

Worked Examples

Two questions, worked through

Finding

A survey of 200 towns found that towns with more public libraries have higher average adult reading scores.

Example 1. What does this finding show?

It shows that two things go together across those 200 towns. That is all it shows, and it is worth something — the pattern is real and it needs explaining. What it does not show is that libraries raised anyone's reading score.

Say the finding out loud in the plainest words you can: "towns with more of this also have more of that." If the sentence you were given says anything beyond that, the extra part came from somewhere other than the data.

Example 2. Name three explanations that fit this finding equally well.

One: libraries give people more to read, so reading improves. Two: towns where adults already read well vote to fund libraries, so the reading came first and the libraries followed. Three: wealthier towns can afford both libraries and good schools, and the wealth is doing the work.

All three fit the same numbers. To choose between them you would need something the survey does not contain — what happened in towns that opened a new library, compared with similar towns that did not.

The Quiz

Ten questions

Answer all ten, then press the button at the bottom. Nothing is timed.

Passage A — questions 1 to 3

A city health department reviewed medical records and found that residents who bought a summer pass to the public pool reported fewer doctor visits for lower back pain than residents who did not. In a press release, the department suggested that swimming prevents back pain and urged residents to buy passes.

A researcher at the community college responded that people who are already in poor health, including people with existing back problems, are much less likely to buy a pool pass in the first place.

1.What do the medical records actually show?

ASwimming prevents back pain.
BResidents who bought a pool pass reported fewer back-pain visits than residents who did not.
CBack pain prevents people from swimming.
DSwimming has no effect on back pain.

2.What is the researcher's objection?

AThe health department counted the doctor visits incorrectly.
BSwimming causes back pain in some people.
CGood health may be the reason people swim, rather than the result of swimming.
DThe pool should be free for all residents.

3.Which investigation would best test the health department's claim?

ARandomly assign volunteers either to a swimming program or to no program, then compare back pain in the two groups after several months.
BInterview pool users about how much they enjoy swimming.
CCount how many residents buy a pool pass next summer.
DAsk local doctors whether they believe swimming helps.

Data B — questions 4 and 5

Town	A	B	C	D	E	F	G	H
Population	8,000	14,000	21,000	29,000	36,000	44,000	52,000	61,000

Crash counts and coffee shop counts for one year, with each town's population.

4.What does the scatterplot show?

ACoffee shops cause car crashes.
BCar crashes cause coffee shops to open.
CThere is no pattern in the data.
DTowns with more coffee shops tend to have more car crashes.

5.Taking the table into account, what best explains the pattern?

ADrinking coffee makes drivers less safe.
BLarger towns have more of both, so population explains the pattern without either one causing the other.
CCar crashes bring extra customers to nearby coffee shops.
DThe two measurements were taken in different years.

Passage C — questions 6 and 7

A company tested a new tap-water filter in 20 households in a single town. Over two weeks, lead levels in the filtered water were on average 60 percent lower than in the unfiltered water from the same taps.

The company then ran this advertisement:

"Proven to cut the lead in your home's water by 60 percent."

6.What is the biggest problem with the advertisement's claim?

ATwo weeks is far too long a period to test a filter.
BThe company should have measured copper as well as lead.
CTwenty households in one town may not represent homes elsewhere, so the result may not hold in a reader's home.
DAn average is never a useful thing to report.

7.Which single word in the advertisement reaches furthest beyond the evidence?

AProven
BCut
CLead
DWater

Passage D — questions 8 to 10

In 1854 a severe outbreak of cholera struck the Golden Square district of London. Most physicians at the time believed cholera spread through foul air. A doctor named John Snow believed it spread through water, and he set out to find evidence.

He mapped every death in the district and found them clustered tightly around a single public water pump on Broad Street. He also looked at two places inside the outbreak area that had almost no deaths: a workhouse with its own private well, and a brewery whose workers drank beer rather than pump water.

Snow persuaded the parish officials to remove the handle from the pump, and deaths in the district fell. Snow himself later wrote that the outbreak had already been declining before the handle came off, because a great many residents had fled the district, and that he could not say from that alone whether the well was still dangerous.

8.Which piece of evidence most strongly supports Snow's idea that the pump water carried the disease?

AMost physicians at the time believed cholera spread through foul air.
BDeaths fell after the pump handle was removed.
CThe workhouse with its own well and the brewery whose workers drank beer had almost no deaths, though both stood inside the outbreak area.
DThe outbreak was unusually severe.

9.Why did Snow report that the outbreak was already declining before the handle was removed?

ATo show that the pump was not the cause after all.
BTo explain a mistake he had made in drawing the map.
CTo argue that the officials had acted too slowly.
DTo make clear that the fall in deaths afterward could not by itself prove the pump was responsible.

10.The workhouse and the brewery were useful to Snow for the same reason a control group is useful in an experiment. Why?

ABecause they were larger than the surrounding buildings.
BBecause they sat inside the same outbreak but did not drink the pump water, so they showed what happened without the suspected cause.
CBecause they kept careful written records of their own.
DBecause beer is safer to drink than water.

Send this line to your teacher

The line records which questions you missed and which answer you chose. That is more useful to your teacher than the score, because it shows where a question went wrong. If a question felt unclear even though you got it right, add its number with a question mark — for example 5? — before you send it.

Score ______ / 10 Missed — write the question number and the letter you chose:
______________________________________________________________

The Key

Answers, and the trap in each one

1. B. The records counted pool passes and doctor visits. Two things going together is the whole of what was measured. A is the health department's explanation, not the finding, and the question asked what the records show. D is just as unsupported in the opposite direction — the data cannot rule out an effect either.

2. C. This is possibility two from the Guide: the arrow may run the other way. Healthy people take up swimming; the swimming did not make them healthy. A imagines the researcher is accusing someone of miscounting. He is not disputing the numbers at all — he accepts them and disputes the story attached to them. That distinction is worth holding on to.

3. A. Random assignment is what breaks the researcher's objection. If a coin decides who swims, then healthy people are spread evenly across both groups and cannot be the reason for any difference. B, C, and D collect opinions or more of the same kind of numbers. None of them separates the two explanations, and gathering more data of a kind that cannot settle the question is a way of looking busy.

4. D. The points rise from left to right. Say what you see, in the plainest words, and stop there. A and B are explanations wearing the clothes of an observation. On this test, the answer that only describes the pattern is very often the right one, and it usually looks disappointingly modest next to the alternatives.

5. B. Look at the populations. They rise in exactly the same order as everything else. A town of 61,000 has more cafés and more crashes than a town of 8,000 because it has more of nearly everything. A is possibility one and sounds almost sensible, which is why the table is there. When a third measurement rises alongside both of the others, it usually deserves to be the first suspect. This kind of hidden third factor is the most common reason two unrelated things appear connected.

6. C. Twenty households in one town share a water supply, pipes of a similar age, and one local water treatment plant. A reader's home may have none of those things. The measurement may be accurate and still not transfer. A inverts a real concern — two weeks is short, not long. D throws out averages altogether, which is too much: the average was a reasonable way to report the result. It is the leap from these homes to your home that fails.

7. A — proven. One small trial in one town does not prove anything. "In our test, filtered water contained 60 percent less lead on average" would have been an honest sentence about the same result. The other three words are accurate. This is the shape to remember: a claim can be false because of one upgraded word while every other word in it is true.

8. C. The workhouse and the brewery are the strongest evidence because they were inside the outbreak and yet were spared, and the thing that set them apart was the water they drank. That is a comparison, and comparison is what turns a pattern into a case. B is the famous part of the story, and it is the weakest evidence in the passage — as Snow said himself. The most memorable fact and the strongest fact are not always the same fact.

9. D. Deaths were already falling because people had left. Something that was going to happen anyway cannot be credited to an action taken partway through. Snow saw that and wrote it down. A misreads honesty as retreat. Snow still believed the water was the source, and he was right. He simply refused to claim more support than one piece of his evidence could carry.

10. B. Same neighborhood, same air, same outbreak — different water. Everything was held constant except the one thing under suspicion, which is exactly what a control group does inside an experiment. Snow could not assign anyone to drink from the pump. He found groups that had already been separated for him and used them as a comparison, which is how investigation works when an experiment is impossible: with outbreaks, with earthquakes, with the deep past.

Your Score

What the number means

8 to 10	Solid. Check topic 5 on your map and go on to Quiz 6.
6 or 7	Close. Read the whole key, then take this again in a few days before you check the box.
5 or fewer	Worth real time, more than any other quiz in Part I. This is the reasoning the test rewards most heavily, and it is the reasoning that keeps working long after the test is behind you.

Misses on 1 or 4 mean an explanation was mistaken for an observation — practice saying findings in the plainest possible words. Misses on 2 or 5 mean an alternative explanation slipped past, so run the four possibilities each time. Misses on 6 or 7 are about scope and strength words, and both are catchable by rereading the claim one word at a time. Misses on 8, 9, or 10 mean the comparison group did not stand out as the strongest thing in the passage, and it usually is.

The plate for this quiz

The gallery image is Snow's own map of the 1854 outbreak. Each death is a short black bar drawn beside the house where it occurred, and the bars stack into a dark mass around the Broad Street pump. It is one of the most reproduced diagrams in the history of science, and it is usually retold badly: the doctor removes the handle, the deaths stop, the story ends.

The real version is better. Snow reported that the outbreak was already fading, said the drop afterward proved nothing on its own, and rested his case on the workhouse, the brewery, and years of further work on which company supplied which household's water. He did not live to see the argument won. The map is public domain and available at high resolution through several archives.