Science Matters

Figures don’t lie, but they can mislead

Boyce Rensberger

(10/2024) Numbers don't always mean what they seem to mean. Consider these two examples:

• The average salary of the 145 women who work at Widget Inc. is $60,000. Widget’s 100 men are paid salaries averaging nearly 60 percent higher--$95,000. Is this an obvious case of sex discrimination?
• Mercy Hospital has a 50 percent higher death rate among its surgery patients than does General Hospital in the same town. If you need an operation, you'd be safer at General, right?

Not necessarily. Not in either case.

Impossible as it may seem, Widget actually pays women significantly better in every job category than it pays men. And your chances of dying during surgery would be lower in Mercy Hospital despite its statistically higher death rate.

This is because, in some cases, numerical data can work in a perverse way that experts say often is ignored in public debates.

"These problems come up again and again in public debate, and because most people have a fairly weak grasp of statistics, the debate just gets muddier," says David S. Moore, a statistician at Purdue University.

Statisticians are well acquainted with situations in which the same raw data can be represented in different ways, both of them technically honest and both accurate and yet point to opposite conclusions. It’s called Simpson’s paradox, named not for Homer, the cartoon character, but for the late Edward Simpson, a British statistician.

Let’s take a closer look at whether Widget Inc. has a sexist pay scale. The company could point out that, in all three of its categories of employment, it pays women higher salaries.

For example, women in Widget management average $160,000 while the men average $150,000. In the engineering department, women get $110,000 to the men's $100,000. And in the clerical ranks, women average $35,000, compared with $25,000 for men. Widget’s salary practices can hardly be sexist if it pays women better at every level. How can this be?

The thing that misleads people is what's often called a lurking variable, which is some other feature underlying the data that you may not realize is there. The trick is to pull apart the numbers and look at smaller categories that might be more relevant.

The lurking variable in the Widget case is that women are not evenly distributed through the ranks. Hardly any women are in management, so the salary advantage of those who are there does little to boost women’s average for the company as a whole. By contrast, women dominate the clerical staff, and even though they are paid better than men in the same department, they are paid so much less than the male-dominated management staff that the company's overall average for women drops.

Clearly, a closer analysis of the data indicates that Widget’s situation is more complex than either side might think.

Here's a closer look at this same kind of paradox in the hospital comparison. Stick with me here, this involves arithmetic. Statistics can be that way.

In one year, Mercy Hospital has 2,100 surgery patients, of whom 2,037 leave the hospital alive and 63 die. Its overall death rate, therefore, is 3 percent. General Hospital has 800 patients, of whom 784 live and 16 die—a death rate of 2 percent. Thus, Mercy’s death rate is 50 percent higher than General’s.

On the face of it, General would seem the safer bet. In fact, the safer choice actually is Mercy. That becomes obvious only when you break the data into smaller categories.

At Mercy, 600 patients were in good condition. Of these, six died, creating a 1 percent death rate for this category. General also had 600 patients in good condition but eight of them died, giving a death rate of 1.3 percent—higher than at Mercy.

What about the patients in poor condition? Mercy had 1,500 of these, and 57 died, a death rate of 3.8 percent. At General, only 200 patients were in poor condition, and eight of them died, a death rate of 4 percent. Again, General had the higher death rate.

In other words, no matter whether you are in good condition or poor, you should go to Mercy Hospital, the one with the higher overall death rate.

In the hospital case, the lurking variable is the difference in the type of patients. The great majority of Mercy's patients arrive in poor condition, and many are simply too ill to survive even the best medical care. That raises its average death rate for all patients by more than it is lowered by Mercy’s good record with patients in good condition.

General’s patients are in better shape to start with, which means that, although it does a poorer job with them than Mercy would, the proportionately high number of those patients significantly boosts the overall average.

Simpson’s paradox is not confined to hypothetical comparisons like these. A famous real-life situation emerged in the 1970s when the University of California at Berkeley was accused of bias against women seeking admissions to its graduate departments. About 44 percent of male applicants were admitted, but only 35 percent of female applicants.

Those numbers were so different from what would be expected in an unbiased selection process that critics calculated the odds of the difference emerging by chance as "vanishingly small."

It turned out there was a lurking variable.

The women tended to apply to departments where competition is much stiffer for both sexes. Men, on the other hand, were more likely to apply to departments with fewer applicants and easier acceptance.

When the numbers were examined for each department seperately, admission rates differed only slightly and, in fact, about as many departments favored women as favored men.

Three Berkeley statisticians who examined the case pronounced it "a clear but misleading pattern of bias against female applicants." Their analysis, after searching out lurking variables, led to a more profound conclusion.

"The bias in the aggregated data," they wrote in a report, "stems not from any pattern of discrimination on the part of admissions committees, which seem quite fair on the whole, but apparently from prior screening at earlier levels of the educational system."

Women, they said, are shunted by their socialization and education toward academic fields that are more crowded and less well-funded. But since so many people apply for them, they are harder to get into. These include such "soft" teaching fields as English or history.

Men, by contrast, are more likely to aim for "harder" fields such as science or engineering, where fewer people are competing for acceptance.

So, how can you guard against being misled?

Statisticians recommend a simple starting point: Don't be too quick to accept any interpretation attached to a set of figures, especially if the numbers lump several categories of the thing being studied. Try to obtain the numbers for each of the categories and see whether the interpretation holds true for each.

Read past editions of Science Matters