Call:
lm(formula = y ~ random_data)
Residuals:
Min 1Q Median 3Q Max
-1.73076 -0.49402 -0.01661 0.47196 1.53067
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.0196589 0.0928258 0.212 0.83282
random_datax1 0.0149596 0.0879601 0.170 0.86539
random_datax2 -0.0577474 0.0903559 -0.639 0.52460
random_datax3 -0.0274725 0.0846988 -0.324 0.74653
random_datax4 0.0932439 0.0782035 1.192 0.23670
random_datax5 -0.1263844 0.1051801 -1.202 0.23311
random_datax6 -0.0334851 0.0949183 -0.353 0.72519
random_datax7 -0.0429656 0.0908107 -0.473 0.63742
random_datax8 0.0069873 0.0930217 0.075 0.94031
random_datax9 -0.1260338 0.0917890 -1.373 0.17361
random_datax10 0.0297859 0.0740732 0.402 0.68869
random_datax11 0.1863070 0.1003028 1.857 0.06697 .
random_datax12 -0.2003405 0.1053486 -1.902 0.06086 .
random_datax13 -0.2916135 0.0991653 -2.941 0.00429 **
random_datax14 0.1702796 0.0819680 2.077 0.04101 *
random_datax15 0.0324912 0.0948048 0.343 0.73272
random_datax16 -0.0938624 0.0914770 -1.026 0.30799
random_datax17 0.1187972 0.0824957 1.440 0.15381
random_datax18 -0.0239293 0.0957729 -0.250 0.80335
random_datax19 -0.0003125 0.0907422 -0.003 0.99726
random_datax20 -0.0332536 0.0851853 -0.390 0.69731
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.809 on 79 degrees of freedom
Multiple R-squared: 0.2875, Adjusted R-squared: 0.1071
F-statistic: 1.594 on 20 and 79 DF, p-value: 0.075313 - Critical data literacy
Data literacy
Introduction
We’re going to end our last new material/lecture (!!!) for the semester talking about some larger issues within data science. I hope that through this semester you have come to feel some confidence in your ability to tackle basic data-analytics tasks using a coding approach (with R, but also with whatever software you end up using in the future). While there is an infinite amount left for you to learn about coding for research, that is because there is an infinite amount for me to learn, as well. I am constantly learning (and relearning) new tricks and methods. With a basic understanding of good coding practice, control flow, and data simulation and wrangling, I honestly think you can tackle any problems that come your way. Therefore, today we’re going to frame the class and discussion a little bit differently, because I want to focus on what I am calling data literacy. While I am going to use coding examples throughout to illustrate the discussion, today isn’t going to be about learning new skills. It’s going to be about becoming better producers (and consumers) of quantitative information–better scientists!
I think of data literacy as a combination of critical thinking about how data are produced, managed, analyzed, and interpreted and a concern with the ethics surrounding each step of the process. As scientists we are often privileged to not engage directly with the ethics of what we’re doing–sure, we have to write IRBs when we do human-subjects research or IACUCs when we work with animals, but we treat our work as inherently a net positive: producing new knowledge is fundamentally a worthwhile goal. In general, I agree with this statement, but I think when we embrace it uncritically, we end up not considering a host of the aspects of how and why we are doing the work we do, and who it ultimately serves.
I will also give a big caveat to the way I am approaching this topic: data literacy is a really huge topic that deserves a class on its own (and if you’re interested in these general ideas, please come talk to me, read the references given here, and consider taking some courses in the Virginia Tech STS Department). In addition, my hope is that by learning how the sausage is made (how many times have I repeated “there is no right answer?” in this putatively quantitative course so far?) you will have developed some critical abilities already. So I am going to present a few provocative examples at various levels that will hopefully get you thinking a bit more about how we can do data analysis better: more humbly, more accurately to reality, and more sensitively to our work’s impact on the world.
To this end, here are a number of resources I suggest you read to get some insight into the topics I am going to only address really briefly today:
- Books:
- Data Feminism, a book and web project by two data-science researchers in the digital humanities that gives some really good principles for thinking about and with data.
- Calling Bullshit, a website and book developed by two University of Washington professors that is entertaining and gives good, non-technical introductions to many aspects of data literacy.
- Counting, a book by Brandeis professor emerita Deborah Stone that gives a non-technical introduction to the politics of making numbers
- Short reads:
- Beall on open-access journals, an article by an academic librarian on the corrosive effects of “pressure to publish” and open-access journals on intellectual and academic discourse
- The gender gap and the 100-meter dash, a wonderful application of basic critical thinking with data applied to a prestigious academic article.
- “The Ethics of Counting” a lecture on the topic of the ethics and politics of making numbers by Deborah Stone (2018)
- “Inference for a Bernoulli process (a Bayesian view)”, by Lindley and Phillips (1976), is the most technical article I suggest, but it does offer a remarkably good introduction to Bayesian statistics and thinking, as well as pointing out some key epistemological holes in the frequentist models
We’re going to start by surfacing many of the issues that point towards the need for data literacy by discussing a particular issue for psychological and social science in recent years: the replicability crisis. After presenting an overview of the topic, this is going to lead us pretty naturally into our next major topic: the p-value and the larger role of Null-Hypothesis Statistical Testing (also often put in the same bucket as “frequentist statistics”, as opposed to “Bayesian statistics”) in the replicability crisis and mishaps in data analysis in general. Finally, we will zoom out further and think about the politics of counting (credit for this general framework to Deborah Stone 2020): what do numbers actually mean?
The replicability crisis
We will begin with a startling article published more than a decade ago by John Ioannidis (2005): “Why most published research findings are false”. In that article, the author provided statistical evidence based in meta-analytical techniques that the generally accepted practice of using the classic \(p < 0.05\) formulation as evidence of a true effect in a quantitative, scientific study was more often than not leading researchers to declare that there was a significant effect when in fact there was not: a false positive. In the decade that followed, rigorous replication studies verified these concerns (with any number of caveats): for example, in psychology, in general reported effects are larger than those found in replicated studies when any effect is found, and in many studies it is impossible to replicate the original results at all (Collaboration 2015). There have been many hypotheses put forward for these problems, including (but almost certainly not limited to):
-
Publication (positivity) bias: in general it is much more difficult to publish a negative (“nonsignificant”) result than it is to publish a significant one. Because academic success and grant funding is predicated on publication, there is pressure to “find” significance. This is a powerful and problematic force in science, and should be taken extremely seriously.
- Related but different is a bias against replication studies, because they tell us “what we already know” (if they reproduce the results) or they produce negative results. This means that in general false positives may remain undiscovered until they become conventional wisdom, at which point disproving them becomes a “positive” result.
- This publication bias also leads to a bias in the kinds of studies that are conducted: this is a type of selection bias. Imagine you are (like me!) a junior, pre-tenured faculty member, or (like you!) graduate students. Are you going to be willing to conduct research that may be unpublishable? Didn’t think so.
- These pressures combine with a culture of celebrity and newsworthiness to overvalue sensational (but perhaps unlikely) scientific results that can form the basis of TED Talks, little news segments, etc. We need to be especially skeptical of these kinds of results: a key principle we might hold dear is “extraordinary claims require extraordinary evidence” (Bergstrom and West 2021).
- “Science” has grown exponentially as an activity over the last century. The sheer volume of the research enterprise has led to the publication of many more results.
- By definition, a larger number of these results will be false positives, even if the proportion doesn’t change…
- …but the proportion is much higher than we might reasonably expect, which points to other systemic problems.
- Statistical incompetence: We all know that statistics are hard, right? We also know specifically (because I’ve said so in this class!) that we often pick and choose statistical and analytical heuristics (“models”) for convenience, not because they are correct. This leads to all kinds of problems that might result in problems with conclusions, of course.
- Picking the “wrong” statistical test. I hope that this class has given you a taste of an alternative picture of statistics, but often we as scientists learn a set of statistical tests as heuristics: we don’t learn the whys or hows, we just learn to apply an algebraic calculation and the magical statistics machine tells us whether our results are significant. In this context, we instead worry about whether our statistical test is “right”, instead of worrying about “what do the data actually tell us”.
- Inappropriate study designs. We are constrained in resources–time, money, help, samples. We often run studies that don’t really represent an ideal: not enough subjects, samples, treatments, whatever. There are many valid reasons for this, but it leads us to make compromises that then ramify out through our conclusions.
- Epistemological vagueness around distinctions between correlation, inference, and causality: we have already been introducede to the idea that our intuitions around patterns and causation are not great as humans, and this problem is actually made worse by the machinery we have for statistics. In general, once we have data, we can run all sorts of tests–whether statistical or simulations or both–on those data. But none of those tests can establish causality. It turns out that causal modeling is a topic in-and-of-itself, and too few scientists actually consider the (usually implicit) causal models that their statistical models instantiate (McElreath 2020).
- I like the non-technical discussion given in the Calling Bullshit case studies explaining how we build causal models in our head by telling ourselves stories.
- …and here I will get more specific: from the sub-point about “wrong” statistical tests about, let me say what I personally believe to be a major problem causing this (and other!) crises: we have a blind faith in the framework of “Null Hypothesis Statistical Testing” and the associated p-values and confidence intervals.
Regardless of the exact whys (and it is almost certainly a combination of these and other factors), science as an enterprise is starting to grapple with how to adapt to all of these problems, but we’re definitely not there yet. For example, the American Statistical Association published a statement about p-values that essentially told people that, yes, we shouldn’t over-rely on them, but they’re fine when used in moderation (Wasserstein and Lazar 2016). Personally, I find this unsatisfying, but there is certainly awareness of the issue.
How can we avoid falling into the replication trap?
So if all of science if facing a crisis with replicability, does that mean we just give up on science?
…
Obviously not. Let’s remind ourselves that science is not infallible and science is not really a belief system: when we get bumper stickers or whatever that say “Science is real” that’s a little bit of a weird statement. “Science” is, to paraphrase Dr. Sadler’s lecture from last week, a set of techniques for socially confirming our theories. It’s real as a process, not as an outcome.
Science is, at its heart, the process of formulating a testable proposition, testing it, and reporting the results to a broader community, which then starts the cycle again. We accrete a body of theory and knowledge. We don’t “mine facts” that exist “out there”. So science doesn’t depend on belief–it is a process.
More importantly, the process of science itself should make us critical thinkers. That should mean we have curious, open minds that act like Bayesians (see below), rather than rigid thinkers who operate on hard and fast rules we come to believe without thinking about them–this makes us martinets.
Anyway, that is to say that problems with replication and with a process of statistical proof–which is a process of probabilistic proof—are not flaws that undermine our system of making knowledge. Rather, they point to maybe a problem in what we’ve come to accept as sufficient evidence in adding to our body of knowledge.
How can we avoid making it worse? We all operate within a system that isn’t working as it should, so our individual power to make change is limited, but I think that we can incorporate some basic ethics of research data production and consumption:
- Be realistic (humble?) about the meaning of your data. This applies both to the analysis and to the communication of your work.
- I don’t want to hear any more about how a tiny experiment is generalizable. It gives us an intriguing further hypothesis to test!
- Allow multiple interpretations and frameworks for the same phenomena to give overlapping information, from which we are able to gain a richer, more faithful model (this comes from Data Feminism and adapts the philosopher of science Sandra Harding)
- Be critical about the how and why of data (and science) production–sources matter.
- Don’t just listen to me–read better-written and better-explained sources on how to do data better, and put them into action.
Some quick thoughts on (Null) Hypothesis Testing (NHST)
\(H_0: \mu_1 = \mu_2\)
\(H_a: \mu_1 ≠ \mu_2\)
We are all familiar with these kinds of formalized hypotheses. They are taught as part of basic statistics classes (I teach about them in my undergrad sensory class, FST 3024). They set up a dichotomy: \(H_0\) and \(H_a\) are mutually exclusive, and so, we reason, if we can “reject” \(H_0\) we can tentatively proceed with \(H_a\).
There are many, many, many problems with this approach, it turns out, as Bayesian statistical methodologist John Kruschke (2010) points out:
In collecting data, you take care to insulate the data from your intentions. For example, double-blind procedures in clinical trials insulate the data from experimenter intentions. As another example, in field research, the observers construct elaborate ‘duck blinds’ to minimize the impact of the observer on the data. After carefully collecting the data, you then go through the ritual invocation of p < 0.05. Did you know that the computation of the p value depends crucially on the covert intentions of the analyst, or the analyst’s interpretations of the unknowable intentions of the data collector? This is true despite the emphasis by the data collector to make the data unaffected by his/her intentions, as will be shown below. Moreover, for any set of data, an intention can be found for which p is not less than 0.05.
The basic argument Kruschke (2010) is making is that NHST has–in an attempt to provide an accessible heuristic for decision-making for the conscientious, working researcher–elided away the idea of uncertainty. We all know the chant:
… \(p<0.05\), we reject the null hypothesis.
But this means we don’t take into account any information we have on the credibility of our \(H_a\), or anything we know about the quality of the experiment, or anything else we, experts in this research might know.
Furthermore, this approach transforms what is really a judgment about probability–a description of our uncertainty–into a binary, black and white situation: if \(p<0.05\) (or any other value), we reject \(H_0\). This just isn’t good reasoning.
Torture the data until it gives you the p-values you want
As an example of this problem, consider the following. I am going to generate 21, 100-number draws from the normal distribution. These are, by definition, random numbers.
What happens when we look for significant \(p<0.05\) relationships?
The wild swings of p-values
Andrew Gelman gives a really thought-provoking demo on another problem with p-values (besides false positives): they are wildly variable for data that “comes from the same experiment”.
The setup for the demonstration is that we run 10 experiments on some phenomenon (the same phenomenon, to be clear), with the same underlying truth, we will end up with a startling range of p-values once we crunch all of our statistics. Let’s take a look.
In Gelman’s example, he just uses abstract figures, but I think that can be hard to grasp. So let’s think about a real outcome, just to ground ourselves. I’m a sensory scientist, so to make it easy for me to think about, let’s talk about differences in liking. Let’s say we’re talking about an average increase in liking of 20% (over a null hypothesis mean of 0): \(\theta=0.2\). Furthermore, we think our experiment is pretty good: we have a standard error of 1/2 the observed effect size–so our variability isn’t so wild that we can never see the effect. And, just to follow our typical assumptions, however our data are generated, they’re typically roughly normally distributed.
set.seed(1)
sims <-
tibble(
# we have 10 experiments
rep = 1:10,
# the "true" effect size is 20%
theta = 0.2,
# with a standard error of 1/2 our effect size
se = 0.1,
# then we run our "experiment": a sample from a sufficiently large
# experiment with our proposed mean effect size and SE
theta_hat = rnorm(10, theta, se),
# and finally we calculate the p-value of our results
p_value = 2 * (1 - pnorm(abs(theta_hat), 0, se))
) |>
mutate(theta_lab = str_c("theta_hat = ", round(theta_hat, 2)))
knitr::kable(sims)| rep | theta | se | theta_hat | p_value | theta_lab |
|---|---|---|---|---|---|
| 1 | 0.2 | 0.1 | 0.1373546 | 0.1695826 | theta_hat = 0.14 |
| 2 | 0.2 | 0.1 | 0.2183643 | 0.0289885 | theta_hat = 0.22 |
| 3 | 0.2 | 0.1 | 0.1164371 | 0.2442735 | theta_hat = 0.12 |
| 4 | 0.2 | 0.1 | 0.3595281 | 0.0003240 | theta_hat = 0.36 |
| 5 | 0.2 | 0.1 | 0.2329508 | 0.0198322 | theta_hat = 0.23 |
| 6 | 0.2 | 0.1 | 0.1179532 | 0.2381866 | theta_hat = 0.12 |
| 7 | 0.2 | 0.1 | 0.2487429 | 0.0128670 | theta_hat = 0.25 |
| 8 | 0.2 | 0.1 | 0.2738325 | 0.0061753 | theta_hat = 0.27 |
| 9 | 0.2 | 0.1 | 0.2575781 | 0.0100014 | theta_hat = 0.26 |
| 10 | 0.2 | 0.1 | 0.1694612 | 0.0901491 | theta_hat = 0.17 |
We can see from this example that, with 10 “reasonably large” (because we’ve gotten our SE down far enough to be able to see differences in means) experiments measuring the same phenomenon, we still have large oscillations in p-values. And, if we actually plot these p-values, we’ll see that they have a non-linear relationship to the actual effect size–how far our \(\widehat{\theta}\) is from our real \(\theta\).
sims |>
ggplot(aes(x = theta_hat, y = p_value)) +
geom_point() +
geom_hline(yintercept = 0.05, color = "red", linetype = "dashed") +
ggrepel::geom_label_repel(aes(label = theta_lab))
I think it’s actually even more dramatic when we run full simulations: not just getting our mean estimate of \(\widehat{\theta}\) but actually seeing how observations from such an experiment might contribute to this effect.
# We can simulate raw data instead to see the same effect. Let's see if I can
# do it (for my own edification).
# We want 10 different experiments, each with the same underlying real effect.
j <- 10
theta <- 0.2
# We then want 100 observations from each experiment, with a standard deviation
# of 1 (resulting in a standard error of 0.1)
i <- 100
sigma <- 1
set.seed(1)
sims <-
tibble(experiment = 1:j) |>
mutate(observed_value = map(experiment, ~rnorm(n = i, mean = theta, sd = sigma))) |>
mutate(exp_mean = map_dbl(observed_value, mean),
sd = map_dbl(observed_value, sd),
se = sd / 10,
p_value = 2 * (1 - pnorm(abs(exp_mean), mean = 0, sd = se)))
sims# A tibble: 10 × 6
experiment observed_value exp_mean sd se p_value
<int> <list> <dbl> <dbl> <dbl> <dbl>
1 1 <dbl [100]> 0.309 0.898 0.0898 0.000584
2 2 <dbl [100]> 0.162 0.958 0.0958 0.0904
3 3 <dbl [100]> 0.230 1.03 0.103 0.0264
4 4 <dbl [100]> 0.252 0.992 0.0992 0.0112
5 5 <dbl [100]> 0.161 1.17 0.117 0.169
6 6 <dbl [100]> 0.155 0.966 0.0966 0.108
7 7 <dbl [100]> 0.000420 1.08 0.108 0.997
8 8 <dbl [100]> 0.200 1.10 0.110 0.0680
9 9 <dbl [100]> 0.210 1.09 0.109 0.0536
10 10 <dbl [100]> 0.204 1.05 0.105 0.0530 Notice my use of nest() here: nest() is like a turbocharged group_by() that lets us use the map_*() functions to do more complex functions on a row-wise (data-set-wise) basis. The actual observed_value column contains our simulated data:
# Each row has `observed_value` of 100 draws from rnorm(mean = 0.2, sd = 1)
sims$observed_value[1][[1]]
[1] -0.42645381 0.38364332 -0.63562861 1.79528080 0.52950777 -0.62046838
[7] 0.68742905 0.93832471 0.77578135 -0.10538839 1.71178117 0.58984324
[13] -0.42124058 -2.01469989 1.32493092 0.15506639 0.18380974 1.14383621
[19] 1.02122120 0.79390132 1.11897737 0.98213630 0.27456498 -1.78935170
[25] 0.81982575 0.14387126 0.04420449 -1.27075238 -0.27815006 0.61794156
[31] 1.55867955 0.09721227 0.58767161 0.14619496 -1.17705956 -0.21499456
[37] -0.19428995 0.14068660 1.30002537 0.96317575 0.03547640 -0.05336168
[43] 0.89696338 0.75666320 -0.48875569 -0.50749516 0.56458196 0.96853292
[49] 0.08765379 1.08110773 0.59810588 -0.41202639 0.54111969 -0.92936310
[55] 1.63302370 2.18039990 -0.16722148 -0.84413463 0.76971963 0.06494540
[61] 2.60161776 0.16076000 0.88973936 0.22800216 -0.54327321 0.38879230
[67] -1.60495863 1.66555486 0.35325334 2.37261167 0.67550953 -0.50994643
[73] 0.81072635 -0.73409763 -1.05363340 0.49144624 -0.24329187 0.20110535
[79] 0.27434132 -0.38952095 -0.36866873 0.06482138 1.37808700 -1.32356680
[85] 0.79394619 0.53295037 1.26309984 -0.10418392 0.57001881 0.46709879
[91] -0.34252003 1.40786781 1.36040262 0.90021365 1.78683345 0.75848643
[97] -1.07659221 -0.37326541 -1.02461261 -0.27340064Then, we can look at our “null” and “real” distributions, and compare them to the sample we drew.
sims |>
unnest(everything()) |>
arrange(p_value) |>
mutate(experiment = str_c("p = ", round(p_value, 3)),
color = p_value < 0.05) |>
ggplot(aes(x = observed_value, color = color)) +
stat_function(fun = dnorm,
args = list(mean = theta, sd = sigma),
fill = "darkorange",
geom = "area",
alpha = 1/2,
color = "transparent") +
stat_function(fun = dnorm,
args = list(mean = 0, sd = sigma),
fill = "darkgreen",
geom = "area",
alpha = 1/2,
color = "transparent") +
geom_density(linewidth = 1,
show.legend = FALSE) +
facet_wrap(~experiment, nrow = 2) +
scale_color_manual(values = c("grey", "black"))
The problem is not that the p-value isn’t “right”; it’s that the p-value acts unintuitively around values drawn with known uncertainty from the same underlying situation. When we rely on the p-value to be the arbiter of “significance”–\(p<0.05\) is the magic number, right?–we will make strange decisions!
A quick summary of these problems
This is the kind of data-mining that seems ridiculous… until we start modeling the relationships among hundreds of predictors without some process for validating our findings. Our intentions, implicitly, which are erased by NHST, are that we want to find significance. So we keep testing until we find some (likely coincidental) relationships. This is a form of the next sort of problem we’re going to discuss: observer or selection bias.
Observer (selection) bias
Calling Bullshit (Bergstrom and West 2021) gives one of the best explanations of the phenomenon of selection bias that I’m aware of. Like most of you, I am more familiar with the idea of a confounding factor: a variable we fail to include in our analysis that explains a spurious relationship. This is the classic shark and ice-cream relationship:
But this XKCD comic points to the other problem–what are called “colliders” in causal theory. The idea is kind of counterintuitive, so this is why I think that the long discussion in Calling Bullshit is worth paying attention to. It is framed, as things so often are, in terms of dating:
A common complaint is that niceness and attractiveness seem to be anticorrelated in a dating pool: nice people are less attractive, and less attractive people are nicer. While it is possible that social reinforcement might explain rudeness in more attractive people (for example), this is equally likely to be a form of selection bias or a statistical collider. We can use data science to simulate the situation.
Let’s have someone named “A” as our example. Let’s assume the (probable reality) that niceness and attractiveness are fully uncorrelated. But also assume that A has some basic threshold for niceness and for attractiveness: A simply won’t date people who are both unattractive and mean. Furthermore, let’s assume that A isn’t a celebrity–they have an upper limit on their own datability. These simple constraints, as we’ll see, are enough to induce a negative correlation between niceness and attractiveness in the group of people available for A to date.
# A tibble: 10,000 × 6
attractive nice datability datable getable reasonable
<dbl> <dbl> <dbl> <lgl> <lgl> <lgl>
1 7.27 2.18 9.45 TRUE TRUE TRUE
2 7.22 7.17 14.4 TRUE FALSE FALSE
3 3.26 7.34 10.6 TRUE TRUE TRUE
4 5.42 5.59 11.0 TRUE TRUE TRUE
5 5.14 3.89 9.03 TRUE TRUE TRUE
6 1.67 4.19 5.87 FALSE TRUE FALSE
7 6.62 2.46 9.09 TRUE TRUE TRUE
8 1.18 4.98 6.16 FALSE TRUE FALSE
9 2.51 5.68 8.19 TRUE TRUE TRUE
10 7.00 5.79 12.8 TRUE TRUE TRUE
# ℹ 9,990 more rowsOK, so we have a bunch of folks whom A might be willing to date–some are nicer, some are more attractive.
What does that look like?
the_pool <-
dating_app |>
ggplot(aes(x = attractive, y = nice)) +
geom_point(alpha = 0.5, shape = ".") +
geom_smooth(method = lm, se = FALSE, color = "black") +
annotate(geom = "label", x = 0, y = 0, label = "italic(r) == 0.006", parse = TRUE) +
theme_classic() +
coord_equal()
the_pool
Great! As we specified, there is no relationship between physical attractiveness and niceness. Now, let’s imagine that A has some form of additive “dating criterion”: people’s physical attributes and personality get summed up into a “datable” score.
datable <-
dating_app |>
ggplot(aes(x = attractive, y = nice, color = datable)) +
geom_point(alpha = 0.5, shape = ".") +
geom_smooth(data = dating_app |> filter(datable), method = lm, se = FALSE, color = "red") +
geom_abline(aes(intercept = 8, slope = -1), linetype = "dashed") +
annotate(geom = "label", x = 0, y = 0, label = "italic(r) == -0.29", parse = TRUE) +
theme_classic() +
theme(legend.position = "none") +
scale_color_manual(values = c("black", "red")) +
coord_equal()
datable
Uh oh. Suddenly, the pool of acceptable dating candidates for A has a negative correlation between niceness and attractiveness, simply because of A’s selection criteria. But it gets even more complicated.
A exists in this same world, so presumably those who are way to the edge of the distribution impose their own nice/attractive criteria on A. Put another way: some people are just out of A’s league!
getable <-
dating_app |>
ggplot(aes(x = attractive, y = nice, color = getable)) +
geom_point(alpha = 0.5, shape = ".") +
geom_smooth(data = dating_app |> filter(getable), method = lm, se = FALSE, color = "orange") +
geom_abline(aes(intercept = 14, slope = -1), linetype = "dashed") +
annotate(geom = "label", x = 0, y = 0, label = "italic(r) == -0.14", parse = TRUE) +
theme_classic() +
theme(legend.position = "none") +
scale_color_manual(values = c("black", "orange")) +
coord_equal()
getable
So, even if A got wise to their own selection bias, the selection bias of others is going to impose a negative relationship on the dating pool between niceness and attractiveness. Yikes.
And… you guessed it, if we combine these effects:
reasonable <-
dating_app |>
ggplot(aes(x = attractive, y = nice, color = reasonable)) +
geom_point(alpha = 0.5, shape = ".") +
geom_smooth(data = dating_app |> filter(reasonable), method = lm, se = FALSE, color = "purple") +
geom_abline(aes(intercept = intercept, slope = slope),
data = tibble(intercept = c(8, 14), slope = c(-1, -1)),
linetype = "dashed") +
annotate(geom = "label", x = 0, y = 0, label = "italic(r) == -0.53", parse = TRUE) +
annotate(geom = "label", x = -1, y = 4, label = "A won't date") +
annotate(geom = "label", x = 10, y = 9, label = "won't date A") +
theme_classic() +
theme(legend.position = "none") +
scale_color_manual(values = c("black", "purple")) +
coord_equal()
reasonable
We get the whole rather sad story of how attributes that are uncorrelated in the population become associated by the (reasonable!) decisions of individual actors: how our positionality affects the reality we experience:
library(patchwork)
(the_pool + datable) / (getable + reasonable) +
plot_annotation(caption = "After Bergstrom & West (2021)")
Call:
lm(formula = nice ~ attractive, data = filter(dating_app, reasonable))
Residuals:
Min 1Q Median 3Q Max
-3.9610 -1.0621 -0.0375 0.9757 4.3747
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 8.15284 0.05796 140.67 <2e-16 ***
attractive -0.52315 0.01038 -50.38 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.381 on 6802 degrees of freedom
Multiple R-squared: 0.2718, Adjusted R-squared: 0.2717
F-statistic: 2538 on 1 and 6802 DF, p-value: < 2.2e-16This may seem like a cute example, but let’s try looking at real-world consequences. Adapting a similar example from McElreath (2020), we could apply this same reasoning to looking at grant funding (or your grades!). We know that often rubrics for funding require rating proposals on a number of (ideally uncorrelated) aspects. For example, a simple rubric might evaluate proposals on how neat it is and scientific quality. These are not necessarily correlated (as you will know if you read the popular scientific press releases), so we are once again in a similar situation. Each of these could be rated on a 10-pt scale, and the funders might decide that they will only consider funding proposals that have a combined score of over 15. We are back in the same situation again, but in an even worse position:
funding <- tibble(neat = rnorm(200, 5, 2),
quality = rnorm(200, 5, 2),
score = neat + quality)
funding |>
ggplot(aes(x = neat, y = quality)) +
geom_point(alpha = 0.5) +
geom_point(data = funding |> filter(score >= 15), color = "red") +
geom_smooth(data = funding |> filter(score >= 15), method = lm, se = FALSE, color = "red") +
annotate(geom = "label", x = 0, y = 0, label = "italic(r) == -0.94", parse = TRUE) +
geom_abline(slope = -1, intercept = 15, linetype = "dashed") +
theme_classic() +
labs(caption = "After McElreath (2020)")
We’ve now created a situation in which if research we fund is neat, it is likely of lower scientific quality, and vice versa, through a simple (and unintentional) selection effect.
Counting
Finally, because we’re way over time, I just want to use the previous examples to point out something important. No-one questioned my construction of my niceness/attractiveness or neat/quality metrics. But these are political choices by themselves. How do we assign people into numbers for their personality and appearance qualities? How do we make rubrics in the first place to make categories for funding? There is no objective reality in these decisions–they are the result of some sort of implicit or explicit social consensus. They change based on context–cultural, social, temporal. Take the infamous 3/5 compromise, proposed into law by the third president, James Madison. To quote Deborah Stone (2018):
Numbers get their authority from people who are able to exert rhetorical and political power to assign words to things and things to categories.
When the delegates at the Constitutional Convention discussed how to count the population in the federal census, the most contentious issue was whether to count slaves as property, in which case their owners would be taxed on them; or to count them as people, in which case the states where they lived would get more representatives in Congress. In the infamous compromise ultimately written into the US Constitution, slaves were counted as three-fifths of a person. In The Federalist No. 54, James Madison defended the rightness of this way of counting:
“[T]he Federal Constitution therefore, decides with great propriety on the case of our slaves, when it views them in the mixt character of persons and property. This is in fact their true character.” (Madison, Federalist Papers No 54)
…
When we left off talking about validity, I asked you to explain what you cut—what didn’t get into the heap of peas to be counted. Now I want to ask you to justify your cuts.
I encourage–no, I ask–you to take a look at the whole lecture, as I think the question of how we count is something we should really be asking ourselves more as data-literate scientists.
And with that, we’re done!
Session info
R version 4.5.1 (2025-06-13)
Platform: aarch64-apple-darwin20
Running under: macOS Tahoe 26.5.1
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: America/New_York
tzcode source: internal
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] patchwork_1.3.2 readxl_1.4.5 ggforce_0.5.0 skimr_2.2.1
[5] lubridate_1.9.4 forcats_1.0.1 stringr_1.5.2 dplyr_1.1.4
[9] purrr_1.1.0 readr_2.1.5 tidyr_1.3.1 tibble_3.3.0
[13] ggplot2_4.0.0 tidyverse_2.0.0
loaded via a namespace (and not attached):
[1] generics_0.1.4 lattice_0.22-7 stringi_1.8.7 hms_1.1.3
[5] digest_0.6.37 magrittr_2.0.4 evaluate_1.0.5 grid_4.5.1
[9] timechange_0.3.0 RColorBrewer_1.1-3 fastmap_1.2.0 Matrix_1.7-3
[13] cellranger_1.1.0 jsonlite_2.0.0 ggrepel_0.9.6 mgcv_1.9-3
[17] scales_1.4.0 tweenr_2.0.3 cli_3.6.5 rlang_1.1.6
[21] polyclip_1.10-7 splines_4.5.1 base64enc_0.1-3 withr_3.0.2
[25] repr_1.1.7 yaml_2.3.10 tools_4.5.1 tzdb_0.5.0
[29] vctrs_0.6.5 R6_2.6.1 lifecycle_1.0.4 htmlwidgets_1.6.4
[33] MASS_7.3-65 pkgconfig_2.0.3 pillar_1.11.1 gtable_0.3.6
[37] glue_1.8.0 Rcpp_1.1.0 xfun_0.53 tidyselect_1.2.1
[41] rstudioapi_0.17.1 knitr_1.50 farver_2.1.2 nlme_3.1-168
[45] htmltools_0.5.8.1 labeling_0.4.3 rmarkdown_2.30 compiler_4.5.1
[49] S7_0.2.0 References
Bergstrom, Carl T., and Jevin D. West. 2021. Calling Bullshit. Random House.
Collaboration, Open Science. 2015. “Estimating the Reproducibility of Psychological Science.” Doi:10.1126/Science.Aac4716. Science 349 (6251): aac4716.
Ioannidis, John P. A. 2005. “Why Most Published Research Findings Are False.” PLOS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.
Kruschke, John K. 2010. “Bayesian Data Analysis.” WIREs Cognitive Science 1 (5): 658–76. https://doi.org/10.1002/wcs.72.
Lindley, Dennis V., and Lawrence D. Phillips. 1976. “Inference for a Bernoulli Process (a Bayesian View).” The American Statistician 30 (3): 112–19.
McElreath, Richard. 2020. Statistical Rethinking. Second. CRC Press.
Stone, Deborah. 2018. “The 2017 James Madison Award Lecture: The Ethics of Counting.” PS: Political Science & Politics 51 (01): 7–16. https://doi.org/10.1017/S1049096517001767.
Stone, Deborah. 2020. Counting: How We Use Numbers to Decide What Matters. W. W. Norton & Company.
Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108.