Seeing Science Like a Language Model

I’m writing a book about the worldview I’ve developed by writing, coding, and living with AI. Last week I published the first piece from it, about the differences between the old (pre-GPT-3) worldview and the new. Here’s the second, about how AI will impact science.—Dan Shipper

Was this newsletter forwarded to you? Sign up to get it in your inbox.

I’m writing this to you from Bocas Del Toro, Panama. I’m living in a little cabin in the jungle at the top of a hill five minutes from the Caribbean sea. When I walk down to the water, the ocean’s flat plane stretches out in front of me like an endless bed, its surface raised here and there with ridges of foam-tipped waves—a bedsheet not quite pulled taught. On this clear evening, I have a front-row seat to our cosmic neighbors’ celestial ballet. The red sun dips slowly behind the horizon line and the moon loiters ominously above my head; the stars wander in circles.

The ocean looks flat to me, but I know—really—it is curved. The sky looks to be teeming with movement but really, I know, I am the one that is moving. This is intuitive to you and me because it’s how we’ve grown up. We know that the way things appear is different from the way they are—we trust the science.

To people in Copernicus or Galileo’s day, these ideas were preposterous. If the earth was in motion, why weren’t the trees blowing in the wind generated by its orbit? Why would an object dropped from a height drop straight down if the planet was shifting underneath it? If the earth was round instead of flat, why didn’t we fall off of it?

Their world was organized quite differently from ours. That we see the world the way we do is a testament to how powerfully ideas shape what we see, and our capacity, through cultural transmission, to know things that we have not directly experienced.

And this is a good thing. This way of seeing the world, as I’ve said over and over again, is responsible for rockets and vaccines, computers and smartphones. It has changed everything about our culture:

When we talk about "forces" shaping society or "momentum" in markets, we're borrowing concepts from physics. If you’ve ever described having “chemistry” with another person, or the people in a charismatic politician’s “orbit”, or needing to get “leverage” in a deal, or the “friction” between two co-workers, you’re using ideas and words from Newton’s time. If you’ve ever read Stephen Pressfield’s The War of Art, and encountered his concept of "resistance," or Robert Greene’s The 48 Laws of Power which tries to identify simple, general, universal laws of human behavior, you're seeing how deeply rationalist Enlightenment thinking has penetrated our understanding of life itself.

But this way of thinking has also begun to reach its limits. Just as it did with machine learning research, the tendency to reduce, to break apart, to make explicit, and to generalize has failed, hard, in many places.

The limits of science

Psychology and the replication crisis

Psychology is an easy example. Most psychology research is built on linear regressions—a statistical tool that assumes straightforward, predictable relationships between variables. When we use linear regression, though, we're also imposing a specific view of how the world works. We're saying: "If we change X by this much, Y will change by that much, plus or minus some random noise."

It’s the standard scientific approach, inspired by Newtonian physics, applied to the complex domain of the mind. But human behavior rarely follows such neat, predictable patterns. Our actions and reactions are deeply contextual, interconnected, and often nonlinear. Small changes in circumstances can lead to dramatic shifts in behavior, while major interventions sometimes produce no effect at all.

The result is a replication crisis: Scientists use tools built for physics to make their intuitive theories fit into statistical models, only to find that their results can't be reproduced. Usually, the replication crisis is portrayed as malfeasance on the part of academics looking to “publish or perish” who are "p-hacking" their way to implausible statistically significant results.

But the deeper issue, per the psychologist and philosopher of science Tan Yarkoni, is not necessarily a crisis of replication but one of generalizability. We tend to overstate how universal what we’ve found is; if we were better at replicating the context of the original study—the lab environment, the researchers, the subject population, the specific interventions, and countless other variables—we’d probably replicate the results.

What’s really going on here?

The usual picture of science is that it progresses through a steady accumulation of knowledge, with each discovery building upon previous findings. Scientists construct careful experiments designed to falsify their hypotheses—to truly put their explanations at risk. If a theory survives rigorous attempts to prove it wrong, we can be more confident it captures something true about reality.

The currency of science is what the physicist David Deutsch calls “hard-to-vary” explanations: explanations that act as precisely engineered machines where every part serves an essential purpose.