Imagine you're using an artificial intelligence to automate a process.
You set up examples for the AI to learn from: sample inputs and their corresponding expected outputs. When training the AI, you score its attempts to learn the process by penalizing when it deviates from the expected outputs. At the end of the training, you find that the AI has achieved a perfect score... but all its outputs are blank! Upon further investigation, you find that instead of learning the process you wanted, the AI has maximized its score by simply deleting the files containing the expected outputs... as if simply refusing to be scored!
Believe it or not, this actually happens in computer science research. And it serves as an allegory for ways that our human brains can sabotage themselves too.
Examples of this have come up before in Superorganizers:
- In a productivity cycle, the “slump” (the period where you aren’t being productive) might be exacerbated by feelings of guilt and worries that it will never go away. You might avoid trying to be productive again, because trying means you have to face that guilt and those worries.
- You might feel distressed just thinking about people, places, or objects associated with your work.
- Maybe you are often distracted by urgent but unimportant tasks that help you to avoid feelings of boredom, loneliness, fatigue, or uncertainty.
- You overdesign an organization system because of fear that if it goes wrong, it will go wrong permanently.
Our brains learn powerfully by association—negative Pavlovian conditioning—to avoid things that harm us. But in all of these cases, we have learned to avoid not the real agent of harm, but the mere thoughts or feelings associated with them. Just like the AI in our story learned to avoid its training entirely rather than avoiding just the incorrect responses, we have also learned to avoid the cognitive tools we use to understand and anticipate bad outcomes rather than the bad outcomes themselves.
We often assume that our internal experiences can be controlled through willpower: that we can consciously decide to control how we feel or think. After all, deliberate conscious control is the approach we take to most everyday problems (like scrubbing hard to remove a stain, or summoning your willpower to climb that last flight of stairs). And we are culturally taught that it is: we are told as children to "control your anger" as opposed to controlling how we act in response to that anger. We try to suppress feelings of anxiety, in a conflation of the experience of anxiety with loss of control. But modern psychology challenges the popular "folk" conception that thoughts and feelings can always be willed away.
Ironic Process Theory
Find Out What
Comes Next in Tech.
Start your free trial.
New ideas to help you build the future—in your inbox, every day. Trusted by over 75,000 readers.
SubscribeAlready have an account? Sign in
What's included?
- Unlimited access to our daily essays by Dan Shipper, Evan Armstrong, and a roster of the best tech writers on the internet
- Full access to an archive of hundreds of in-depth articles
- Priority access and subscriber-only discounts to courses, events, and more
- Ad-free experience
- Access to our Discord community