He Built an AI Audience Simulator. It’s the Future of Customer Research.

TL;DR: Today we’re releasing a new episode of our podcast AI & I. I go in depth with Michael Taylor, a freelance prompt engineer and the author of our column Also True for Humans. We get into how good AI can be at simulating an audience—and how we can use that to test our ideas and assumptions. Watch on X or YouTube, or listen on Spotify. Here’s a link to the episode transcript.

Was this newsletter forwarded to you? Sign up to get it in your inbox.

Michael Taylor has perfected the art of getting AI to speak in tongues. He’s taught it to mimic the voices of your customers—so you can see how they would respond before you ship.

Michael is the creator of Rally, a market research tool that lets you simulate an audience of AI personas. He built a simulator that lets us A/B test Every’s headlines on an audience that mimics the real Hacker News audience. It’s become a part of my writing workflow, and I love it because you test your assumptions quickly, cheaply, and without any of the risks of putting something out into the world.

Besides Rally, Michael co-authored a book on prompt engineering for O’Reilly, and he writes a column for Every about managing AI tools like you would people. In a past life, he founded a growth marketing agency which he grew to 50 people and sold in 2020. One of the reasons I’m drawn to Michael’s work is because he has a tinkerer’s mindset. He’s always exploring the limits of what a new technology can do, and what he’s into today, everyone else will likely discover six months later. We spent an hour talking about using language models to judge your work, best practices for assessing an AI’s performance, and Michael’s flow inside Cursor. He also demos Rally live on the show, testing three different potential headlines for an Every article.

You can check out our conversation here:

If you want a quick summary, here are some of the themes we touched on:

AI can be very good at simulating people (00:04:30)

If you’re a regular Every reader I hope you’re using role prompting, or telling the language model to respond like an expert in a specific field. You’d be surprised at how effective it is. Almost a year ago, I gave GPT-4 some information about me, like my tweets and a few journal entries, and got it to take a personality test as me. Then I had my girlfriend at the time take the test while pretending to be me. GPT-4’s responses were more aligned to my own than hers. 

Get the wisdom of the crowds (00:08:45)

Our editor in chief Kate Lee and I had a difference of opinion in October last year. I wanted to describe Every as a “meta” media company, while Kate liked the prefix “multimodal.” Michael happened to be writing an article about simulating 100 AI personas for market research, so we enlisted a virtual focus group to settle the debate. The AI personas chose “multimodal,” with each one detailing the reasons for their choice, and as I read through them, I changed my mind. This was one of the pushes for Michael to build and launch Rally. I love the tool because it's a way to get fair, quantitative feedback on your ideas in a risk-free environment. 

Why your ideas still matter (00:12:23)