1010: Getting the Most Out of Generative AI at Work with Jeremy Utley

Pete Mockaitis
Jeremy, welcome.

Jeremy Utley
Thanks for having me.

Pete Mockaitis
I’m so excited to chat, and I’d love it if you could kick us off by sharing one of maybe the most fascinating and surprising discoveries you’ve made about some of this AI stuff with all your poking and prodding and playing.

Jeremy Utley
I’ll poke the bear right from the get-go. My observation is most people are what I call prompt hoarders, which is that they’ve got a bunch of Twitter threads saved, and they’ve got a bunch of PDFs downloaded in a folder, marked, “Read someday,” but they aren’t actually using AI. They’re just hoarding prompts.

And I think of it as empty calories. It’s a sugar high. And what a lot of people are doing is they are accumulating, for themselves, prompts that they should try someday, but they’re never trying them, which is akin to somebody eating a bunch of calories and then never exercising.

And my recommendation, like, here, I’ll give one simple thing that somebody would probably want to write down. Hey, when you’re jumping into advanced voice mode, isn’t it annoying how ChatGPT interrupts you? Well, did you know that you can tell ChatGPT, “Hey, just say, ‘Mm-hmm’ anytime I stop talking, but don’t say anything else unless I ask you to”?

Everybody who’s played with advanced voice mode one time is like, “Oh, my gosh, I got to do that. That’s, oh, it is annoying.” And I guarantee you 95% plus, people who even think that, will never actually do it because they think it’s more important to listen to the next 35 minutes of this conversation than actually hit pause and go do that. And my recommendation would be, stop this podcast right now, go into ChatGPT and actually do that. That would be like going to the gym.

Pete Mockaitis
I’m thinking I’m doing that right now. Is that okay? Is that rude?

Jeremy Utley
Yes, of course. No, it’s great.

Pete Mockaitis
I think I’m following your suggestions. So, in ChatGPT, iPhone app, I’ve got Pete Mockaitis, I just issue the command, like, “Remember this”?

Jeremy Utley
I would open a new voice chat. So, from the home screen, on the bottom right, there’s kind of like a little four-line kind of a button. If you hit that, that’s going to open a new conversation in Advanced Voice mode. And the first thing I would say is, “Hey, I want to talk to you for a second, but I don’t really need you to say anything. So, unless I ask you otherwise, would you please just say, ‘Mmm-hmm,’ one word only and let me keep talking.”

Pete Mockaitis
Okay. Hey, ChatGPT, here’s the thing. When I’m talking to you, what I need you to do, if I ever stop talking for a moment…there, he just did it.

Jeremy Utley
Isn’t that hysterical? Yeah, that’s hysterical.

Pete Mockaitis
Oh, Amber, when I’m talking, I need you to remember to only interrupt with just the briefest mm-hmm, or yes, or okay until I ask for you to begin speaking. Do you understand? And can you please remember this?

Amber
Be as brief as possible with confirmations and wait until prompted to speak further.

Pete Mockaitis
Thank you. It’s done.

Jeremy Utley
Now what you need to do is you actually need to continue the conversation. And you need to see, “Does ChatGPT respond with mm-hmm?”

Pete Mockaitis
You know, I like that. And I love those little tidbits in terms of, “Hey, remember this and do this forever.” Sometimes I like to say, well, I have. I have said, “Give me a number from zero to 100 at the end of every one of your responses, indicating how certain you are that what you’re saying is, in fact, true and accurate and right.”

Now, its estimates are not always perfectly correct, but I know, it’s like, “Okay, if he said 90, I’m going to maybe be more inclined to do some follow-up looks as opposed to if I get the 100.”

Jeremy Utley
Yeah, I think that’s great. I think there’s all sorts of little things. The problem is, right now, people are accumulating, or they actually aren’t even accumulating, but they think they’re accumulating for themselves all these tips and tricks, but they aren’t using any of them. And so, to me, what I recommend folks do, I actually just wrote a newsletter about this just yesterday, it went out this morning.

What I recommend folks do is take 15 minutes per day and try one new thing. It requires two parts. Part one, a daily meeting on your calendar that says “AI, try this.” And that’s it. It’s just 15 minutes, “AI, try this.” And thing number two, you need an AI-try-this scratch pad, which is just a running list of things that you heard.

So, like everybody’s scratch pad right now, if they’re listening to this conversation, should include, one, tell ChatGPT to only say mm-hmm unless you want a further response. That’s not forever, but at least in a one interaction, right? And, two, they should tell ChatGPT to always end its responses with a number, an integer between zero and 100, to indicate its conviction of its response.

Everybody literally what? We’re 10 minutes into this conversation, not even, everyone should have two items on their scratch pad. The problem is most people are going to get to this, to the end of this interview and they aren’t going to have a scratch pad and they aren’t going to have any time blocked on their calendar to do it.

And the next time they use ChatGPT, it’s going to be mildly disappointing because they’re coming off a sugar high and they think the treadmill’s broken, basically. So, I mean, obviously, there’s a ton there that we can unpack, but I think for most people, what most people fail to understand is the key to use is use.

And just like a treadmill doesn’t help you combat heart disease unless you actually get on it, AI is not going to unleash your creativity or your productivity unless you use it and learn how to use it. And that, to me, that’s pretty much my obsession these days, is helping people be good collaborators to generative LLMs.

Pete Mockaitis
That’s lovely. And I suppose we could dork out about so many tips and tactics and fun things that you can do. But I’d love it if you could just orient us, first and foremost, in terms of, if there’s research or a powerful story that really makes the case that, “Hey, these things are really actually super useful for people becoming awesome at their jobs for reals as opposed to just a hype train or fad.”

Jeremy Utley
I’ll tell you about my good friend, let’s call him Michael. It’s not his name. Names have been changed to protect the innocent. But Michael was a lobbyist in Washington, D.C. and he and his family wanted to move back home to Tennessee.

And he was looking for a job, and he got a job offer from a firm. And he reached out to me and said, “Hey, I’m kind of bummed because I feel like this firm is low-balling me. But my wife really just wants me to take it because she wants to be back near family in Tennessee, and I’m really struggling with knowing ‘Should I push back?’ because I know that I deserve more, but I don’t want to screw up this opportunity to get close to family.”

And I said, “Well, have you role-played it with ChatGPT?” And he said, “What do you mean roleplay with ChatGPT?”

Pete Mockaitis
Of course, the question everyone asks.

Jeremy Utley
Right. And I said, “Well, you can roleplay the negotiation and just kind of get a sense for what the boundary conditions are.” And he’s like, “Okay, wait. What do you mean?” And I said, “Well, open ChatGPT and tell it you want to roleplay a conversation. But, first, you want ChatGPT to interview you about your conversation partner so that it can believably play the role of that conversation partner.”

“You want it to start as a psychological profiler and create a psychological profile of your counterpart. And then once it creates it, you want ChatGPT to play the role of that profile in a voice-only conversation until you say that you want to get feedback from its perspective and a negotiation expert’s perspective.” And he’s like, “Give me 15 minutes.”

So, he leaves, texts me in 15 minutes, “Dude, this is blowing my mind. What do I do next?” I said, “Well, Michael, the next thing I would do is tell your conversation partner that you want it to offer less concessions, and you want it to not be nearly as amenable to recommendations because it’s had a bad day or it’s slept poorly or something, okay? I want you to get a sense for what does it feel like if the conversation goes badly, right?”

He goes, “Okay, I’ll be right back.” Comes back, “Dude, this is blowing my mind.” And he did a series of these interviews, and I touched base with him. And a couple of days later, I said, “Michael, what’s up?” And he said, “Well, three things. One, I didn’t know what my leverage in the conversation was until I roleplayed it a handful of times. Two, I didn’t have clarity on what my arguments were until I roleplayed it a few times, what the sequence of my argument should be. And, three, and most importantly, I’m no longer nervous about going into this negotiation.”

And then a week later, he dropped me a note saying, “By the way, we’re moving back to Tennessee, and I got a much better salary than they had originally offered me.”

It turns out one of generative AI’s unique capabilities is imitation and taking on different roles. As an example, you can go into any conversation you’ve ever had with ChatGPT and just say, “Hey, would you mind to recast your most recent response as if you’re Mr. T?” And, instantaneously, “Yo, fool, I can’t believe you didn’t believe the last thing I said,” just immediately starts doing it. It doesn’t take much.

And the power, actually, emotionally and psychologically, of having roleplayed with a very believable conversation partner has a profound psychological and confidence boost effect to the person who’s engaging the roleplay.

Pete Mockaitis
That’s perfect in terms of, yes, that is a top skill that the AI has, and about the most lucrative per minute use case I can think of a typical professional doing. And you’re right, that confidence, I have actually paid a real negotiation coach, and he suggested we do a roleplay. And I had the exact same experience, like, “Oh, you know what, I guess I don’t feel so silly asking for what I wanted to ask for now. It seems fairly reasonable for me to do so. And I’m going to go ahead and do so.” And it worked out rather nicely. And so, to know that you can do a decent job for near free with AI instead of hiring a phenomenal negotiation coach is pretty extraordinary.

Jeremy Utley
It’s remarkable. And so, we actually, my research partner, Kian Gohar and I wrote a weekend essay in The Wall Street Journal about this topic. But think about a salary negotiation as a flavor of a broader thing, which is difficult conversations. Maybe it’s a performance review. Maybe it’s a termination conversation. Maybe it’s talking to a loved one about the fact that you’re not going to come home for the holidays.

There’s all sorts of scenarios where roleplaying the interaction increases your confidence, strengthens your conviction, helps you, perhaps, exchange perspectives. Perspective taking is a really important thing, to understand, “How did this land to the perspective of my conversation partner?” That’s actually something that’s really hard for humans to do but an AI can read it back to you in a way that’s really reflective of your conversation partner, and, in a way, that you can understand.

So, we wrote a whole article about this but that’s just one class of activities. But the point is it really helps when you actually do it. Again, the tendency is for somebody right now to go, “Oh, cool, roleplay.” But if they don’t pull out their scratch pad, and say, “Ask ChatGPT to be a conversation partner in this upcoming salary negotiation, or my quarterly performance review, or my conversation with my loved one about our care for our kids,” or whatever it is, you just won’t do it.

I’ve even built, and you can link it in the show notes if you want, I built a profiler GPT, which is basically, it’s a version of ChatGPT which remembers who it is, unlike Drew Barrymore in “50 First Dates” where you have to remind ChatGPT who it is every time. A GPT is just like a Drew Barrymore who has memory, right, and like a real human being.

And what this GPT is instructed to do is interview a user about their conversation partner as a psychological profiler would, and then create an instruction set to give the user to copy-paste into a new ChatGPT window of instructions to GPT to perform the role of the psychological profile that it created. So, that’s totally free, but somebody can just open that up and you can say, “My significant other, Sherry,” and all of a sudden, this GPT will just interview you, ask you a bunch of questions, you answer them, and then it spits out an instruction set to a new GPT to play the role of Sherry in the scene that you have told it about.

Pete Mockaitis
I love that. And it also illustrates one of your core principles to effectively using AI is to flip the script a little bit and say, “No, no, you ask me questions.” Can you tell us a bit more about that?

Jeremy Utley
I mean, why is our default orientation that I’m the one with the questions and an LLM is the one with the answers? That’s how everybody approaches it, right? Because that’s how Google works, right? We never think, “Google, ask me a question.” It’s like, “Uh, what are you talking about?” A language model is not a technology, it’s an intelligence. That’s how I would invite people to think about it.

And you can get to know another intelligence, in a weird way, that sounds kind of crazy, but one of the things you can do is another intelligence can help you get to know yourself better. And the simple way to think about it is, here’s another thing for your AI-try-this scratch pad, folks. Get ready to write this down.

Think of a difficult decision you’re trying to make in your life, “Okay, should I take this job? Should we make this decision? Should we move? Should we put our kid in this other school?” whatever it might be, think of that decision, and then go to ChatGPT and say, “Hey, I’d like to talk about this. But before you give me any advice, would you please ask me three questions, one at a time, so that you better understand my perspective and my experience?”

Well, that is right there. If you say you were trying to figure out whether you’re going to send your kid to a new school, I have four children so it’s a very realistic kind of decision for me. I can Google and learn all about the school. But should I send my child to the school? I’m just going to get their marketing material and it’s not going to be contextualized to me at all. But if I go to ChatGPT, and say, “Hey, I’m thinking about sending my child to this school, I’d love to get your advice. But before you tell me anything, would you please ask me three questions?”

All of a sudden, well, it’ll… “Tell us about your child’s favorite subjects.” I’ll tell it. “Tell us about any weaknesses or difficulties that your child has had in school thus far.” I’ll tell it. “Tell us about your child’s favorite teachers.” I don’t know, but an LLM will ask questions like that. And then it will say, “Based on your answers, here’s how I would approach this conversation.”

That’s what I mean by turning the tables on an AI, is put it in the position of an expert that’s getting information from you rather than the default orientation, which is you’re the expert and you’re getting information from the AI.

Pete Mockaitis
Okay. Now, we’ve been saying the words ChatGPT a lot. I’m curious, in the world of LLMs, we got your ChatGPT, we got your Claude, we got your Perplexity, we got your Gemini, we got your Grok.

Jeremy Utley
Don’t forget Llama.

Pete Mockaitis
Do you think of them as having different strengths and weaknesses? Or are they kind of all interchangeable for whatever you want to use them for?

Jeremy Utley
I don’t think they’re interchangeable, but I don’t think it’s necessarily because of the underlying model. I think a lot of it is a UX thing. I think that the best AI is an AI that’s available to you that you will use. Again, the key to use is use. So, which is the best AI? Well, it’s the AI that you’re going to use. So, where are you? Most of the time you’re on your mobile. So, I would say it’s probably the AI that’s got the best mobile experience.

And what’s your default orientation? My belief is that the far better orientation towards AI is voice, not fingers. If you think about how you typically interact with a machine, you’re typically typing stuff into a machine. And I like to affectionately refer to my fingers, like as I wiggle them in front of the screen, as my bottlenecks. These are my communication rate limiters right now.

Notice you and I aren’t typing to each other. Like, that sounds absurd, right? And yet that’s how we talk to most machines. I’m typing into the terminal. Well, now, I mean, OpenAI, besides developing the world’s fastest growing consumer application, they created the world’s best voice-to-text technology. And furthermore, now they’ve got AIs that actually just process voice, don’t even go to transcription.

But the point is AIs are now capable of understanding natural language. We talk about this phrase, natural language processing. You probably hear that phrase, natural language processing. And that means something technically. I think to humans, the important thing about natural language processing isn’t what happens technically, but it’s actually you as a human being can now use your natural language, which is your spoken word with your mouth instead of your fingers.

And I would say to anyone who’s listening to this, if your default orientation to any AI, ChatGPT or otherwise, is fingers, you are limiting yourself. You’re trying to run with crutches. It’s, like, you’re in a sack race, okay? Use your voice, lose your thumbs, and watch the level of your interaction skyrocket.

Pete Mockaitis
Okay. Well, as we speak in late October of 2024, as far as I know, having played around with the apps, it seems like, indeed, OpenAI’s ChatGPT has got the voice natural interaction thing down the best, as far as I am aware of. Is that your experience?

Jeremy Utley
In my experience, it is. The only other comparison I would say is Meta’s Llama has voice as well, which you can access via WhatsApp or anything like that. The caveat, I would say, is, you know, I was doing a demo. I had a reporter at my place yesterday, kind of I was doing a demo of how I how I use AI in my personal workflow as a writer. And one of the things that I was showing was I’ll use OpenAI ChatGPT voice mode, but then I’ll often grab all the text with my cursor or with my mouse, and I’ll drop it into Claude, and I basically will parallel process ChatGPT and Claude.

So, the fact that Claude doesn’t take voice input isn’t a hindrance if I’m on my computer. When I’m on my mobile device, which, I’m probably on my mobile more than I’m at my computer actually, Claude doesn’t handle voice input, and it’s a little bit unwieldy to go back and forth in apps on your mobile relative to toggling between windows on your computer. So, it’s not to say that means ChatGPT is the best, but when you say, if you have to choose one, right now the model which is most optimized for voice interaction in a – intuitive interface. That, to me, is the way that you should prioritize, is, “What’s intuitive? What can handle the widest range of human input?” And ChatGPT’s got great vision and great voice recognition. And, therefore, I would use that. I’ll give you another example. I’m taking Spanish classes with my kids, okay, and we’re doing these lessons and we have a tutor talking to us on a bi-weekly basis.

And I get this assignment. I’ve got to conjugate a particular verb, and she wants us to write it down. We got to take pictures of it right now. Write it down in my notebook. I’m trying to conjugate this verb, and I kind of get stuck. And I’m thinking, in my mind, like, we only get her twice a week. I’m not going to be able to talk to her until Thursday. It’s Tuesday afternoon. And I thought, “I wonder if ChatGPT can help me.” And I just take a photo of my notebook and my crappy chicken-scratch handwriting, okay, in Spanish, by the way.

I take a photo, I say, “Hey, you’re my Spanish tutor. Can you tell me what I’m doing right now?” “Oh, it looks like you’re trying to conjugate the verb “estar,” and it looks like you’ve missed seven accent marks. If I were going to correct your paper, I would do this,” and rewrote all of the statements that I just made, but properly. “I made this change because of this. I made this change because of this. I made this change…”

And I go, “Dude, it read…” I mean, if you see my handwriting, it’s abysmal. But I did miss all the accent marks, it got that right, because I’m not an accent marker. But, anyway, the point is, the vision capabilities are spectacular too. And when you start to think, again, right now, write that down on your AI scratch pad, people.

Like, people are listening, and the thing is it’s like popcorn at a movie, and we’re just like, “Nom-nom, that’s so interesting. Oh, photos of AI should do that.” You will not do it if you don’t write it down. I’m obsessed with this idea. As you probably know, I’ve got this AI podcast called Beyond the Prompt, which we have amazing kind of experts and lead users and things like that.

We had a guy, who’s former dean at Harvard, 30 plus year learning scientist veteran named Stephen Kosslyn, recently. And he’s kind of the father of the school of thought called active learning. Maybe some folks have heard of it. Active learning, some people mistake as, you know, learning by doing, which isn’t exactly correct, but doing what you learn is an important step.

And what he says is he would contrast what’s typically known as passive learning, which is just consumption, but he would say it’s not actually learning at all. It just happens to you. It’s like you’re renting it. And that information has a very short shelf life and a very short expiration window. Any information that you consume but do not use, you effectively did not consume it.

Pete Mockaitis
Okay. Yes, well said. Well, I’d also love to get your pro take here. It seems like we’ve got a whole lot of cool things we can do that are very handy. What are some things you recommend that we not do, or some limitations like, “No, no, you’re not prompting it wrong. It’s just not going to do what you want it to do right now”?

Jeremy Utley
You know, I’m not a fanboy, I’m not a stockholder, I don’t have any secondary shares. I have yet to butt up against the limitations of use, to be honest with you. I think, right now, most people’s primary limitation is not the technology, it’s their imagination. I would say, like, one way that I’ve put it to students at Stanford is, “The answer is yes. What’s your question?” “Could it…?” The answer is yes. The problem is, for most people, they don’t actually have a question.

Pete Mockaitis
Okay. Well, Jeremy, if I could put you on the spot a little.

Jeremy Utley
Yeah, please, please, please, by all means, but the challenge is actually finding a question worth asking.

Pete Mockaitis
Okay. One thing I’ve tried every which way I can to say, “Yo, here’s a transcript of a podcast interview. What I want from you is to give me 10 options for titles that would be great, that are kind of like these dozens of title options I’ve written for you right here, I previously selected, or teasers.” And then whenever I do that, I get 10 or 20 options, and I go, “Hmm, not one of them am I like, ‘Yes, that is intriguing. That is awesome. That’s a phenomenal title that I want to use.’”

Now, it can nudge or steer me in some good directions, like, “Okay, that was a good phrase there. That was a good word there.” And maybe that’s sort of good enough in what I should expect from it in terms of, yeah, you can have a back-and-forth dialogue, it’s not going to spit out the perfect thing the first time, and be grateful for that. But I don’t know, since you are the master, any pro tips on how I can make it do this thing it just doesn’t seem to be able to do?

Jeremy Utley
So, this is great. What I’m hearing you say is actually a great case study of what we observed in our study, which got published by Harvard Business Review and Financial Times and NPR. We studied teams trying to solve problems, and you could call “Titling this podcast” as a problem that you’re trying to solve. We studied teams and individuals trying to solve problems with generative AI and studied “What do they do?”

And one of the kinds of natural problems that people have is they treat an LLM like it’s an oracle. Like I give it a question and it just magically gives me the right answer right off the bat. And what we would say is teams that treat AI like an oracle tend to underperform. But that’s not to say that everyone who uses AI underperforms. There’s a small subset of folks we studied who actually outperform.

The difference is they didn’t treat AI like an oracle. They treated AI like a co-worker, like a collaborator, like a thought partner. And so, what that interaction might look like is you ask for, say, 10 or 20, “Make it like this.” And then you get the output, and what it looks like to…let me ask you this. If an intern gave you 10 titles that you thought were mediocre, what would you do? Would you fire the intern?

Pete Mockaitis
Well, no, I would say, “Oh, hey, thank you for this. This is my favorite. This is my least favorite. That kind of what I’m looking for is, generally, more actionable, more intriguing, based on the needs of our listeners,” da, da, da, da.

Jeremy Utley
Do you do that to ChatGPT?

Pete Mockaitis
I’ve tried it sometimes.

Jeremy Utley
Yeah, you got to kind of, you got to critique the model’s output. You got to give it feedback. And I had that experience, actually. I had a hero of mine, Ed Catmull on my show a while ago, founder of Pixar, and I wanted the perfect title, of course. It’s, like, got to be the best title ever, right? And I asked for 10 and then I immediately always asked for 10 more.

I don’t even read the first 10. I asked for 10 more and never had ChatGPT say, “Dude, come on, you didn’t read my first ones, you know.” And they’re mediocre, you know, they’re okay. And I said, “Hey, I like one and three in the first set. I like seven and nine in the second set. Can you give me 10 more like those?” What do you think, are they better or worse?

Perfectly the same. Like, not any better, not any worse. And I was like, “Huh, but why? Why didn’t I like one?” I said, “Huh, okay,” I had to think. And what’s funny is, in our study, people who underperformed, AI feels like magic to them. It’s, like, they don’t do as well, but they’re like, “Wow, it just happened so fast.” People who outperform, who use AI to get to better work, it doesn’t feel like magic. It feels like work.

And that’s actually, that’s kind of a fundamental tension. I think we expect it to feel like magic or it sucks. And the truth is it’s just like working with another collaborator, and you do get to better outcomes if you’re willing to put in the work. And in this case, for me at least, the work was, I like number one because I’m a nerd and it has like an obscure movie illusion. I like number three because there’s a pun, and I’m a punny guy. I like number seven because there’s a movie reference baked in and I like number nine, whatever it is.

Then I said to ChatGPT, “Would you leverage that rationale as design principles for another 10, please?” six of the 10 were better than anything I had thought of. But the point is, it does require that collaboration. Now, that being said, that’s as a one-off interaction, Pete. I think what you should do in this case, if that’s it, and what anybody should do is, if there’s a routine workflow, like, how often do you title a podcast?

Pete Mockaitis
At least, twice a week.

Jeremy Utley
Okay. So, to me, that’s kind of square in the crosshairs of a task that it’s kind of a creative challenge, probably takes some amount of time. There’s a potential, you know, so there’s, call it, there’s a two-by-two somewhere that you would hire BCG to spit out, right? But you got a two-by-two, and this probably falls in the top right corner in terms of, like, it’s in GPT’s wheel housing capabilities, and there’s enough regularity that it would meaningfully impact your life or productivity. Great. Okay, there’s your two-by-two. I think that that’s a prime candidate for making a GPT.

Pete Mockaitis
I’ll just make a full-blown GPT?

Jeremy Utley
Yeah, why would you not make a podcast-naming GPT? And then you would put in its knowledge documents, all of the titles and your rationale. And then, importantly, it’s not that you make a GPT and you’re done. You make a GPT, then you try it, and then you see where it’s deficient, and you work to get it right, and then you reprogram, you iterate the instructions to the GPT relative to the work that you had to do in addition.

And what’s the process for that? I would say probably you’re going to instruct the GPT, “I want you to analyze the transcript. I want you to find what are the key points of emphasis in the conversation. I define emphasis as we spent more than two minutes on it or whatever,” I don’t know, right? “I want you to find wherever there is more than five back and forth, that’s evidence that this was particularly engaging.”

Or, furthermore, you might develop a protocol where, after your calls, you have a two-minute Zoom call with yourself, where you say, “Hey, here are the four things I thought were interesting.” And you load that into the GPT as well. I don’t know, “Consult the transcript and the follow-up call transcript that I’ve provided for you. Look for these points, then distill these into these brand guidelines, perhaps, or whatever it is. Then do this, then do this.”

You’d kind of walk the GPT through, you would actually articulate and codify that workflow. And then you would test it, and then you’d iterate it, and you’d test it, and you’d iterate it. And you’d get to the point, I would say, probably, if you’re doing it twice a week, by the end of the month, you’ll probably get to the point where, if you really take it seriously to iterate the GPT’s instruction set, over the course of a month, you’ll have something that’s really great.

Now, the problem is most people aren’t really systems thinkers and they just want to do like a one-off kind of like band-aid solution, which is fine. I’m probably more that way myself, unfortunately. So, I’d rather just, it’s less painful on a one-off just to do the work again myself. Systematically, it’s much more painful to do it one-off every time by myself. And so, you kind of got to decide.

And to me, that becomes a function of “What is a task whose output you would refuse to settle for less than exceptional?” That’s a great task for a GPT because you’re not going to be okay with anything less than a really good GPT. And it summons the requisite activation energy required for you to continue to invest in iterating it.

Pete Mockaitis
Understood. Okay. So, it starts with a mindset of, “Okay, don’t talk to it like it’s an oracle. Expect we’re going to need some back and forth, some collaboration, some iteration, some refinement.” And then it’s your bullish take that, at the end of the day, it’s going to cut the mustard and deliver the goods.

Jeremy Utley
Unequivocally.

Pete Mockaitis
All right. Beautiful.

Jeremy Utley
That, to me, is it’s unfathomable that it can’t deliver on that use case.

Pete Mockaitis
All right. You heard it here first.

Jeremy Utley
I mean, really and truly, and I’d be happy to workshop with you if you’d like. But, to me, that is absolutely a use case that GPT can shine with.

Pete Mockaitis
Okay. Well, we talk about use cases. You’re real big on idea flow. It’s getting a whole lot of ideas, a whole lot of creative options generated. Tell me, how do you use AI in that endeavor well?

Jeremy Utley
Well, the easiest thing to do is, which you did well in your example, is request options. I think, for most people, they ask one question, they expect one answer. And with a probabilistic, non-deterministic model, which means LLMs are probabilistic in nature, every time you ask a question, you’re going to get a different answer.

And sometimes the answer is there’s a higher degree of overlap, sometimes they’re radically different, even within the same instruction sets. You could say it’s a bug. I actually think it’s a feature because I believe in variability of thinking is actually what drives creative outcomes. And so, when you realize that, then, “Wow, I could hit regenerate and it will reconsider the question again?” “Yeah.” “Well, why wouldn’t I hit regenerate five times?” Great question. Why wouldn’t you?

And most people go, “I’ve never hit regenerate.” I think it’s actually probably the most important button on the screen. Because you have a collaborator, you and I are going back and forth, and I say, “Hey, Pete, what do we do about this?” You go, “Well, here’s an idea.” And I go, “Okay. Well, what else?” And you’re like, “Okay, let me dig deeper,” and then you say something. I go, “Okay. Well, what, like five more ideas?” And after a while, you’d be like, “Dude, I gave you all my ideas.”

But ChatGPT is not like that. AI is not like that. And so, one of the simple tricks for idea flow with AI is recognizing you’re not going to tire itself out. In fact, you need to recognize your own cognitive bias. I mean, it’s one of my kind of nerd obsessions is what’s called the Einstellung effect, which is the tendency of a human being to settle on good enough as quickly as possible, demonstrated since the 1940s by Abraham and Edith Luchins, where they’ve kind of documented, very clearly how human beings kind of get in a cognitive rut, and they just want a good enough answer, and they don’t actually get the best answer. They just get a good enough answer.

And so, to me, the key to maximizing idea flow with an AI is recognizing that the creative problem in that collaboration is actually your human cognitive bias, not the AI’s bias.

Pete Mockaitis
Okay. Thank you. Oh, boy, Jeremy, I could talk about this forever. But before we hear about some of your favorite things, could you share any other top do’s and don’ts?

Jeremy Utley
One thing, I think, is a really simple thing that you can do, and it’s not unrelated to your idea of asking ChatGPT or whatever, for a number, kind of saying how confident it is. One thing that you can often do is ask it to evaluate its own work, “Scale of zero to 100, how great was the previous response? Be like a tough Russian ballet instructor, give me critical feedback.” And it’ll go, “Oh, it’s a 60 out of 100 for this reason.”

Well, then you could say, “Okay, based on that feedback, can you rewrite it as 100 out of 100? Rewrite it as 110 out of 100. Now, regenerate it. Now, regenerate it again. Now, grade that one. Is it really 100? Bring in another Russian judge. What does the second Russian judge think?” So, one thing that you should definitely do is get AI to evaluate its own work. It’s far better at being objective.

Like, as a simple example for me, and then I also want to mention chain of thought reasoning, so make sure I come back to that. But one thing I’ll do is I’ll do kind of parallel processing between ChatGPT and Claude, and I’m having both work on something. I take their output and I feed it to the other, and I ask, “Which one is better? Is Claude’s work better or ChatGPT’s work better?”

You would think that they both advocate for themselves. They don’t, but they almost always agree. It’s fascinating, actually. There are times where ChatGPT is like, “I actually prefer Claude’s response for this reason, this reason.” And if I go to Claude, it goes, “I think my response is stronger for this, this, this.” And half the time, it’s the other way.

But it’s actually exceedingly rare that they disagree. They often will say the other’s is better, but they almost always agree with the other’s assessment too, which is fascinating, which is to say you can have models evaluate one another’s work. The other thing, the other huge do, probably the single greatest empirically validated finding is that the best way to get better output from an LLM. is to prompt it with what’s known as chain of thought reasoning, which is to say, tell the language model to articulate its thought process before answering.

And so, humans have this tendency, so do AIs, of what we all know as ex post rationalizing. So, if I ask you, “What’s your favorite color?” You say, “It’s blue.” “Well, why did you say blue?” You go, “Oh, well, I like the sky, and I like the ocean, and da, da.” But if instead, I say, “Hey, tell me how you think about what your favorite color is,” and you go, “Well, I probably think about my favorite things.”

And then I go, “Well, what are your favorite things?” You go, “Well, my wife, obviously, and I think about her eye color, and they’re green. You know, green’s my favorite color.” “Well, is it blue or is it green?” Actually, and for me, even as I think through that thought exercise, green, emphatically. I take my wife’s eyes any day over the sunset. That’s a no-brainer, right?

Well, similarly, language models do the same thing. If you ask it for an answer, and it says blue, and then you go, “Why did you say blue, ChatGPT?” it will ex post rationalize. And blue is very subjective, but even with things that are objective, more objective, it will ex post rationalize its answer. If, however, you say, “Hey, before you answer the question, would you walk me through how you’re going to think about solving this problem?” It will articulate its answer and it arrives at, from a research perspective, empirically better, more valid, more cogent, etc. responses.

And the reason it does so is because of how language models work. They aren’t premeditating their answers. So, what it’s not doing, as Pete asks a question, and then it thinks of its answer and writes it out. That’s not what happens. What happens is Pete asks a question and it reads the question and says, “What’s the first word of the answer?” and it says it.

And then it reads your question, and the first word it thought of, and says, “What’s the second word?” And then it reads your question and its first and second word, and thinks, “What’s the third word?” So, it’s not premeditating responses. It’s, literally, only predicting the next token. And so, when you ask it for an answer, the only thing it’s predicting is its answer.

If, however, you ask for reasoning and then answer, it first next token predicts reasoning, and then it incorporates the reasoning that it has articulated in its response, which results in a much better response because it’s not only considered your question, but it’s also considered reasoning first. And as a user on the other side of the collaboration, what that enables you to do is not only, one, get better responses, but, two, you can interrogate its reasoning too.

And you can say, “Actually, it’s not that I have a problem with your answer. I have a problem with how you approach the question. I actually think you should do this.” And then you can guide its reasoning path because you’ve asked it to make its reasoning explicit. Those are the two probably biggest do’s, I would say, when you ask for do’s and don’ts.

Pete Mockaitis
That’s cool. And it sounds like the key is that you ask for it in advance as opposed to, “How did you come up with that?”

Jeremy Utley
Yeah, exactly. That’s ex post rationalizing. It will give you a great answer. It’s a sycophant. LLMs have been programmed to be helpful assistants. And when you realize what that means, it’s a euphemism for suck up. So, if you ask it what it thinks, it’s going to say, “I think that’s a really great idea, Peter.” But if you say, “I don’t want you to compliment me. I want you to be brutally honest. Don’t pull any punches,” like, you got to really ask an AI to level with you to get honest feedback.

When you’re aware of that, it influences how you collaborate with the model, which goes back to the question earlier about idea flow. It’s recognizing your own, I mean, there are limitations to the technology, but a lot of times the truth is we want a suck up. I don’t want to hear how my first draft sucks. I want to hear, “Actually, you don’t need to do any more work. You go have a coffee.” That’s what I want to hear.

And if I don’t realize that the model has been trained to be a suck up, I ask it, assuming I’m getting the truth, and then when it tells me I’ve done great work, I say, “Well, let’s take a break, boys. We’re all done here.” Whereas, if I realize, “You know what, unless I really push it to give me straight feedback, it’s probably going to tell me I’ve done a great job. And I know my human cognitive bias is to overweight the response that I did a great job, and to underweight…” So, you have to understand yourself. In a way, the key to good human-AI collaboration is to really understand our own humanity.

Pete Mockaitis
That’s powerful. Thank you. And now could you share a favorite quote, something you find inspiring?

Jeremy Utley
One is Thomas Schelling, Nobel Prize winner in Economics, who said, “No matter how heroic one’s imagination, a man can never think of that which would never occur to him.”

The second quote that I love is Amos Tversky, Danny Kahneman’s lifelong research partner, who died prior to receiving the Nobel Prize. But Amos Tversky was once asked how he and Kahneman devised such inventive experiments. And he said, “The secret to doing good research is to always be a little underemployed.  You waste years when you can’t afford to waste hours.”

Pete Mockaitis
Awesome. All right. And a favorite study or experiment or bit of research?

Jeremy Utley
I think there’s a great one that I always come back to called the creative cliff illusion, which is conducted by Nordgren and colleagues at Toronto, I want to say. You can look it up, creative cliff illusion. But the basic idea is when they ask participants what their expectations of their creativity over time were, there is an illusion that one’s creativity degrades to a point that reaches a cliff where it almost asthmatically falls off. And people’s, their expectation is, “I’m just going to run out of creative ideas.”

The paper is obviously called the Creative Cliff Illusion because then, when they test people, it’s not true. They don’t run out of creative ideas. They, actually, their creativity persists. And my favorite part of the study is the shape of the creativity, over time, the variable that it’s most highly correlated with, i.e. “Does creativity dip or does it increase?” because it does increase for some people. The variable that determines the shape of your creativity over time is actually your expectation.

So, if you expect that you will keep having creative ideas, you do. If you expect you will cease having creative ideas, you do. And so, that to me is just totally fascinating.

Pete Mockaitis
Totally. And a favorite book?

Jeremy Utley
I love Mark Randolph’s book about the founding of Netflix called That Will Never Work. It’s a fascinating story about entrepreneurship, about grit and perseverance, and about ideas. And there’s a lot of very practical takeaways about the importance of experimentation in finding product market fit and succeeding.

Pete Mockaitis
All right. And a favorite tool?

Jeremy Utley
I’ve got an electric chainsaw, and I love tromping around the woods, just chainsaw in hand, like, just in case I need it. It’s just so fun.

Pete Mockaitis
Awesome. And a favorite habit?

Jeremy Utley
NSDR, non-sleep deep rest protocol, Andrew Huberman. It’s, basically, laying down and becoming totally still, not for the purpose of sleep, necessarily. It’s okay if you do sleep, but it’s not in order to sleep, but to facilitate neurological replenishment, connections between neurons, and codification of memory. And I try, if I can, to NSDR once a day.

Pete Mockaitis
Okay. And is there a key nugget you share that seems to really connect and resonate with the folks; they quote back to you often?

Jeremy Utley
I talked earlier about the value of variation in one’s thinking. And the truth is ideas are naturally occurring phenomena, which is a nerdy way of saying they’re normally distributed. So, you got some really great ideas, very small, it’s a bell curve, right? You got a lot of ordinary ideas and you got some stupid ideas. Steve Jobs called them dopey ideas. He regularly shared dopey ideas with Sir Jony Ive.

Taylor Swift says, “It’s my hundreds or thousands of dumb ideas that have led me to my good ideas.” You got dopey or dumb on one side of the spectrum, you got delightful on the other side of the spectrum. The quote that I often say that people remember and resonates, and they take with them is, I tell people, “Dopey is the price of delight.”

The only way you get good ideas is by allowing yourself to have bad ideas. And the reason most people don’t have better ideas is because they won’t allow themselves to have worse ideas.

Pete Mockaitis
Okay. And if folks want to learn more or get in touch, where would you to point them?

Jeremy Utley
JeremyUtley.design And LinkedIn, I’m happy to chat with folks on LinkedIn. My website, JeremyUtley.design, I’ve got a newsletter folks can subscribe to. I’ve also got an introductory AI drill course where you get two weeks of daily drills for, you know, they say you need 10 hours of practice with AI to start to become fluent. This gives you daily practice to get your first 10 hours under your belt.

Pete Mockaitis
Okay. And a final challenge or call to action for folks who want to be awesome at their jobs? Sounds like we just got one.

Jeremy Utley
To me, it’s very simple. Do one thing you heard here.

Pete Mockaitis
Okay. Jeremy, this is fun. This is fascinating. Thank you. And keep up the awesome work.

Jeremy Utley
Thank you. My pleasure.