281: Making Better Decisions by Thinking in Bets with Annie Duke

Pete Mockaitis
Annie, thanks so much for joining us here on the How to be Awesome at Your Job podcast.

Annie Duke
Well, thank you for having me.

Pete Mockaitis
Well, I’m so excited because I cannot think of any other guests I’ve seen on TV prior to interviewing. Well, maybe Dan Harris, the news anchor but, yeah, you and Dan, that’s it.

Annie Duke
Well, there you go.

Pete Mockaitis
It’s so great to have you here. And so tell us, I noticed when we were kind of getting situated, you mentioned that you prefer to not keep score when you’re playing tennis. What’s the backstory there?

Annie Duke
Oh. Well, so, I grew up in a family where there was a lot of competition going on. My father is a really competitive guy. He, in fact, was quite an accomplished regional amateur tennis player, lots and lots of championships. And I used to play cards with my brother and my father, and my brother was older than I was and so I didn’t do a lot of winning but I really wanted to win.

And then, obviously, when I was playing poker it’s a very competitive environment. When I retired from poker in 2012, I just, I don’t know, I started seeking out situations where it felt much more sort of win-win as opposed to zero-sum. So when I played tennis with people, although they don’t always comply, I ask not to keep score.

I mean, like we’ll keep score for a game because I think that’s just important because there is strategically, obviously, it really matters within a game how you’re playing, but in terms of the sort of overall score, who won or lost, I try not to focus on that too much and try to think about, “How can I improve my game strategically?”

What I usually do with anybody that I’m playing tennis with is I usually drag them to my tennis lessons because I feel like if they’re better then I get better. And the games that I play with my kids tend not be very win-lossy, so like one of our favorites is apples to apples, it’s just sort of hilarious.

Pete Mockaitis
It is. Is the card game hilarious comparisons and so, yes, I do like that.

Annie Duke
Yeah.

Pete Mockaitis
It’s so funny, I thought you were going to say you don’t like keeping score at tennis because it’s so weird. Was it 15 and 30 and 40? I never understood that but maybe it makes somehow to someone.

Annie Duke
I bet the French could explain it. I think that that’s where it started. That is strange. No, yeah, it’s just that I’m sort of thinking about it more as, “What can I do to improve my game regardless of whether winning or losing, it’s the person that’s across the court from me?”

Pete Mockaitis
That’s great. Thank you. Well, so I’m pretty intrigued by your book called Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts. That sounds perfect for what we’re about here, sharpening universal skills for professionals. That’s a big thing that comes up a lot. So tell us, what’s sort of the main idea behind the book and what led you to write it here and now?

Annie Duke
Yeah, thanks for asking me that. So the main idea of the book is that we act like the results of our decisions and the quality of our decisions are really closely linked. So we think that the world is a much more predictable place than it actually is, and we aren’t really acknowledging how much uncertainty there is, that how much it’s the case that you can make really good decisions and have very bad outcomes, because of them you could make very, very bad decisions and have really good outcomes because of them.

And not only do we not acknowledge how much uncertainty there is in that which really stems from sort of two places. One is luck, that even if you make a perfect decision there’s still luck involved in how the future unfolds. But also hidden information is another place where there’s a lot of uncertainty, meaning there’s lots of information that’s hidden from you, there’s just lots and lots of things that we don’t know or can’t know as we’re trying to make decisions.

So decision quality is really dependent on how much are we taking into account luck but also what do we know, like what do we know as we’re trying to make that decision? And we don’t acknowledge or kind of admit to how much uncertainty there is. We’re much more sure of the things that we believe than we probably should be, and we’re much more sure about how the future will turn out than it should be.

And as the title might suggest, one of the things and ways that I think about that is to say, “Let’s start thinking about things through the frame of thinking in bets,” because when we’re challenged to a bet, what it does is it really exposes the uncertainty and whatever it is that we’ve just been challenged to bet on even something really simple.

Like if I were to say, “Citizen Kane won Best Picture,” and you said, “Wanna bet?” All of a sudden, I would step back and say, “Hold on a second, maybe I’m not so sure of that.” It exposes the uncertainty. And if we can expose the uncertainty more, we’ll be better decision-makers because it’s just more accurate to how the world is.

Pete Mockaitis
Oh, Annie, that’s so good and there’s so much I want to dig into there. So you say that we should make a distinction between the quality of a decision and the outcome of a decision. That is very wise and simple and yet I think totally overlooked. Could you expand on that a little bit?

Annie Duke
Yeah, sure. So I think that we kind of – we sort of know when we talk about it in the abstract that you can have a bad outcome as a result of a good decision, and a good outcome as a result of a bad decision. So, I mean, just sort of in the abstract. If I say to you, “When you run red lights do you sometimes get through safely.” You say, “Yes,” which is obviously a good outcome.

And if I asked you if that was a good decision, you would agree, no. And if you run green lights, obviously, you can get to an accident. So, just because, you know, if you’ve been at a bar and you choose to drive after you’ve had one too many, and you get home safely, which is a good outcome, I imagine that you’ll agree with me that that doesn’t necessarily mean it was a good decision.

So in the abstract we kind of get this. The problem is that our behavior doesn’t really get this. So one of the examples that I gave in the book that I actually open the book with has to do with the 2015 Super Bowl. And Pete Carroll’s Sea Hawks, at the one-yard line, The New England Patriots, they’re down by four, it’s second down and they have one timeout.

So what happens there? It’s a very famous play. Pete Carroll has Russell Wilson throw a pass, that was someone unexpected. They were expecting him to hand off to Marshawn Lynch. He throws the pass and Malcolm Butler intercepts the pass in the end zone and, obviously, the game is over. So this is clearly a disastrous result.

But what you see is that people don’t behave as if it’s that abstraction of, “Well, did he run a red light or did he run a green light in order to get to that disastrous result?” Instead, people just announce that this is a horrible decision.

Chris Collinsworth, during the game, really, really slams the decision, and then the next day, in the headlines, it’s really not an argument about whether there was some sort of method to that madness, it’s, “Was it the worst decision in Super Bowl history or the worst decision in NFL history?” Period.

Pete Mockaitis
Those are choices.

Annie Duke
Those are the choices. So, we can get into it if you want but I would highly recommend people go read what, for example, Benjamin Morris wrote about this on Slate, to see that there’s a lot of good analysis that suggest that this actually was a pretty mathematically good decision. Bill Belichick himself, by the way, has said that he agrees that it was a good call.

The one thing I will throw out there is that the chance of an interception, over the last 15 years in the NFL, was only about 1% to 2%. In fact, in that particular situation, zero balls had been intercepted during the course of the season. So I think that we can just sort of have this jumping off point that if an interception was such a rare occurrence that that in itself suggest that maybe the decision-making couldn’t have been the worst in Super Bowl history. And yet everybody acted like it did.

Certainly, the pundits the next day, the newspapers did, most football fans agreed that this was a really terrible decision even though the chances of this really bad outcome were so low. And it seems like you could really make an argument that that was just quite unlucky. So that’s obviously a football example. But everywhere in the world, even though in the abstract we kind of understand this concept that outcome quality and decision quality are pretty loosely correlated.

We don’t act like that. We act like once we know the outcome that we can see right into the quality of the decision and that those two things are very tightly linked.

Pete Mockaitis
Understood, yeah. I think that that’s connecting. And I want to hear maybe sometimes where, so that’s a football example, maybe like in the professional world or careers or executives or business, like where this sort of conflation occurs again and again and it’s bad news.

Annie Duke
Sure. I mean, I’ll give you a variety of examples.

Pete Mockaitis
Let’s do it.

Annie Duke
Right. You hire someone and they don’t work out, “Ugh, I can’t believe. We should’ve known that was such a bad hire.” That would be an example. You launched a product and the product fails, “I should’ve known it. That was a terrible product to launch. Why did I do that? That was so ridiculous.” Here’s the opposite. You hire someone and they turn out great, “I’m so good at that.”

Pete Mockaitis
“I’m a brilliant judge of talent.”

Annie Duke
Right. “I’m a brilliant of judge of talent. You should bring me in for every hiring discussion we ever have or vice versa.” And I gave a pretty detailed example actually in the book following the football example from business, where a CEO that I was working with was really down on himself for what he thought was a horrible decision that he had made about a year prior to my working with him.

And he had just a president of one of his subsidiaries, and the subsidiary was underperforming compared to the market. “What I knew at the beginning was I fired him, and I haven’t been able to replace him and it’s been a total disaster, and it was such a bad decision to fire this president of this subsidiary.”

And I said, “Well, gosh, you haven’t given me enough information to know whether that was actually a bad decision because all you’ve told me is that it didn’t work out. So why don’t you tell me some of the things about the decision?”

And as we walked through the decision, and they were the kinds of things you might imagine. Like I asked if they had worked to identify skill gaps, if they had tried to fill the skill gaps, had they hired a coach in order to really work with the president, had they looked at the rest of the market to see whether they thought there was a good available talent pool to hire from which obviously would be important in that decision.

It turned out that not only had they done all that and thought about those consequences, but then they had actually even considered splitting the job into two so that that president could actually be sitting where their strengths were and then they could hire somebody else into where the skill gaps were. And they had decided that that actually wasn’t a good idea from morale reasons but also just financially, obviously, now you’re essentially paying for two presidents when they thought, given their experience in making high-level hires, that paying for one person would really do.

So it seemed like a really thoughtful decision that hadn’t worked out. And he’d been really beating himself up for a year, and it was affecting his future decisions because he had linked this together so tightly. So it’s a problem that we call resulting and you can see why it’s called resulting. It’s taking the quality of the results and working backwards to the quality of the decision in a way that it’s using it as too much of a signal for that.

And he had been resulting for the last year and it was really affecting his decision-making going forward and put him in a really bad place to sort of morale-wise and psychology-wise. So this is actually a really important problem to be able to sort of address.

Pete Mockaitis
Well, resulting, there is a word, you know, it sounds like a good thing when you say it, “Yeah, results are great, and the root form of that was to be also great.” But, no, no, you’re saying that is kind of a form of cognitive bias or suboptimal mental work there.

Annie Duke
Yes, exactly. You know, I think we think that when we’re results-oriented that that’s a good thing. It turns out that maybe being process-oriented would be a better thing and trying not to be as caught up in the results. And the main reason why is that when we’re analyzing a given decision we’re only analyzing that one decision.

We haven’t run Monte Carlo where we get to sort of do a computer simulation of that whoever we’ve hired or whoever we’ve decided to let go 10,000 times in order to dig down into what the actual answer might be. And it’s very hard to know just from the result whether we’re in a kind of Pete Carroll decision where a one-percenter hit, or whether we actually made a decision which 80% of the time wasn’t going to work out well so that the result was actually quite expected.

The problem is all of that is kind of hidden from view. It goes back to what I said about these two sources of uncertainty, there’s the luck element, and we don’t have a lot of control over that obviously, and then there’s this information element, you know, “What are we taking into account? What kind of information are we gathering in the process of the decision?”

And it’s hard to know after the fact where we’re supposed to sort of lay the blame for or maybe even not blame for the decision for the way that it had turned out on both sides whether it turns out well or whether it doesn’t.

Pete Mockaitis
Understood. Well, so I guess I’m wondering, in your example of the executive, it seems like he internalized something about that decision and how it turned out and what it means for him and his abilities. And so I’d like to get your take on sort of what is the optimal practices or approach to learn from your prior bets and to be sharpened and more brilliant each time you make another bet?

Annie Duke
Well, I think that the first step, I mean, there’s kind of a lot to this so let’s try to unpack it a little bit at a time because I think there’s a lot here. So let’s start with the first step is to change your thinking from viewing decisions as right or wrong in the first place. So let’s start with the idea that right and wrong are kind of a bad construct through which to look at the world.

So what do I mean by that? Well, you know, there’s some luck and there’s some skill involved in every decision and even if we make a decision that’s pretty good, maybe there’s a better decision to be made. So the second worse decision isn’t wrong. You know, it’s better than the worse decision that you could make, and the second best decision isn’t necessarily right.

And even if you make the best decision that’s available to you, in the future you might find out that there was other information that was available that could’ve even made that decision better. So once we start viewing things through this idea that we’re kind of always in this under construction or in progress phase, what happens now is that when we now get an outcome that’s either good or bad, which is another place which where maybe we can understand that outcomes aren’t 100% good or 100% bad, right, so we can do this here as well.

But once we get an outcome and we kind of know the quality of the outcome, we’re much less likely to use that to sort of make some sort of categorical claim about the decision, and rather take it as evidence or as an impetus to go in and examine the quality of the decision that we understood was in progress or under construction in the first place.

So, and I think that one of the best ways that we can see why this is valuable is in terms of beliefs. So beliefs are always informing the decisions that we make. So, I mean, if we think about the CEO, he had beliefs about what the talent pool looked like if he was going to go out and try to recruit somebody into that job as an example, right? So that would be the kind of belief that he might have.

He had beliefs about whether, after what he had done, the skill gaps that still needed to be reinforced were able to be sort of patched up for that CEO. So there’s all sorts of beliefs that inform any decision that we have. And if we view our beliefs as right and wrong as opposed to in progress, as we’re just trying to move toward an accurate model of what is actually the objective truth, then what happens is that it becomes very difficult to be open-minded to new information and also to be hungry for new information because you’re just either wrong or right.

So if you’ve got yourself in a category of right there’s nothing new to be learned and you’re not going to be open-minded to other people who might hold a different point of view because naturally they’re wrong. And if you’re wrong, you might completely reverse a belief where parts of that belief or some percentage of that belief was actually a pretty accurate view of the world.

So I like to think about wrapping that uncertainty in as the first step to trying to solve for this because then you aren’t as hungry for the answer in the first place. It’s more like, “It didn’t work out. Hmm, that’s kind of an interesting data point. I can bring that back in to look at the decision.” As opposed to, “Well, that was just terrible. I should’ve known. That was awful.” And now all of a sudden you’re just reversing something completely that shouldn’t be reversed in the first place.

So that would be the first place that I would start.

Pete Mockaitis
Well, that’s interesting. So it’s not like we’re not open and shut and done with it having seen a good or bad or spectrum of optimal to slightly less optimal, you know, outcome. But rather that is sort of a starting point, “Interesting. We have one data point and we’re going to dig in further,” rather than saying, “Okay, that’s that.”

Annie Duke
Right. And I think that you can think about it this way. because in places where we really know where the luck sits, I think it becomes easier to reason around this. So we want to think about our outcomes more like we would think about the outcome of a coin flip that if you have 10,000 coin flips there’s a lot that you can say about the coin, that it will land heads 50% of the time it will land tails 50% of the time.

But if I flip a coin once, and I called tails, and it lands heads, hopefully you aren’t saying that I’m really bad at calling coins. Likewise, if I flip a coin and I call heads and it lands heads, I hope that you’re not saying that I’m really good at calling coins because it’s only one outcome. And for most of the kinds of outcomes that we have in our lives, we can’t collect 10,000 of them. We only get to have a few, a handful of them.

I mean, for some things like choosing your partner. Hopefully, you’ve only done that, you only do that once. But even so, like the craziest person it might be four times which still isn’t enough to tell very much about whether you’re actually good at that or not.

You’re generally not making 10,000 hires into the exact same position in your whole life as a, you know, if you’re in HR, that kind of thing. So we want to be careful about how much signal we’re taking from an outcome in the first place. And, again, this is a place where in the abstract people seem to be better with this. When I describe this coin flip problem, people say, “Yes, of course, it would be silly for me not to acknowledge the uncertainty in a coin flip, and to think that just because it landed heads one time or tails one time that I’m supposed to know something more than that was just luck.”

And we want to try to treat our life’s decisions and our life outcomes a little bit more like that.

Pete Mockaitis
Well, Annie, now you got my wheels turning a little bit because, you know, there is a decision that I make again and again, I’ve made about 300 times, well, 300 times in the affirmative, and that is, “Which guest comes on the show?” And you made the cut, Annie. Nice work.

Annie Duke
Well, thank you.

Pete Mockaitis
Yes. And so I guess I’m thinking, “Well, hey, I am making this decision hundreds of times.” So we talked a little bit about the mindset associated with not being binary and trying to be curious and interested and to learn from it. So if you do find yourself in such a position, what are some best practices to keep getting better and better at it?

Annie Duke
Well, I think it’s to be open-minded to try to learn from what’s working and what’s not and understand that whenever you decide that it’s always going to be a working hypothesis, so that when new information comes in that doesn’t necessarily conform to your working hypothesis, that you just don’t reject it as not actually being informative.

So let me try to explain what I mean by that. So let’s say that you have guests come on the show, and you have a guest come on who does analytics. Right. Okay. And he’s a terrible guest. So you might now have a working hypothesis that people who are in analytics are terrible guests. So that’s what you don’t want to do, is say, “Well, now, I’ve just made this decision and so I’m not going to invite anybody who’s in analytics anymore because I’ve now made this hypothesis based on the one guest.”

What you want to be, as much as possible, be open-minded to what are the qualities, what are the actual qualities in the guest that actually make them great on the podcast, and what are the things that are signals that maybe they might not be such good guests. And while you might have working hypothesis about why that is, you should always be trying to do some A/B testing, right, and try to disprove yourself, number one, right?

And number two, be open-minded when someone doesn’t fit the mold that maybe you need to re-jig your model. So I think what’s important, even when you do have 300 guests because you are choosing them one at a time, is to make sure that you don’t become rigid in whatever your hypothesis is. Don’t think that the facts that you’re observing and the way that you’ve modeled it is an answer like two plus two equals four as an answer.

Again, always treat it from that standpoint of uncertainty that you have a hypothesis, a working hypothesis that is in progress and under construction so that you are much more open as you have new experiences that might inform your future choices.

Pete Mockaitis
That’s good. And you know what? I like that notion of being open-minded and humble because I’ve been surprised both ways, you know, of my guests, I mean, I thought that would just be smash hits in terms of his downloads, if that’s the outcome we’re measuring, and they weren’t, and those who I thought, “Okay, you know, that was a good chat. We’ll see,” who really were record-breakers. And so it’s fascinating to see, and I think that’s useful, I think just to realize, “Huh, I am wrong a lot. How about that?”

Annie Duke
Well, let me just suggest something that maybe you’re not wrong.

Pete Mockaitis
Okay. Thank you. There you go. I did it. I did it. I’m guilty.

Annie Duke
You did it. See? So let’s think about it this way. Maybe, given the information that you have and the past guests that you’ve had on, and the categories that you know that your audience really enjoys listening to, and the information that you had about the guest, you made literally the best decision that you could in inviting them on. And it just happened not to work out.

Because I think that you can agree that even if you have the best information, you know, the most information that you can, and a lot of experience, and you’ve accumulated a lot of experience and expertise in inviting a guest on, that it’s never 100% sure that that guest is going to be a hit. There’s always some percentage of the time that it’s just going to turn out that they aren’t what people wanted to listen to, or there wasn’t chemistry between the two of you perhaps. Maybe they were a great guest on a different podcast but it didn’t work out with you or vice versa.

And so saying, “Ooh, I was wrong.” I’m not sure that that’s constructive, instructive rather, right? Maybe it’s, “Huh, that was really interesting. Let me think if there was some sort of thing that I could’ve seen that would’ve helped me come to that conclusion.” And if there wasn’t, well, then maybe you just had bad luck.

Pete Mockaitis
Yeah. Well, Annie, this is so humbling and powerful, I’m like meta in real time. It’s like we just talked about the right and wrong thing, and here I am going right there. And so I think that’s just illustrative for me and hopefully for listeners, not so much that, “Pete is a moron and who doesn’t listen.”

Annie Duke
No. No.

Pete Mockaitis
But rather, we have it really ingrained in us, this right and wrong notion. It’s just like our natural default setting.

Annie Duke
Well, I think that this brings up a really good point. So, you know, I’m listening, too, and, of course, I heard you do it, and so I said, well, maybe what you can do is if a guest doesn’t work out, instead of saying that you were wrong, try to figure out if there was something more that you could’ve found out about them that would’ve helped you to have predict it, right? So try to sort of figure out what you could’ve learned from that experience as opposed to calling yourself wrong.

Now I don’t know for sure but you may catch me saying wrong or right as well. And the reason is that this stuff is really hard. Just as you said, we’re sort of hardwired into this very black and white thinking, we’re very hardwired to connect outcomes to decision quality, to declare ourselves or somebody else wrong, or ourselves or somebody else right. And it’s really, really hard to overcome particularly on your own. It’s hard to spot in ourselves.

The good news is, and this is sort of again getting down and to digging down into that question you asked about, “How can we help make this better?” is that we’re pretty good at spotting it in other people. So it’s not surprising that I spotted it in you. It’s easy to see that. If I had made the same declaration, “Oh, well, I’m wrong all the time.” You’d say, “But wait a minute, Annie, you just said why you’re thinking about things as right and wrong?” because you would’ve spotted in me immediately as well.

So that’s the hint is that on your own it’s very hard to overcome these biases. The science is pretty strong on that that it’s hard to overcome on your own. But in groups, now that’s a different story. In groups we can really help each other out because we can watch each other’s information processing and decision-making facts, let’s put it that way.

Pete Mockaitis
That’s good. So when it comes to groups do you have a couple of tips on best practices to leverage that well? Because I guess group think can still happen where don’t need any of that benefit.

Annie Duke
Yeah, so we definitely want to be careful of creating a situation where the group is essentially ourselves on steroids.

Pete Mockaitis
You’re so smart, Annie. I love every idea you have.

Annie Duke
Thank you. But that’s kind of, we know, gosh, I mean, in looking at what’s happening in politics right now, we know that that is our natural tendency is to kind of go to this confirmatory style of thought. So it does take some intention on the group’s part.

So here’s number one. Make sure there’s three of you at least not two. And the reason why you want three of you at least not two, is that, I’m going to steal this from Phil Tetlock who wrote an amazing book called Superforecasting which I would highly recommend to anybody. You want two to disagree and one to referee. So it’s very helpful to have a referee involved. So try to get three. If you can get more that’s better.

And then what you want to do is actually explicitly make an agreement with the people in the group that you are going to interact with each other in a way that actually goes against what the normal, the kind of social norms are, the way that we normally interact with each other.

So normally we sort of want to be team players and we want to build consensus and we want to be agreeable, right? We want to be people who aren’t the naysayers. And, obviously, when we do that, that creates this kind of, that’s much more likely to create this kind of echo chamber-y group think kind of thing going on.

So here’s what I suggest is. As an agreement within the group, you agree to three things. Thing number one is that your goal is going to be accuracy. Now, what does that mean? That you are saying, “As a group we’re going to help each other to work on the focus being less about being right and more about being accurate.” What’s the difference between the two because they sound obviously the same?

We can think about being right as, “I have these beliefs, I think that the world is a certain way and I’m going to reason about the world in a way that’s just going to confirm what I already believe.” So you can think about it this way. Like if you have a very strongly held political belief that you’re noticing all sorts of things that conform with the beliefs that you have, you tend to be reading news sources, or watching television channels, news stations that already agree with you. So your views are being reinforced.

And then on the flipside, you’re not actually noticing or seeking out information that disagrees with you. And if you’re confronted with the information that disagrees with you, you’re pretty good at trying to discredit it.

Pete Mockaitis
“Now, that research was funded by, you know, someone who is biased so we can’t trust it.”

Annie Duke
Exactly.

Pete Mockaitis
“I’m going to dismiss it wholeheartedly.”

Annie Duke
Right. Or, “I know this person is really biased in their thinking so, clearly, they’re not telling me the whole story.” And what is that? That’s really saying, “Well, that person disagrees with me so I’m just going to say that their opinion is invaluable because of that.”

And this is the way that we tend to process the world, right? So if you’re on the Liberal side of the aisle you tend to be watching MSNBC. If you’re on the very Conservative side of the aisle, you tend to be watching FOX News, and that’s kind of where you stay. And if you did go and take a peek at MSNBC, you would figure out all the ways that what they were saying was biased, and if you did go to take a peek at FOX News, you would figure out all the reasons why what they were saying was biased. And that’s just kind of the way we process the world.

And it’s true certainly outside of politics as well. When we have a particular strategy about how we think that marketing should be done, people who don’t think the same thing as us we tend to be very dismissive about. And vice versa, we think they’re wrong. So we aren’t very open-minded to things that don’t conform with the beliefs that we already have. So that’s reasoning about the world to be right.

The reason why someone who’s Liberal is watching MSNBC is because they want to hear that they’re right. The reason why someone who’s Conservative is watching FOX is because they want to hear that they’re right. So that’s reasoning about the world to be right.

Reasoning about the world in a way that’s trying to be accurate means we’re going to acknowledge. So let’s say we’re in a group together and we’re trying to form this agreement. We’re going to acknowledge that there is some sort of objective reality, there’s objective truth, and that we’re both trying to work together to construct the most accurate view of what that objective truth is.

Now, we could think about this if we go back to this idea of thinking in bets. When we think in bets, when we create that consequence, that downside consequence to having a belief that is not accurate, it focuses in on the accuracy piece. Because the person who wins in a bet, if we’re betting against each other, the person who has the most accurate view of the world, beliefs about the world, is going to win against the person who just wants to believe that the things that they believe are true.

So what happens is that, like if I were to announce, “Citizen Kane won Best Picture,” and you said to me, “Do you want to bet?” It forces me to focus on the uncertainty, it forces me to focus on what I don’t know, “What do you know that I don’t know? Why might be I wrong?” And those are all questions that have to do with accuracy. “Is this view accurate?” And it forces me to pull out Google.

So that’s the first thing that we’re going to agree to in the group is that if I hear you reasoning a way where I think you’re just reasoning to be right, or you’re being overly critical of views that disagree with you, or too accepting of things that you read or say or hear that do agree with you, I’m going to call you on it, and I’m going to say, “Hey, why might you be wrong? Like, why do you think maybe you might be biased?” So we’re going to have a commitment to accuracy. That’s number one.

Number two is we’re going to be really tolerant of diverse viewpoints. So when we’re going to seek out sources of information that disagree with us, we’re going to agree to discuss them. if you disagree with me, I am not going to dismiss you. I’m going to work my best to not be defensive when I hear it, and I’m going to try to figure out what it is that you might be right about that I hadn’t considered.

What are the things that will help me construct a better view of the world? And that doesn’t mean that if I have some beliefs that disagrees with you that I go from 100% sure of that belief to 0% sure of that belief because you happen to disagree with me. And maybe that I was 80% sure that my belief was accurate, and now because of things that I’ve talked to you about, I’m now 72% sure. I’ve just moderated the belief in some way, which is most of what you’re going to do.

So you try to allow dissent into the equation and you agree that dissent is going to be okay in the group that just because I disagree with you doesn’t mean I don’t like you. And, in fact, that my disagreement with you is helpful to you because it’s going to help you to create an accurate view of the world. So we have accuracy and dissent. And then here’s the third piece that is really important. We’re going to hold each other accountable to visiting in that way.

Pete Mockaitis
Okay.

Annie Duke
So you know that if I hear you say something that’s biased, or if I hear you say, “Wow, I had this guest on and they were terrible. I was so wrong to invite them,” then I’m going to call you on it, and I’m going to say, “Well, are you sure that you should be thinking about that as wrong?” And that’s going to be okay and I’m going to hold you accountable to that kind of thought.

Pete Mockaitis
Oh, that is powerful. I love it. And like when you put it in those terms, in terms of we’re going to establish this agreement upfront, we’re going to kind of – it’s like a pact or an alliance that we’ve designed, this is how we work, how we roll here. It seems like it almost turns the whole thing into a fun game as opposed to – it just kind of removes the whole lot of like the tension and the politicking and the jockeying for approval. It’s like that we’re all just playing a game called, “Let’s get super accurate and sort of have fun with it,” you know.

Annie Duke
So I love that you framed it that way. You’re a man after my own heart with that because I think that it gets down to kind of what the secret sauce is here, and I think that you’ve really pinpointed it in a really great way.

So why is it that we want to affirm the things that we already believe? Why is it that we reason that way in the first place? Well, the things that we believe, these beliefs that we have are really the fabric of our identity. And as we sort of look over the history of our lives, and this is a lot of the way that Daniel Kahneman really thinks about this. Another book recommendation that I would give is Thinking Fast and Slow. I think it’s a great overview of this kind of where biases and heuristics get in our way.

But he really talks about it through this lens of we want to have a positive self-narrative. We want to think that we’re smart and we’re competent and we’re good actors and we’re good people. And a lot of that means that we want to feel that the things that are part of our identity are right, that we weren’t wrong about stuff. That doesn’t feel good, it feels like a downgrade in our self-narrative.

So a lot of what’s driving this information processing and the way we process information is because we want to feel good about ourselves. We just kind of want to feed our ego. And the way that we’re sort of feeding our ego and feeling good about ourselves, getting that positive update to our self-image is like, “Oh, yeah, that thing I believe was totally right.” And that makes us feel really good.

So what the group does, because it forms this, it creates a different goal to the game. Like now the goal is, “I want to be the best mistake admitter, because if I go to you and I say, ‘Oh, I really think I messed this decision up. Let’s talk about it,’” that’s when you, because we have this agreement, your face is going to light up and you’re going to give me all sorts of social approval for doing that because you’re going to recognize that I am executing on our charter in the best possible way.

In a way that’s so much better than me coming up to you and saying, “I made this brilliant decision. Let me tell you all about it.” And you’re going to be like, “Oh, that’s not really part of the charter.” Right? But if I go to you and I say, “Man, I really think I made a mistake,” you’re going to be like, “Yes, let’s totally talk about that,” and that’s now going to be what makes me feel good. So it shifts what the rules of the game is. It’s like how are you keeping score? How do you get a point?

The way we come into the world is I get points for just like, “Oh, that thing I believed is really true and I’m so smart.” Once we form this group and we work together, now I get points for mistake admitting, and credit giving, and belief calibration, and changing my mind. That’s now what I get points for, and that’s what I feel good about, that’s now what contributes to my narrative.

Pete Mockaitis
Oh, this is powerful. This is transformative stuff. Thank you. I’ve actually a couple of things I want to touch before we have to conclude or hear about your favorite things. And so one of them is intuition.

Annie Duke
Sure.

Pete Mockaitis
We talked a lot about being rationale and collecting data and refining and being predictively accurate. Well, you know, you are a World Series, a Poker Champion. I imagine you’ve relied on your gut a time or two in gaining this accomplishment. How shall we be thinking about intuition and to the extent we can trust it and play that game?

Annie Duke
Sure. So the way that I think about intuition is that intuition is an incredibly useful tool but it should never be a reason for doing something. So let me explain what I mean. A lot of the decisions that we make actually have to rely on intuition because you don’t have time to map out some kind of probably realistic decision tree.

Here’s an example of a time it’d be really good to use intuition. You’re driving along the road and a deer jumps in front of your car. I hope you’re not taking a whole lot of time to think about it, and you’re just using your gut to figure out what you’re supposed to do there. But, certainly, at the poker table you have to make a lot of quick decisions.

You’re in a sales meeting and you sense that it’s going south and so you make a judgment call right in that moment that may be off script in order to get it back on track, right? And that’s really, really important. Like we need our intuition for a lot of decisions because we just don’t have time in order to actually go through a deliberative process for a lot of the kinds of things that we have to decide, so let’s be very thankful for intuition.

However, what’s really, really important, I think, is to make sure that intuition gets held accountable to a deliberative process in the same way that I want to be held accountable to you. So that means that when you ask me later, “What was it? Why did you change strategy in the sales meeting?” You should not accept from me as an explanation, “My gut told me so.” That’s not enough of an explanation, right? That’s just a cop-out.

I should be able to tell you what was it that I saw or felt that made me think that I needed to change course, because if I can’t properly explain that decision to another person such that they could understand it and execute it itself, themselves rather, then I should be questioning the intuitive response itself.

So I need to be able to explain to you after the fact, “Well, now that I think about it, you know, this is what I saw and this was what his body language was,” or, “This is what she seemed to be doing where it felt to me like she was about to let the deal break,” or whatever it might be, like I should be able to explain that to you in retrospect. And if I can’t, if the only thing that I can tell you is my gut told me so, then I should go in and re-examine my intuition.

Because let’s think about what intuition is in the first place. Intuition is like the whole of your life experience informing some sort of a gut reaction that isn’t being driven by a conscious process but it’s certainly being informed by all your past experiences. So what I want to do then is make that accountable to a deliberative process so I know, so I can create a new experience to then further inform and refine my intuitive or gut response.

One of the best ways to get intuition to line up is actually to teach, and that’s sort of what I’m asking somebody to do there. If I’m explaining, if you’re demanding from me that I explain why I did what I did, you’re asking me to teach you why I did it. And I could tell you from poker that I started teaching poker in seminars sometime around like, I want to say, 2004 or so.

And when I started teaching, my game really changed. All these things that I’ve been doing intuitively that, you know, I’ve been working okay for me, but I hadn’t really thought about it in any kind of explicit way. As I went to try to teach them in a seminar, there was a certain set of those things which I realized, “Oh, I can’t actually really justify that. I can’t really explain that very well to these people so that they could go do it themselves.”

And then I went in and re-thought what my gut feeling was about what I was supposed to do in those situations, then I realized there was actually a much better answer, and my game ended up changing for the better through that teaching process and it was because I was refining my intuitive or gut responses.

Pete Mockaitis
Oh, well said. Well said. And I’m intrigued, this is like a smidge off topic, but like when it comes to body language or tells or, I guess, that’s a whole another episode.

Annie Duke
Yeah.

Pete Mockaitis
You know, is there anything that you think that we humans pick up on a lot from other people and that’s totally valid and we should just, you know, articulate it explicitly and say, “Yeah, that’s real and that counts”?

Annie Duke
Well, you know, so there are a lot of things. I would highly recommend people go and look up a guy named Joe Navarro on Amazon.

Pete Mockaitis
Oh, yeah, What Every Body is Saying.

Annie Duke
Exactly. You will find a lot of stuff about body language in there that is going to be articulated much better than I ever could. He is brilliant and it’s an amazing book if people should just go pick it up. But, you know, there’s lots of things that have to do with – I mean, we’re pretty good at spotting deception in general.

Sometimes we talk ourselves out of it because we want to believe and we want to trust but I think that the signals are pretty strong for those kinds of things. You know, spotting discomfort, spotting openness. When somebody is really open in a conversation I think that it’s very easy to spot. In fact, it’s really interesting. The next time you’re in a restaurant and you see people who are clearly like on a date, you should be able to tell pretty quickly whether the date is going well or not by whether they’re leaning into each other or not. So that would be a really strong sign of comfort for example.

So I think that we’re pretty good in understanding these things, and you can understand from an evolutionary standpoint why that might be. You know, we need to know whether someone is an enemy or whether someone is about to attack us, or whether somebody is a friend. And so I think that these signals can be very strong and we can train ourselves to really pick up on them which is incredibly helpful obviously. So go read Joe Navarro. He’s much better at that stuff than I am.

Pete Mockaitis
All right. Thank you. And I also want to get your take, you know, we talked a lot about uncertainty and sort of probabilities and refining the model as you kind of learn more and integrate that. So I think that, at the same time, you know, there is an irrational force called fear. We’re just like fundamentally uncomfortable with uncertainty even though it’s all around us, it’s there. We don’t like it and we sort of the devil you know versus the devil you don’t, a lot of us will default to the devil you know. So any pro tips on kind of pushing past fear or getting comfortable with uncertainty?

Annie Duke
Yeah, what I would say is that I think that the more that you actually write down the decision and the process and get other people involved with it, so work through these decisions that you’re afraid of because of the uncertainty on the other end. Really map it out, you know, “There is decision A that I can do, there’s decision B, there’s decision C. I’ve got to choose among these three. Let me really think about decision A and what the scenarios are that I think might result from decision A.”

Really think about it not just by how might it work out but really dig into that how might it go wrong piece. Get those scenarios setup, take a stab at what the probability of each of the outcomes is, and just take a stab at it. You’re not going to be perfect but taking a stab is better than pretending like it’s 0% or 100%. I mean, it might just be a range, “Well, I think this will happen 30% to 50% of the time. Well, that’s okay because that’s better than 0% to 100% at a time.”

And it starts to get you to be comfortable with that uncertainty. So do that for each of the decisions with other people. Write all of that down. Look across that. Decide which decision you think is best. And realize that no decision is still a decision so you better map that one out, right? So you better map out sticking with the status quo, “And what do I think the outcomes are from sticking with the status quo?” And treat it as if it’s a new decision. That’s a big mindset shift.

Because I think that one thing that causes people to stick with the status quo is they aren’t treating it like a new decision. So if they’re not thinking about it as a decision, then if it doesn’t work out it’s like, “Well, I didn’t decide for it not to work out. That wasn’t my fault. I didn’t do that.” But treat it like it’s a new decision.

Now, memorialize all of that, figure out what you’re going to do, and take that and put it on a whiteboard, and just leave it there with all of the probabilities that go along with it because now whatever way that decision turns out, it’s up on the board. You’ve already thought about that in advance. And I think that’s the best way to start getting comfortable with it and it’s really just ripped from the pages of the poker world.

When I make a bet, I understand that there is some chance that you call, there’s some chance that you raise, or some chance that you fold. And, hopefully, I’m making decisions that are going to get me the best outcomes in the long run but I know I’m going to lose a lot of the hands. I have that kind of wrapped into the decision process upfront because I’m considering those scenarios in the first place.

And I think that’s what allows you to get it. In a weird sense, being afraid of uncertainty comes from not facing it down. But if you just go ahead and face it down you won’t be so afraid of it.

Pete Mockaitis
I love it. Cool. Well, Annie, tell me, anything else you really want to make sure to highlight before we shift gears and hear about some of your favorite things here?

Annie Duke
No, I think we’ve covered a lot. I mean, I’ve really enjoyed the conversation. I think we’ve gotten to a lot of great places.

Pete Mockaitis
Oh, me too. Thank you. Well, I’m going to be chewing on this in the future, I’m sure. So, for now, can you share with us a favorite quote, something you find inspiring?

Annie Duke
Yes, so I find that my favorite quotes tend to change like weekly, and it kind of depends on what I’m sort of thinking about at the time. I got this quote, I’ve actually been thinking about this quote this week which is from Heinz von Foerster, which is, you know, an obscure person, the quote is, “Objectivity is the delusion that observations could be made without an observer.”

And I’m really loving that quote because I’ve been thinking a lot recently about how people use data to make their case rather than to find the truth as if data is somehow some objective thing that exists in the world independent of human beings, you know, collecting it, analyzing it, interpreting it. So this is something that’s been on my mind a lot.

It’s actually in my newsletter this week because I’ve been thinking about it, and so I found this quote, and so this quote is my favorite quote right now and it will be different than my favorite quote likely in a week or two. But that’s what it is right now.

Pete Mockaitis
Thank you. And how about a favorite book?

Annie Duke
So it depends on what category. You know I have this trouble when people ask me for my favorite movie as well because there’s a lot of them. So I have favorite books that are like Animal Farm and Catch-22 and Lolita, books that are kind of in that fiction category that are these really, you know, Slaugtherhouse Five is another one, that are books that I read when I was young that had humongous impact on me. And I still consider those among my favorite books.

And then there are books that I sort of put in the category of things that I think about in my intellectual sort of the work that I do and in the decision-making world, and those kinds of books are things like Thinking Fast and Slow or Predictably Irrational or Superforecasting or The Power of Habit. And you can go lots of places if you just start with those four. They’ll lead you into a web of thinking about things in this space.

So it’s always hard for me to choose because I really think about things in different categories and what part of my soul they’re informing or feeding, I guess.

Pete Mockaitis
Well, we’ll take them all and link them all. Thank you.

Annie Duke
Yeah.

Pete Mockaitis
And how about a favorite habit, a personal practice of yours?

Annie Duke
So I would say that one of my favorite personal practices is that I really do, as a practice, try to shut everything down by dinnertime in terms of my work life. I am really, really committed to time with my family and time with my partner, and it’s important that I get to spend as much time with him as possible and as much time with my kids as possible.

And I think it’s really easy to lose that balance if you don’t set really strict guidelines around it. So once it’s time for me to start making dinner, it’s family dinner, and no work after that. As much as I possibly can. Now, it bleeds in a little bit but I would say that just setting that in motion makes it happen so much less because I have to kind of break a contract with myself so it needs to be really, really pressing and really important for me to allow that to happen. And I think that the interesting thing is because I do that I think I actually get more done rather than less.

Pete Mockaitis
Yes. Thank you. And, tell me, when you’re sort of teaching some of your area of knowledge here, is there a piece that really seems to kind of resonate and get folks nodding their heads and agreeing and quoting yourself back to you?

Annie Duke
I’m going to paraphrase myself here because I don’t have the exact quote in front of me but it’s something that I said in a piece for Nautilus that I have said elsewhere, and this is the thing that I’ve seen quoted back the most, which is, “We should spend much less lot time blaming ourselves from bad outcomes but also much less time patting ourselves on the back so hard for the good ones.”

Pete Mockaitis
Excellent. And, Annie, if folks want to learn more or get in touch, where would you point them?

Annie Duke
So a couple of places. One is I’m very active on Twitter so you can see me at @annieduke on Twitter. I post lots of content there. I have a website annieduke.com where if people want to hire me or just get in touch with me there’s a Contact form there, and I see all of that and I’m happy to respond to any hiring request or just a question, if you have a question go there.

You can also find archives of my newsletter on there. And if you like it, you can subscribe to my newsletter on my website. My newsletter comes into your inbox every Friday and it’s really kind of discussing these issues as they apply to things that are actually happening in current events either in the political world or the business world or in science.

So it’s a way to kind of get an idea of how to apply these kinds of concepts and ideas and practices in the real world. Plus, I always put a fun visual illusion in there too because I’m a big fan. So those are the best ways to get in contact with me.

Pete Mockaitis
Oh, cool. And do you have a final challenge or call to action for those seeking to be awesome at their jobs?

Annie Duke
Yes, I do. My call to action is that people go and look at whatever their social media feed is that their favorite, either their news feed, their Google news feed, their Twitter feed, whatever it might be, wherever you’re getting your news, and go look and see if it’s balanced. Try to figure out, “Am I only on one side of the aisle here mainly?”

And really, really try to make sure it’s 50-50 and read from both sides of the aisle not with the intention of reading from that side of the aisle that you are not on to see why they’re so wrong. But try to make it so that you are really focused on trying to change your mind about one thing every day.

Pete Mockaitis
Oh, perfect. Well, Annie, thank you so much for this. This is powerful stuff and I’m looking forward to, hopefully, having more and more great outcomes from great decisions but not, you know, directly ascribing one is the result of the other, and falling for some of those errors. So this was a ton of fun and I wish you lots of luck in all you’re up to.

Annie Duke
All right. Well, thank you very much for having me on. I really enjoyed it.