1133: The Philosophy of Scores: How to Measure What Truly Matters and Stop Playing Someone Else’s Game with C. Thi Nguyen

Pete Mockaitis
Thi, welcome!

Thi Nguyen
Hello. Hello.

Pete Mockaitis
Well, I am so excited we have a learned philosopher here that, I think, the first time we’ve had a full-blown, legit, credentialed philosopher on the show, and I’m stoked.

Thi Nguyen
I’m sorry, I’m about to under-meet your expectations.

Pete Mockaitis
We’ll see. Well, your book, The Score, I found absolutely riveting, fantastic, plowed through it. And I’ve got one burning question I want to address right away, which was, you mentioned you loved just about every game you’ve played, or rather, more pointedly, the one game you regret playing was the computer game Civilization. What’s the story here?

Thi Nguyen
I think one thing I realized, one of the, actually, origin points about thinking about this, about what kinds of things that we did in our lives are worthwhile and not, is that, I don’t know, I spent a whole summer playing Civilization and I just have this memory of a vague anxiety sweat blur of like nothingness.

And when I think about other times I’ve spent with other games, like everything from rock climbing to Go to like really interesting video games, like, Baba Is You, I have this thick memory of how many interesting things happened, how many things I did. And Civilization is just, I don’t know, that time is just gone.

Pete Mockaitis
Interesting. So this kind of reminds me of the distinction between liking and wanting. It sounds like if your whole summer went away, you had a whole lot of wanting, you had to get back to it, see what happened with your trade routes or your armies or your whatever. But there wasn’t a rich memory that felt uplifting afterwards.

Thi Nguyen
Yeah, I also think, one of the things that exposed to me is something that I think we can be wrong about whether we’re having fun or having a good time. And I’ve had this in relationships, friendships, and games.

And I think my favorite example of this is my experience with the first Star Wars prequel movie, which I was so excited for at the time, in which I convinced myself that I had a great time at. And I spent a week being like, “That was awesome.”

And it took me a week to realize that I was talking myself into thinking that it was valuable, that I kind of overwritten the actual experience with my need to think of it as a valuable experience. And I think that’s actually something that can happen in a lot of places.

And I think when I play Civilization, I tell myself, “This is fun, I can tell. Like, it’s a good game, I’m into it.” But, actually, afterwards the time seems valueless.

Pete Mockaitis
Oh, yeah. I think that’s so true that we can talk ourselves into it. For any number of reasons, we want to think we haven’t been duped, we want to justify the money, the time we’ve invested. And, yeah, I think we totally have the capacity, a great capacity for self-deception.

Well, let’s get into things a little bit with regard to The Score: How to Stop Playing Somebody Else’s Game. Could you maybe tell us a little bit of the backstory for how these ideas got swirling around your head and seemed like a worthwhile thing to spend years researching and putting together some great writings on?

Thi Nguyen
So, a bunch of this comes, I’ve spent a lot of the years of my life working kind of an insane path through philosophy. I’ve been obsessed with a part of philosophy that I think isn’t even supposed to be part of my field, which is the philosophy of games.

I’ve been trying to figure out what games are and why they’re valuable. And this started way back when, because I was reading a bunch of stuff with some students about video games. They wanted to know whether video games were art, and we read about, you know, a bunch of articles about it.

And most people were trying to say that video games were art because they were like a movie. And I was like, “Yeah, some games are like a movie. Some games have cool cutscenes.” But, actually, the thing that I really care about is how it feels to be playing, the fact that I get to make interesting decisions or interesting movements.

And when I was trying to figure this out, I found this moment from my favorite game designer, Reiner Knizia, he’s a German board game designer. And in one of his talks, he says that the most important part of his game design toolbox is the scoring system because it tells the players what to care about. It sets their desires.

And that, I think, is the moment where I was like, “Oh, this is so interesting. This is even more interesting than I realized.” Because games, I think they don’t just create worlds, they create alternate selves for us to plunge into.

And so I wrote an entire book about the beauty of games and how game designers use scoring systems to push around our motivations, to give us alternate ways of caring, and to create all this incredibly beautiful, rich action.

And then I started worrying about gamification because I think a lot of people were saying like, “Oh, you love games. You must love gamification. Let’s gamify the classroom and let’s gamify the workplace.” And I thought that if we actually understood what made games really valuable, we would understand why most gamification was rotten and why it sucked out what was really valuable.

And so I ended up telling a story about what is wrong with gamification, and what is wrong with thin metrics in the workplace, especially when they start to capture our values and change our sense about what’s valuable in our life.

And I got to this point where I realized I had an entire story where scoring systems in games turned out to be beautiful, delightful, the seat of joy. And then I had this story about how scoring systems in bureaucracy, government, and corporations, seemed to suck the life out of people, and I wanted to understand why.

And that’s basically why I wrote this book. I was obsessed with why scoring systems were basically responsible for my favorite parts of life and my least favorite parts of my life.

Pete Mockaitis
Yeah. And I think you do a masterful job in the book, as you say, that it hits different when we’re scoring in a game versus when we’re being scored at work or with the KPIs and the metrics associated with performance there.

Can you maybe tell us a cautionary tale for professionals of why should this be on our radar? Why should we have our antenna up to these philosophically concepts in terms of our real-world life, happiness, flourishing career implications?

Thi Nguyen
Let me tell two stories. One is personal and one is institutional, and I think there’s a lot of lessons to be learned from both. So the personal story, I mean, I went into philosophy because I thought, I mean, philosophy is a terrible career decision, right? Like, jobs are hard, pay is terrible, work is hard.

The only reason you do it is because you love it, because it’s this weird, fascinating thing. And there are particular kinds of questions and ways of asking about the world that I loved. And then I got plunged into the world of philosophy.

And philosophy, just like every other world, is a profession that has its internal metrics and indicators. There is a status ranking of journals and a status ranking of universities. And I, like a lot of other people, were brought up to aim intensely at getting articles published in the highest status journals. Let me really clear how we did that.

What you did was that you had to write a very specific, very narrow, very technical kind of philosophy on very specific kinds of topics. And it was really boring. And I, basically, spent five years in this field that I was supposed to love, working on things that were more and more boring to me, precisely because I’d come to be guided by a system that represented, not like my sense of what was important, but some kind of like external redigestion and like vomiting back of what people in general thought was important. There’s one story.

I think another one, one thing that I’ve started to think about is not just like the external metrics are external, that they’re somebody else’s, but there’s a particular flavor to them. And I think most people know what I’m talking about. They feel inhumane, in a way. They feel distant, they feel rigid, they feel like they don’t capture what’s actually important. And I’ve been trying to figure out why.

And there’s all these examples that I find really interesting. So here’s one. Sally Engle Merry is an anthropologist who studied human rights work, and she got really interested in the ranking of, so the US State Department issued something called the Trafficking in Persons Report. It’s their sex trafficking report. And they’re rating countries on various countries’ ability to combat sex trafficking.

And the primary metric is conviction rates of sex traffickers. And what Merry points out is that this is actually a terrible metric because a lot of sex trafficking is highly correlated with ambient poverty. So if a country manages to decrease general poverty, uplift the economic status of its poor citizens, actual sex trafficking evaporates.

But since it’s evaporated, there are no sex traffickers to convict. So, by the metric, an actual successful lifting of general citizen well-being, and an actual reduction in sex trafficking, counts as a failure because the thing that the metric is picking up on, it’s not actual sex trafficking, but a very rough proxy of when that kind of activity enters into the government site via a particular kind of interaction, the arrest of a sex trafficker.

I think one of her reasons she says why this happens is because actual sex trafficking is actually incredibly hard to track partially because, by its nature, it occurs out of view and because it’s really, like what counts as a sex trafficking victim and an actual sex trafficker is really, really fuzzy on the edges.

She has this incredible example. She says, imagine someone who’s starving, a woman who’s starving, who crosses an international boundary to work in a brothel. And then the next year, they bring their friend, who’s also starving, across the boundary to work in the same brothel. Is the first woman now an international sex trafficker, right?

Really hard to answer those questions. You don’t have to answer them if you go to conviction rate, right? It’s so easy. It’s so bright. It’s right in front of us. Another example nearby is the example of Charity Navigator. Do you know Charity Navigator?

Pete Mockaitis
I’ve been there many times.

Thi Nguyen
So Charity Navigator is supposed to be a nonprofit watchdog that watches on other nonprofits and rates them for how good they are at charity. And for a really long time, it’s changed recently due to these exact criticisms, but for a long time, over a decade, I think, their primary metric for rating nonprofits was a throughput ratio.

It was a ratio of how many donations were given versus how many of those domain donations emerged on the other side, and were given as resources or money to the other place, to whatever the target is.

It turns out, this is, again, a terrible metric. And the reason it’s terrible is because by that metric anything spent internal to the nonprofit counts as waste. The metric depends on this image that nonprofits are just kind of pipelines for money. And, say, any money that a nonprofit wants to spend on an internal expert, will make it plunge in the rankings.

This example is so interesting to me, because the reason that we seize on that metric is because, in order to actually rate nonprofits, we would actually have to know a huge amount about their very specific domain.

So we would have to understand, like, about the housing crisis in one particular part of the world, and the lack of doctors in another part of the world. And we would somehow have to figure out how to compare those in a clear way. That’s really hard.

Accounting, on the other hand, is really stable and nonprofits do it in a similar way. And so if you focus on that layer, right, the accounting layer, you can find a kind of similar-enough quality that automatically outputs a kind of quantitative measure.

So you can generate an objective ranking, but you’ve generated that objective ranking by shifting the target over from what actually matters to a topic matter that has been chosen precisely because it can create an easy ranking.

There’s more kind of large-scale explanation to unpack, but I think these are really interesting examples of how the process of measuring things at scale seizes on certain kinds of parts of the world and has a lot of trouble coping with the parts of the world that actually might matter.

Pete Mockaitis
Yeah, and I hope, I know the way some people’s brains just fire off. They see implications right away. And sometimes you need a little bit of a bridge. And so, as you’re saying these, and I’m hearing it, and I thought like, “Whoa, this is actually epically high-stakes transformational stuff.”

Because when you say what counts as a sex trafficker, I mean, like you bring it into a business, like, “What counts as revenue? What counts as a customer? What counts as customer satisfaction?” And then, “What is the underlying view?” Like, nonprofits are pipelines for money. You make a point here that metrics are not neutral and objective. They’re values-laden.

Like, a person decided, “This is the metric we’re going for and this is what counts and this is what doesn’t,” based upon what they were trying to accomplish. And that has major downstream amplifications or ramifications for everyone that’s engaging with these numbers.

Thi Nguyen
So I’m not a philosopher of science, but I hang out with lot of philosophers of science, and there’s philosophy of science in this other field nearby called science and technology studies.

One of the primary themes that emerges from this work is that a lot of people tend to think that a lot of scientific tools, a lot of measurement tools, and a lot of data collection systems are value-neutral. They just represent the world in a kind of neutral way, but actually these tools are value-laden. This means they represent a particular point of view with particular interests. And they often do that by what they put in and what they leave out.

One of the places this becomes really clear to me is just thinking about maps. So maps are a kind of representation. You might think that a map is neutral, but a map, by definition, deletes most of the world, right? What a map is doing is something that highlights certain parts of the world. Who chose what to highlight, right?

Maps are good for certain kinds of things and not good for others based on decisions we made in the background. So most of the maps I look at are very good for telling you how to find a business or how to drive there. They’re not really good at telling you where the places that sound pleasant are, where the friendly neighborhoods are.

And they could do that, right? You could create a map that represents where nightlife is or represents where nature is. And people do make those maps, but the standard maps leave that out. They represent an interest.

The data system, I think about this a lot, so at my school, the administration is interested in lifting student success, which sounds awesome. But student success is largely defined in terms of graduation rate and graduation speed, and it’s not defined in terms of things like reflectiveness or thoughtfulness or creativity or ethics, right?

So, for example, if your primary measure for student success is graduation speed, and I meet a student who is bored, their major isn’t doing well for them, they’re not that good at it, they’re kind of frustrated. And I convinced them that the major is the wrong one for them.

And we talk and we’re like, “Oh, that’s what they’re really interested in,” and we help them find another major that’s better fit for them, that makes them happier, that fits their skill set more, but it’s going to take another year, that counts as a failure.

Hobbes, Thomas Hobbes, the political philosopher, I think a lot of people might know that he said that morality comes down to political power, and political power comes from the tyrant, whoever has the ability to, like, get people to do what they want.

And Hobbes actually says that the real source of political power, the ultimate form of political power is not military strength or might, or even economic strength. It’s power over language. Because if you can dictate to them what success and failure mean, then you can control them from the inside. And these are the stakes, I think.

What metrics actually are, are a way of fixing what counts as success and fixing what we should all be moving towards. And if there’s some systematic slippage between what actually matters and the kinds of things that it’s easy to build metrics about, then our entire internal guidance system is going to be deeply rewritten at its core based on somebody else’s values, right, some particular person that made the measurement system.

So I was reading this paper from a philosopher named Philippi about values and measurement systems. And he was pointing out how value-laden the idea of intelligence is, right? So intelligence testing is a very value-laden measure.

So you might know that IQ tests are racially and gender-biased. That might be true, but that’s not the center of what he’s talking about. Here’s the value-laden system inside our intelligence tests. The intelligence tests we have right now all encode into them the view that mathematical and logical ability is more important than emotional sensitivity, right?

There is such a thing as emotional intelligence. But think about the fact that our intelligence tests either don’t test it or, if they test it, it’s barely weighted, right? That is a particular set of interests and a representation of how people should be that’s baked into a measurement system that then looks objective once it’s become like a kind of standard use measurement system.

People just think that’s the way the world is, “Of course, that’s what intelligence is,” But it’s a decision that somebody made to weight things that way.

Pete Mockaitis
Yeah. And I’m thinking about, so we missed the emotional intelligence, but we also missed some of the creativity-type intelligences, or I’ve got a posse of craftsmen who are fixing some water damage right now and doing some amazing things, and that’s a different kind of intelligence than what shows up on the ACT.

Thi Nguyen
Imagine what it would be like if we didn’t use that system. You could start to think about people as having hundreds of different capacities, each of which you could think about in a different way. Some people are good at drawing, some people are good at sensitivity, some people are good at telling stories, some people are good at logical and mathematical ability.

But instead, what we’ve done is we’ve created something that says, “No, all of these boil down to one thing.” And the idea that there’s one thing that is intelligence is a worldview that’s subtly been baked into a kind of metric.

Pete Mockaitis
Absolutely. And it’s so interesting when you talked about scoring sets their desires, and it’s true. I think about strategy games. You probably know the name for this. There’s a principle at which, if there is a dominant strategy, folks are going to do it. And so a great game, they say, is one in which there’s not a clear cut, overpowered strategy, but rather a variety that you can choose from.

And I found myself doing this. Like, I’m playing this game, ostensibly, to have fun. And yet, if I find there’s a winning thing I can do, even if it’s not fun, I do it. And I wanted to hear your take on values capture. I remember back when I was consulting at Bain, we had what’s called a sell weekend or offeree weekend, in which the folks who had received offers to work at this consulting firm come out and we try to show them a great time, and how cool it is to work at Bain and all that.

And this one person, I remember, she also had an offer from McKinsey & Co., a rival consulting firm. And she was torn, you know, between this. And then she said, “I feel like I’m choosing between happiness…” which was Bain, “…and prestige or pleasing my parents,” which was apparently Mackenzie, is what these things represent to her.

I don’t know if that’s really a fair summary, but that’s how she saw the world. But she did, she went with McKinsey. And so the scoring, in terms of like the rankings of prestige, kind of like the philosophy schools, whether they’re consulting firms or whatever, does, in fact, have the ability to set our desires, and we can just kind of default to stuff without even thinking about it.

Can you expand upon your concept of values capture? How does it happen and how do we guard against it?

Thi Nguyen
So value capture is a term I came up with to describe something I was feeling all the time, and I think a lot of people were, which is that your values are rich and subtle or they’re developing, and then you get put in some kind of institutional or social setting that presents a simplified, typically quantified version, and then the simplified or quantified version takes over in your heart.

I want to be clear that value capture is not the same as being incentivized. So I think a lot of people know Goodhart’s law, right? When a measure becomes an incentive, it stops being a good measure. And Goodhart’s law is gesturing at the same thing. It’s gesturing at this idea that incentives don’t capture what’s important.

But I think there’s a big difference between the first stage, when a metric incentivizes us, right, when we think like, “Oh, if I go to the higher-ranked thing, I’m more likely to get a job. I’m more likely to get more money.” And the second-stage thing of when the metric intrudes into you and starts to become how you conceive of the point of everything.

I think is that different? For me, like, here’s an example. I think that’s a big thing to ike, “Oh, you know, I’m on Twitter. I need to build a bigger following in order to get my message across.” But if you understand that you’re doing it just to get the kind of power to do the thing you really want, you’re not going to sacrifice your message just to get a bigger following.

But if your soul gets redefined and you start thinking like, “No, the thing that I care about in the end is just having the most followers and likes,” then that’s going to transform your entire way of interacting with that system. I’m not sure about your case.

I think there’s a big difference between a student, for example, who thinks, “I’m more likely to get a job if I go to a higher-status university,” versus a student that thinks, “Success in life is going to the highest-ranked university.”

Because once you go to the second thing, you no longer have a standpoint to reflect on whether or not the metric is working for you. If it’s the first thing, if it hasn’t gone all the way to your core, you can always think to yourself, like, “Well, I know this is important for money or resources or whatever, but is it really worth it to me? Is it really making me happy?”

And you can step back. People can step back from high-paying jobs and high-status jobs because it’s making them less happy. But if you’ve redefined in yourself that that’s what success is, then you’re no longer going to think to yourself, “I should stop doing this because it doesn’t make me happy.”

And I think that’s actually one of the cases I’m most interested in. Like, what happens when you forget to listen to your own sense of happiness or your own sense of value? And it’s gotten overwritten by this easy, clear, outside meter.

Pete Mockaitis
Yeah. And I think the easy, clear, outside meter is what’s so darn tricky about it. Like, money or compensation is, I think you call it, the most fungible of scores in terms of, yeah, you could turn that into a lot of different things, and we all understand it, and it is compared across many different opportunities.

And yet, it could be a lot harder to evaluate two options, like, “Well, one is clearly more money. Cool. But the other makes my heart come alive more, I think, but I can’t quantify heart coming aliveness on it as quite nearly as readily and directly as I can money.”

Thi Nguyen
Yeah. I mean, this is one way to put it. In many cases, I’m not saying that the metric is bad or even that it doesn’t track something real and something important. I’m interested in the fact that easily measured things tend to win out in justification fights against less easily measured things.

Like, should you eat rich cheeses high in saturated fats? On the one hand, there’s data about correlations with lifespan and heart attack rate. And on the other side, there’s the fact that it’s delicious and it makes you happy.

And it’s really hard to hang onto that in the face of those other numbers, especially when you have to have a public fight, right? This is the weird thing. Like, before I say this next thing, I just want to say I’m not anti-science. I believe science gets real truths.

But the world in which all our policies need to be evidence-based and data-based is a world in which we can only target things that are easy to count and easy to data-fy, and we lose our grip in things that are hard to data-fy. And I think, if you think that everything in the world that matters, can be counted by bureaucratic processes, then you have no problem.

But I think we have a really good reason to think that much of what’s really important tends to elude the specific institutional character of large-scale counting processes.

Pete Mockaitis
Yes, absolutely. And when you mentioned there, when there’s a slippage in metrics between what we’re going for and what actually is captured in the cut and dry definitional metrics, all kinds of implications can ensue. I think, perhaps the most terrifying part of your book was when you described five conditions that, when they’re present, folks who game the system will prosper or rise. And so could you just lay this out for us?

Thi Nguyen
So this is based on an article from Paul Smaldino and Richard McElrath, who are scientists, anthropologists, and computer modelers, and they have a paper called “The Cultural Evolution of Bad Science.”

And they do a computer model of a scientific community where the following is true. One, status is connected to publication rate. Two, if you use the most rigorous, most careful method, you’re going to publish more slowly.

So the people that will gain the most status and power are the people that are going to game the system and use the least rigorous method that will get them just barely over the finish line of publication.

And then if you assume, the last assumption of the model is that if young scientists imitate people with higher-status jobs and their methodologies, then we should expect science to turn pretty crappy pretty fast. And I thought this argument generalized.

The general version is, first, if you think there’s a gap between what’s important and what’s easy to measure, and then you think that the institutions we have tend to reward people with power and resources for hitting the easy-to-measure metric, then you should think, “Look, then there’ll be two populations.”

There’ll be the people who still care about what they care about and are aware of the importance of the metric, but trade off between them. And then they’re the fully value-captured people, the people that are just going to go all out and just game the system and ignore what’s really important and just aim at the metric. And we should expect that the latter population, the narrow hyper gamers, are going to be the people that win out, right?

And if they win out and they, in particular, if they re-tune their institutions once they gain power to make the metrics even more powerful, then you should expect a terrifying feedback loop where, over time, the systems will tend to sort for the people that are willing to ignore the quiet whisper of what’s really important, and just target hell or high water the thin metric that is written, and that kind of narrowness will systematically gather all the social power. That’s the model. Then you can decide for yourself about whether it fits reality.

Pete Mockaitis
Well, and it’s spooky and it’s partially explanatory for all kinds of things, you know, why the rich get richer, why we have corrupt politicians, why that idiot is the boss, you know, in terms of, “Oh, well, they were rewarded not so much based upon their inspirational leadership. They were rewarded because they hit a number,” or, “The board thought they really liked that guy and he seemed sharp.”

And so, like, there was a gap between what was readily measurable or observable versus what we would hope is the fundamental thing. And the first thing that came to mind when I was reading this was just about politicians and votes.

Like, we would hope in, like, in a representative democratic republic situation, “Well, yeah, votes are kind of the measure by which we have, and which we feel someone’s doing a good job of representing us or should represent us.”

And yet, that is a tremendously gameable metric via monies that can just blast enough advertising to, apparently, get enough votes, as well as polarizing messaging, “What gets people mad enough to actually show up at the polls instead of just sitting on the sidelines?”

And then you could just imagine this in all kinds of scenarios, like, “It seems like this person is at the top, but they don’t deserve to be?” Head scratch, “What’s behind that?” Well, I think, a decent amount of the time, it’s exactly this underlying dynamic that you’ve spelled out.

Thi Nguyen
I think a big part of the background is that I think a lot of metrics are extremely usable if used with care, if we know that they’re just a rough approximate proxy. It’s only when we treat them as all important.

I think there’s an important piece of the background puzzle, of the background picture to put in right now, which is why metrics have this character, right? I mean, one response you might have to all of this is, “Let’s just fix them. Let’s just get better metrics. And then the people that are all hell-or-high water gaming the metric, will just do what’s good.”

But I think that’s not going to work. I had this intuition that it wouldn’t work. And I think the best explanation came from a bunch of historians. So, Theodore Porter, I think, helped me understand a lot of what was going on. He’s a historian of quantification culture.

And he’s trying to explain why he thinks that bureaucrats and politicians compulsively reach for quantificative reasoning even when the metric they know is bad, right, even when it’s just a blatantly terrible metric.

His explanation was that qualitative and quantitative reasoning were different styles of thinking and justification that were good at different things. So qualitative reasoning, he said, is rich and subtle and context-sensitive and dynamic, and can capture all kinds of complexity, but it travels badly between contexts because it requires a lot of shared background knowledge to understand. And it doesn’t aggregate.

And part of why it doesn’t aggregate is specifically because it’s working on so many different dimensions, right? When I’m writing qualitative descriptions to my students about their philosophy essays, I’ll talk about their originality, their rigor, their carefulness, their curiosity, all along different dimensions. And then somebody else might write in their qualitative assessment a bunch of different dimensions. How do you aggregate those?

To make quantitative data, says Porter, we identify a context-invariant kernel and we stabilize it across context. So we make it rigid. And to make it work across context, we have to figure out the bit that everybody understands the same way, which means that bit cannot depend on specific context, specific background, or specific sensitivities.

So, for me, this is like letter grades, like A, B, C, and D. There’s not a lot of information there. But what information there is, is thin enough and simple enough that everyone can understand it the same way. And so that message travels, right? And it aggregates instantly.

So Porter’s insight is that quantitative measures communicate well and are an incredibly good way to connect people and coordinate people because they’ve been designed to communicate, right? His claim is that quantitative data is portable, it travels well between contexts. But portability is a design achievement achieved at the cost of high context, and it’s not just accidental.

Removing context is the thing that makes it extremely accessible and extremely cross-cutting across contexts. And that vision, I think, is really terrifying to me. Because, for me, this shows me that this is not a trap we can get out of, right?

The essential thing that gives metrics their juice and makes them so powerful is that they are narrow by design, and that narrowness is precisely what makes them so usable and so dangerous.

Pete Mockaitis
Yeah. And I guess, maybe to have some hope in here, I think just having a bright light of awareness, which you are providing here, can work wonders for ourselves and our own spheres of influence, and leadership, and families, and organizations, and communities, and careers, to know, “Okay, a narrow, dogged, ferocious, focus on a metric or two or three will necessarily be incomplete and comes with some major downsides. So we got to really check ourselves and note, ‘Okay, this is a rough gauge and it gets an approximation of a thing and it can’t be the whole thing.”

Thi Nguyen
And yet, we also see why it’s so tempting to treat it as the whole thing. But, yeah, there’s not a world in which we can have our institutions work without metrics and without measures that we coordinate over, but they’re so dangerously thin, and it’s so easy to forget. And part of why it’s easy to forget is, if you use them, you’re instantly comprehensible to everybody. And I think that’s very tempting.

Pete Mockaitis
Very much. Well, Thi, tell me, as you think about this domain of knowledge, and when it comes to individual professionals navigating their work lives, their careers, do you have any top do’s and don’ts you think that come to light or emerge out of this rich set of ideas?

Thi Nguyen
I have a goofy idea I want to run by you and see what you think. So a lot of the problem of metrics comes from their being established very distantly at scale and being rigidified, right? They’re distant measures of our success.

I’ve been trying to think about various solutions. And, for me, games are an inspiration. And one of the ways they’re an inspiration is that game scoring systems aren’t distant. They’re modifiable in a few ways. You can move between them. You can change them. You can house-rule them. You can design your own.

And so I’ve been wondering if this can be applied to institutions, too. So here’s something I tried. In the age of trying to figure out how to grade students in the era of ChatGPT, in my last class, I let the students design their own grading system.

I let them design, as part of the class through a conversation, what the assignments would be and how they would be graded on them, given what their goals were and what the limits were, and how the world was changing.

So here’s one thought. Maybe in the workplace, if you’re a manager, what you can do is constantly redesign your metrics in conversation with the people that are being measured about to capture what is a value. Is this goofy and insane? Part of the problem is that you’re not going to be able to export them readily, and you’re not going to be able to take off the shelf a pre-established form of measurement, but roll your own. What do you think?

Pete Mockaitis
Yeah. Well, I like it a lot. And I’m thinking about how I’m on the board of a nonprofit. And so the board always says, “Hey, Pete, so you’re going to run the performance review for the executive director, right?” And I’ll say, “I guess I’m the How to be Awesome at Your Job guy, so, okay, I’ll take that on.”

And what I think is cool is that, well, each year she has her goals, and each year the goals are different in terms of what is needed at the time. And, thusly, each year, the metrics can change. And I think, because we’re, well, it is, it’s context rich, you know.

Like, I’m in it. I understand it, like what we’re doing, what we’re trying to achieve and how these numbers are incomplete and they’re are means to an end, but also like they do have value and give us a gauge in terms of, “Does this look more or less like a win, or like a loss, or something in between?”

And so, if the numbers are way, way low when they should be higher, it’s like, “Well, yeah, that looks like a loss, even though they’re imprecise and imperfect, that will show up accordingly in the performance review.” So I think that it is, it’s very much doable. And, in a way, kind of fun, keeps it fresh and relevant. But it does, it takes a heightened level of commitment, as opposed to a one-size-fits-all, “Here’s your production goal. Make sure you hit it each quarter. Boom!”

Thi Nguyen
Yeah, and you won’t be able to compare between groups. The whole point is that it’s not that metrics are bad in and of themselves. It’s that the thing that makes them insensitive is their fixity at scale. And that’s also what lets them aggregate easily.

And so the proposal here is just at a different point in the trade-off scale. Now you’re going to be thinking in a context-sensitive way about what you care about at particular moment, and you might set up a metric for a period of time and then change it.

But you won’t be able to auto-compare teams, you won’t be able to auto-compare one person’s performance over a huge amount of time, but you will be able to generate metrics that are responsive to the details of what matters in the case. But, again, it’s a massive trade-off.

Pete Mockaitis
Oh, thank you. Can we hear a favorite quote, something you find inspiring?

Thi Nguyen
William James, philosopher, pragmatist, says, “When you’re seeking truth, there are two totally different goals that people confuse. One is to get the most truths in the end, and the other is to avoid error. And they’re totally different and they suggest different strategies.”

If what you want is a lot of truths, you might actually want to be really risky and take a lot of risks and make a lot of mistakes. Because if you take a lot of risks and try a lot of things and make a lot of mistakes, you will move more quickly towards the truth.

If you want to avoid mistakes, you should be very careful and conservative with what you try out. And these are two totally different strategies.

Pete Mockaitis
Well, yeah, that’s exciting. And I’m imagining, I don’t know who said it, but I thought there was some wisdom to it. Like, most great thinkers, a fair critique to just about all of them is, “Nice idea, but you took it too far.” And that really sounds like the first group that’s trying to maximize truths. It’s like, “Ooh, yeah, you made some mistakes, and there was some mess and dirt along the way there. But you did, you really advanced some stuff, and all of mankind is enriched as a result of having done so.”

And how about a favorite study or experiment or piece of research?

Thi Nguyen
I think one of the most interesting studies and a huge inspiration for this book was James Scott’s Seeing Like a State. And this is a book about how, you’ll recognize the themes, but it’s a book about states.

And by states he means governments and corporations, and about how they can only see the parts of the world that are processable through large-scale bureaucratic means, exactly what we’ve been talking about so far.

And then the second part of the study is an argument that states, “In order to make the world more processable, tend to reorder it to make it easier to count.” So they tend to want to even things out to make things easier to count. And it’s a study that crosses about 50 different historical case studies, and it’s extraordinary.

Pete Mockaitis
And a favorite book?

Thi Nguyen
Well, it’s in a book, it’s a paper. It’s by Annette Baier. It’s called “Trust and Antitrust.” And it’s a paper that taught me, I think about the most important ingredient in social life. She thinks that the center of human life is trust. And what trust is, is making ourselves vulnerable to some external power by putting something of ours in their power.

And that we have to do this to extend ourselves and to cooperate, but that human life is, essentially, one where we are constantly at risk because we constantly are so entangled that we’re entrusting ourselves to other people. And this, I found, just incredibly explanatory of the state of the world.

Pete Mockaitis
And a favorite tool, something you use that helps you be awesome at your job?

Thi Nguyen
I think my favorite tool is walking away. I think, sometimes, the right answer to being stuck, for me, is to put everything I’m stuck, write everything I’m stuck, on a whiteboard so I don’t forget it, and then just walk away. Leave the office. Leave the desk. Leave that problem. Go work on something else for days, weeks.

Pete Mockaitis
Okay. And is there a key nugget you share with students or readers that really seems to connect and resonate, they retweet, they Kindle book highlight, they say, “Professor, this was amazing”?

Thi Nguyen
I often give students the argument that Bernard Suits, this philosopher of games that I was really inspired by, that he gives the end of his book. And he says at the end of his book, “Imagine utopia where we’ve solved all our practical problems. What would we do if we cured all medical problems and all technological problems?”

He says, “We would play games or we would be bored out of our minds. So games must be the meaning of life.” And, you know, it’s kind of a goofy argument, but what he’s really saying is he’s restating an old nugget from Aristotle, which is that the meaning of human activity can’t come from stuff we make or the outcomes of our actions. It has to come from the action itself. It has to come from the process of doing.

And if we drain out all the interesting processes of doing just to make a ton of stuff, then we’ve accidentally leaked away a meaningful life.

Pete Mockaitis
Thi, if folks want to learn more and get in touch, where would you point them?

Thi Nguyen
My website is Objectionable.net. I’m on Bluesky @add-hawk, A-D-D, underscore H-A-W-K.

Pete Mockaitis
And do you have a final challenge or call to action for folks looking to be awesome at their jobs?

Thi Nguyen
Every time you look at a metric, be suspicious and ask what values it’s imposing on you.

Pete Mockaitis
Perfect. Well, Thi, thank you. This was a treat. I wish you many, many high scores of the most meaningful sort.

Thi Nguyen
Thanks, man.