Transcript 0:00 Hey, folks. It's Quinn. Since we recorded this conversation, we've renamed the podcast. It used to be called Important Not Important, which is what you'll hear in the episode and is still the name of the company. 0:12 But now the podcast is called The Most Important Question, that is, what can I do? 0:18 It's the question we get the most from all of you, and it turns out it's the question I've really been asking guests for almost two hundred episodes now. So here we are, The Most Important Question. 0:29 Please enjoy this old conversation. Thanks for listening, and thanks for giving a shit. 0:33 [upbeat music] When is a cancer scare, a rejected mortgage loan, a false arrest, or predictive grading more than a glitch in the AI? 0:52 That's today's big question, and my guest is Meredith Broussard. Meredith is a data journalist and associate professor at the Arthur L. 0:59 Carter Journalism Institute of New York University, research director at the NYU Alliance for Public Interest Technology, and the author of several books I loved, including More Than a Glitch: Confronting Race, Gender, and Ability Bias in Tech and Artificial Unintelligence: How Computers Misunderstand the World. 1:18 Her academic research focuses on AI in investigative reporting and ethical AI with a particular interest in using data analysis for social good, which would really just be great. 1:29 Um, Meredith has appeared in the twenty-twenty documentary Coded Bias, an official selection of the Sundance Film Festival. 1:36 She's an affiliate faculty member at the Moore-Sloan Data Science Environment at the NYU Center for Data Science and a twenty-nineteen Reynolds Journalism Institute fellow. 1:45 She's a former features editor at The Philadelphia Inquirer. She has also worked as a software developer at AT&T Bell Labs and at the MIT Media Lab. 1:55 Meredith's features and essays have appeared in The Atlantic, The New York Times, Slate, and other outlets. 2:02 If you have ever gone out into the world or even just stayed at home and turned on a computer or used the internet in some way to apply for something or literally anything, this one is for you. 2:17 Welcome to Important Not Important. My name is Quinn Emmett, and this is science for people who give a shit. 2:24 Our mission is to understand and unfuck the future, and our goal is to help you answer the question, "What can I do?" [upbeat music] Meredith, welcome to the show. Thank you so much for having me. 2:38 It's great to be with you. Hundred percent. Uh, like I said, I will try not to spend the entire time, uh, trying to pick apart what's on your bookshelves, but it looks interesting. Is that a mushroom up there? 2:49 Is that a- Uh- A lamp? What is that?... entirely possible. Uh, up here? Blue and... Yeah, yeah. Oh yeah, that's a, uh, that's a paper flower. The little girls who live down the hall from me, uh- Mm-hmm... 3:01 on one rainy day decided they were having a, uh, a craft sale, so I bought- The best... a paper flower from the little girls down the hall. That's wonderful. I mean- Yeah... you couldn't spend, uh, money in a better way. 3:13 That's amazing. Yeah, yeah, exactly. It's the best fifty cents I ever spent. Hundred percent. Um, two stories, and then we'll get to, you know, uh, AI ethics. One, my uncle has a story about, um, 3:25 s- uh, just walking out the door when he was little, uh, with all of, uh, I mean, not many, they were pretty poor, but my grandmother's candles and I guess, uh, candelabras and such and trying to sell them to the neighbors [chuckles] who immediately knew what was going on 'cause he was five. 3:38 I don't know, something like that. Um, so sure. 3:42 Uh, and then my daughter recently similarly, uh, she's, uh, out at camp, like, in the woods somewhere, uh, in town, and, um, her and her friend have decided they're gonna start making and selling, uh, you know, little friendship bracelets at, uh, the camp store. 3:57 She's s- she's gone full capitalist very quickly. Good for her. As is typical, one of my sons is trying to figure out why he doesn't get a cut of it without participating at all, and I'm like, "This is the thing. 4:07 This is the problem." [chuckles] Yeah. Ah. Yeah, yeah. Pretty standard. Yeah. 4:12 Have you, uh, have you taught him about management yet and, uh, you know, possibly about algorithmic, uh, algorithmic surveillance so he can, uh- Mm-hmm, mm-hmm... 4:21 he can convince his sister and her friend to, like, join a platform that he started and he can take a cut? [sighs] Right. For, for doing, like, what work in particular? Something. Right. Yeah. Uh, it's, it's coming. 4:36 We're... Ugh, God, killing me. Anyways, this is so great to have you. Um, I love your paper flower. 4:42 So, um, Meredith, we usually start with one ridiculous question to get things going, and I've asked it almost two hundred times, and at some point I keep 4:50 feeling like I should retire it, but, um, because it's pretty tongue in cheek, but at the same time, it's fun, so why not? Uh, Meredith, why are you vital to the survival of the species? 5:01 And I encourage you to be bold and honest. Well, one of the things that I really care about is empowering people around technology. 5:11 Uh, algorithms are increasingly being used to make decisions on our behalf, and those decisions are often unfair or unjust. And- Mm-hmm... 5:22 uh, you need to know a little bit about algorithms and how AI works in order to push back, uh, against those unfair algorithmic decisions. 5:32 And so one of the things that I do in my work as a data journalist and an algorithmic accountability reporter is, uh, I help people to understand the inner workings and outer limits of AI and algorithms so that more people can enter into the conversation and, uh, we can hopefully get rid of a lot of the algorithms that are making unfair decisions about people's lives. 5:59 Well, I think that's fantastic. Thank you for sharing that off the cuff. That was impressive. Um, some people just laugh at me, and then usually we get something profound out of it, but that was great. 6:08 And, and I guess that really [chuckles] That really gets to the crux of it, right? 6:12 We need more people realizing they have been using versions of these tools for ten years, and so they need to be part of the conversation because right now the conversation is mostly a very specific group of people who are making and profiting from these things and, you know, benefiting some more folks, but mostly hurting other folks. 6:30 Do I have that- Yeah. Close? Absolutely. Absolutely. Algorithms have been held up for a long time as being, uh, kind of more fair or more objective or more neutral, and, uh, that is not at all the case. 6:45 Uh, one of the things that Cathy O'Neil, who wrote Weapons of Math Destruction, has often said is that algorithms are opinions embedded in code. So good. It, it sums the whole thing up. Yeah. They're opinions. 6:58 I-- And so we do not benefit by putting all of our faith into algorithms. The ideal situation is human decision-making coupled with algorithmic decision-making, not one or the other. Sure. Um, so I like to, for, for, 7:17 less so for you, more for me [chuckles] and for our listeners who run the gamut of, you know, young students to wealthy philanthropists to policymakers and bartenders, it doesn't matter. 7:28 But I find it does help a little bit to set the tone, um, and I try to do it in my own writing as well and, and one-on-one conversations I have with just a, a brief sort of CliffNotes, uh, understanding of why 7:42 everyday things are the way they are, at least that someone might be trying to interact with, right? If they're like, "I wanna donate to this, why is it so hard to do this?" And we talk about it briefly. 7:52 How we got to the systems we have today, who has the power behind them, like you were saying, who, who gets to be in the room, who benefits from them, uh, who is hurt, and what the costs are, whether we're paying them in or for naught. 8:02 And, and obviously, like, climate is a big part of that. And there was such a great specific example, I mean, among a million of them, in, in your most recent book, More than a Glitch. 8:13 I wonder if you can talk about what is Y2Gay, and why is it such a pain in the ass, but also emblematic of these systems as a whole? 8:22 The Y2Gay example, uh, is in the chapter of More than a Glitch where I talk about gender and databases. 8:30 So one of the things that most people don't realize is that databases that we use today have these nineteen fifties ideas about gender embedded in them, right? 8:41 So computers came to prominence in the, uh, in the sixties, really, uh, and that's when a lot of the systems that we use today, still use today, got developed, right? 8:54 So, uh, computers used to be these giant machines that would take up, you know, multiple rooms, and now they're, you know, so small they can fit inside your watch. 9:04 Uh, that's really cool, but the, uh, the thing is, like, computers still function the same basic way, right? 9:11 It's still about electricity and gates and binary, and then there's a whole bunch of other stuff running on top of it. 9:18 So my last book, Artificial Unintelligence, has a whole section about, uh, how does the physical reality of a computer work. And then More than a Glitch is about, uh, race, gender, and ability bias in tech. 9:31 And I started thinking about the way that, again, nineteen fifties ideas about gender get embedded in databases. 9:38 So when I was taught to, uh, to make databases, uh, back in the eighties and nineties, I was taught to optimize my code, uh, for a certain kind of normative aesthetic. 9:55 This was not just aesthetic, it was also about memory because memory used to be expensive. Sure. Computers used to be expensive. 10:04 So one of the ideas behind optimizing your code was to make it run in as small a package as possible because that was the most economical way to do it because if you ate up all the memory with your code, then, you know, your computer would crash and your program wouldn't work. 10:21 One of the things you think about when you're trying to optimize your code is you think about how much space does a variable take up, right? Because code depends on variables, and the variables are of different types. 10:33 You know, you can have a number, you can have a string, you can have a binary. And these variables take up different amounts of space on the computer, different amounts of memory space. 10:44 So a string would take up, like, this much space, and then a number would take up this much space, and then a binary, a zero or a one, would take up this much space. 10:54 So when you thought about the, uh, entries in your database, the individual records, you had to decide, okay, what kind of record is this going to be? Like, what type of variable is going to go into this record? 11:07 And gender generally got recorded as a binary because in the nineteen fifties and then in the nineteen sixties, people thought that gender was binary. 11:19 Well, now we know that gender is a spectrum, and if you're gonna have gender in a database, you're going to need to make it an editable field. 11:27 Uh, but the systems that we're using, the computational systems, again, were designed in the nineteen sixties, so, you know, your bank, uh, application, like, is pretty much still running... 11:38 Like, there are some levels of your banking application that are still running on Fortran, which is a programming language that maybe you've never heard of. It's really, really old. 11:50 Fortran programmers get paid so much money right now because it's such an old language that nobody uses it anymore. Mm-hmm. 11:55 But there are still things that have not been rewritten in, uh, bank systems and insurance systems. They're still written in Fortran. Uh, it's a, you know, it's a major technical debt issue. 12:06 But at any rate, we are working with these very old systems, and so when we make new code, it gets stapled on top of the old systems, and you have to think aboutHow does the old system talk to the new system? 12:18 Well, you gotta think about these variables, and if you try and, uh, feed a string, you know, a sequence of letters, to, uh, a program that is expecting a binary, it's not gonna work, right? 12:36 So the idea that gender is binary goes back a very long way, and it is hard-coded into a lot of systems. 12:45 So, uh, the Y2gay issue came up as a result of marriage equality because, uh, in the same way that gender was hard-coded as a binary, marital relationships in software systems were encoded as being between one man and one woman, right? 13:08 It was a compulsory heterosexuality, uh, situation. And once gay marriage became legal across the US, people had to redesign their software systems because, uh, you know, uh, you... 13:23 there were all these rules in place saying, "Nope, we're go- not going to have a marriage relationship between these two users unless one of them is a man and one of them is a woman." 13:34 Well, okay, marriage equality, now we can have marriages between two men or two women, but needed to redesign the software systems. 13:44 So it was called Y2gay as a, a kind of nod to Y2K, which was the, uh, the previous, uh, time that people discovered that there was a kind of shortsighted, uh, way that our databases had been constructed. 14:01 So until, uh, 1999, people were encoding date as, uh, the year as only- Mm-hmm... 14:10 two numbers, and they realized that when we switched over to the year 2000, the prefix, 19, was no longer going to be, uh, the default, and computers were gonna think that the year was 1900, not 2000. 14:26 [laughs] And that was gonna break a whole bunch of things. So- Sure... it was a big software programming nightmare. 14:31 So any rate, the, the big idea here is that human values are coded into software systems in all kinds of ways that we don't necessarily expect, and we need to reflect on those human values, and we need to update our computer systems accordingly. 14:49 I love that. Thank you for that. I rem- [laughs] remember where I was sitting for Y2K, uh, for both Y2K and Y2gay. I remember Y2K, sitting there at midnight. 14:57 I was at, like, a local concert or something and thinking, "Well, w- let's... Here we go. [laughs] Let's, let's find out." 15:04 [laughs] And of course, Y2gay is both never as simple in the build-up or in the aftermath of, you know, the White House, uh, shining rainbow lights that night, right? Um, it's th... it, it... 15:17 decades and decades and decades of, of work and l- and, and loss and, and suffering and struggle to get to that point. And then, 15:24 again, everything, even the most mundane things that, y- like you said, are hard-coded into how we operate as a society and an economy, uh, have to be rewritten. And 15:34 I, I noticed in ear- uh, both in the book and in what you were just, uh, commenting on, it's not just that we suddenly had to offer more options, right, than male or female or more and m- uh, man and woman. 15:49 It was that it actually needs to be editable. And, I mean, I, I think that's obvious, and you, you spend some time thinking on in the book. 15:56 But let's dive right into that, about why does it need to be editable, and why is that such a difficult technical challenge as well as a societal challenge in so many places? Obviously, we've got deadnaming. 16:09 We've got any number of just things that are kinda horrific, we've decided, uh, and understandably so on the societal scale and on the ethical scale, but are, are difficult to change. But we, we need to do them. 16:22 So I wonder if you can talk about that version of it because, again, that is a technical challenge, is it not? Hey, parents. 16:28 You know that moment when your kid is asking you something for the 47th time while you're trying to make three different breakfasts, pack lunches, and figure out if your house insurance is about to get canceled because of climate change? 16:39 And all you can say is, "Not right now." I'm Quinn Emmett, founder and host of Important Not Important. And I'm Claire Zuelke, writer and producer of the Evil Witches Substack newsletter. 16:49 And we're here to talk about, generally, with tears often, parenting in these absolutely just crazy times. 16:56 Every week, we're having honest conversations about raising kids while trying to stay somewhat sane, whether that's debating screen time- Explaining wildfire evacuations, and what fascism is, and why coups has a P in it... 17:10 surviving another parent-teacher conference and behavioral reflection forms... [laughs] or just trying to get everyone to eat something other than dinosaur-shaped nuggets, which, by the way, are delicious. 17:20 We're not here to tell you how to be the perfect parent. We're here to remind you that you're not alone in this chaos. 17:26 Join us for our new show, Not Right Now, a show for parents who know that loving your kids and wanting to hide from them in the pantry can both be true at the same time. 17:36 Because we're all just trying our best while googling, "Is this normal?" in our bathrooms at 3:00 AM. Not Right Now, new episodes every Thursday wherever you get your podcasts. 17:45 I got interested in this kind of accidentally because I was trying to, uh, do something that was not particularly pro-social. [laughs] Uh, so I was trying to ride the train using, uh, my husband's rail pass. 18:01 So we lived in Philadelphia at the time. Okay. And he would get this monthly SEPTA pass. SEPTA is the, uh, the train system, the rail system. Mm-hmm. 18:10 And you could get this monthly pass, uh, to ride the train an unlimited number of times, but it would have a sticker on it that said M or FAnd I wanted to use my husband's train pass, but he said, "No, you're probably not gonna be able to pass as an M." 18:25 And I was kind of annoyed by this, and then I wondered, well, all right, I'm annoyed by this, but I should think about what is the experience like for people who are not like me. 18:36 Uh, and so I found out that there was a, an activist group called Philly RAGE, uh, Riders Advocating for Gender Equality, uh, that was trying to eliminate- Incredible name... I know, such a great acronym, right? Mm-hmm. 18:51 Uh, they were trying to eliminate the gender stickers on the SEPTA passes. I thought this was a great idea. 18:58 But trans and non-binary folks and gender non-conforming folks, uh, would have this problem when they went to get their rail passes, where, uh, they would face microaggressions because, uh, you know, the person at the SEPTA window would give them a hard time about, uh, which sticker they were gonna get, or would give them both stickers, and then they had to, you know, get a hard time from the conductor and- Mm... 19:23 it was just this, this situation that did not need to happen. Like, there was all of this confusion and all of these bad feelings that, like, there was no reason for people to have to go through this particular pain. 19:36 Because what are we talking about? Sure. We're talking about a totally unnecessary sticker on a train pass, and yes, more people should be riding public transit, period, all the time. Sure. Of course. 19:46 And yes, we need more pu- funding for public transit- [laughs] Yes... in general. Yeah, yeah, yeah. This was how I got interested in the issue of gender and software systems, right? 19:57 Because I was thinking about the, uh, the kind of mechanisms of this rail pass. I started thinking about what is it like for people who are not me, and I discovered this whole world of activism. 20:11 And of course, because I think about how software systems are made, I, I started thinking about, okay, what are the back end changes that SEPTA would need to make? Mm. Uh, 'cause it's not just about the sticker. 20:23 There's also always some kind of back end system computationally that is driving people who make decisions about forms and stickers and what have you. So... 20:36 And the thing is about computers is computers are all about bureaucracy. And I'm a data journalist. I find stories in numbers. I use numbers to tell stories. 20:47 A lot of my stories have to do with bureaucracy and understanding it and understanding- Mm... kind of technocratic decision-making, which is way more interesting than it sounds. 20:59 It is much more interesting, and we, and we can dive into all that. 21:01 I, I wonder, though, if we can, uh, laterally and in the same world, you've ran into a very similar-- You're so good in your books and your writing about sharing your own personal stories. 21:10 Um, and, uh, you wrote about at the doctor's office having a similar experience, but, um, less about gender and more about, uh, racism. Could you explain that? 21:19 Because again, I imagine same technical issues on the back end. Yeah. Same, same kind of issue. 21:26 One of the things that I did in, uh, in one of the chapters in More than a Glitch is I took my own mammograms, and I ran them through an open source, uh, cancer detection AI in order to write about the state of the art in AI-based cancer detection. 21:43 So if you, uh, were to read the mainstream media, you might have developed the impression that, uh, AI-based cancer detection is right around the corner, right? Sure. And the reality is way more complicated than that. 21:57 I got interested in this because I was diagnosed with breast cancer a few years ago. I'm totally fine now. I'm really grateful to all of the, uh, all of the doctors and the medical professionals who took care of me. 22:08 But one of the ways that I reacted, uh, when I got this news is I got really obsessed about reading everything I could, and especially about reading my entire electronic medical record. 22:19 And so buried deep down in, uh, in the electronic medical record, I found this note that said, uh, "This scan was read by Dr. So-and-So and also by an AI." I thought, "What?" 22:34 Like, "What did this AI find, and who wrote this AI? What kind of bias is embedded in it?" Right? Oh, yeah. Because all AI has human bias embedded in it. You know, what are the implications for my care? 22:47 I had a lot of questions, but I kind of didn't pursue it at the time because, you know, I had cancer. I was kinda busy. Uh, but then once I was a little bit recovered- In the middle of COVID, right? Yeah. Yeah. Yeah. 22:58 I, I got diagnosed- So lots going on... uh, I had surgery, and then there was COVID, and it was just, like, a huge nightmare. Yes. Uh, disaster on so many levels. 23:06 So what I did in the middle of all of this, uh, when I was feeling a little better, is I decided to do this experiment. 23:14 Uh, it was a replication experiment with an N of 1, uh, where I was gonna take my mammograms, which I knew-- Like, I knew there was cancer in them- Mm... 23:24 and I was gonna run them through this AI in order to write about the state of the art. Uh, well, it turns out that, uh, there were a couple things going on. 23:32 I was completely wrong about how I imagined that the AI would work- Mm... which was really interesting and sort of instructive, because many of us have, uh, completely wrong ideas about how AI works, right? 23:47 So I thought that it would take my entire medical record, like my whole chart, all of my tests- Mm... 23:54 uh, all of my notes from all of my doctors, all of my images, the, you know, the flat images and the videos and the ultrasounds and the mammograms, that I thought it would take all of this, process it, and then give me, like, a diagnosis. 24:10 Mm-hmm. Totally wrong. What it does- Mm-hmm... is it takes one flat image, right? Just kind of a, it's a, I don't know, it's a semicircle with kind of blobs inside of it. Mm-hmm. 24:21 It's kinda X-ray looking.What it does is it takes this single flat image and draws a circle or a square around an area of concern, 24:32 and then it gives a score, not a percent chance that something is malignant, but a score between zero and one. And the doctor, uh, is given this image with the circle or square and the score. 24:45 Importantly, the, uh, the radiologist does not have access to this data until after they've entered in their own diagnosis, right? Mm-hmm. 24:55 So the process is, you know, you go, you get your mammogram, the mammogram data gets sent to the radiologist, the radio-- human radiologist reads it, enters in what they think is going on, and then after they press save, they get this AI image that has a circle or square in it. 25:15 Sure. Totally different than what I expected. So- Right... the-- 25:20 if you had gotten the idea that AI-based cancer detection is around the corner, like, you would be totally justified at that based on mainstream media coverage, but that's also wrong, right? 25:32 It's not actually around the corner. There are all kinds- Mm-hmm... of issues that have not yet been worked out. 25:38 One of the issues that, that most people don't realize is that all AI systems, uh, because AI is just math, right? These are mathematical systems. 25:48 They're-- uh, and the AI is giving you a statistical likelihood that there is something malignant on this, uh, on this image. 25:55 Well, every AI system is set to, uh, have either a higher rate of false positives or a higher rate of false negatives. Mm-hmm. I thought this was so interesting. It's so fascinating, right? Yeah, yeah. 26:08 And people usually don't think about this. We think about AI- Mm-hmm... as, you know, as being a kind of entity, or we think about the Hollywood stuff around AI, but that's, that's not how it works. It's just math. 26:19 So this particular math, uh, has a setting. Now, a false positive in cancer would mean that the system says, "Eh, maybe cancer," but there is no cancer. 26:33 And a false negative would be the system says, "Nope, no cancer," and there is actually cancer. 26:39 So people have agreed rightfully that the, uh, the cost of a false negative in cancer detection is way higher than the cost of a false positive, right? We would much rather have a false positive than a false negative. 26:55 And- Totally... you know, both of them are terrible by the way. Um- Sure... getting a false positive is terrible. 27:02 But one doesn't result in you turning out-- like, basically going home and going about your day, and it turns out you have cancer. Right. Right. 27:09 So false negative, you like go home, you go about your day, then like eventually, you know, it gets you. But a false- Yeah... 27:15 positive means they're like, "Oh, well, there might be something here," and then you have to go into the pipeline of additional tests, which I can tell you- Sure... 27:21 is a, like, terrible pipeline, and you have several weeks usually of uncertainty, uh, and fear and terror. Uh, so both situations are really bad. 27:34 Uh, we also have to think at this point about the, uh, about the funding because, uh, there are economic incentives always, uh, when we're talking about AI. 27:46 So let's think about who gets paid, uh, when radiologists read films and when AI reads films. So an AI reads a film, and nobody gets paid, right? 27:59 A radiologist reads a film, and the hospital gets paid, right? Because the radiologist is, you know, is a human being, is a laborer. 28:09 And so let's think about, okay, well, who's incentivized to, uh, to have more AI reading films over radiologists reading films? Well, it's not the patients, it's the insurance companies. 28:23 Because the insurance companies are the ones who are paying when, uh, you know, paying the hospitals for the radiologist to read films. 28:30 The hospitals certainly are not incentivized because the hospitals get paid when the radiologists [laughs] work. Uh, and I can say from the perspective of a cancer patient, I do not want, 28:44 uh, a medical system where, you know, you kind of step up and, like, have your ex- your test in a machine, and then, you know, you get like a printout that tells you the result. Like, that's terrible. 28:57 I want a medical professional who is interpreting the results for me, who's telling me things, who's caring for me, uh, especially when it's a really high stakes decision like cancer. 29:07 So we really need to think harder about when it's appropriate to use AI. Uh, it's not just a matter of can we, but it's a matter of should we. 29:18 Like, absolutely, yes, we should use all kinds of mechanisms to try and have earlier detection and better detection, but the- Totally... 29:26 mechanisms that we're using right now are not necessarily going to be our salvation. I really appreciate the nuance you bring to this, and obviously, your personal experiences enhance that and, and complement it. 29:42 I've, I've had a few really wonderful semi-related conversations over the past year, um, quite a few in private and a few in public that, that I'm thinking about right now, and one is, um, with an author, uh, Professor Susan Liautaud at Stanford, who wrote a book, uh, and teaches about basically ethics in a rapidly changing world and, and building and using frameworks more often in your life, so that when you encounter things on the margins, like we're constantly encountering, that you're more ready to apply those and be ready to tackle those. 30:15 Um, and another one with, uh, Professor Emma Pierson at Cornell Tech, uh, I believe, um, who, uh, as she put it, and I'll probably mangle that, she likes to use new technology like this and machine learning and such to answer old hard questions. 30:32 Um, it was great conversation. So I've read most of your work, and obviously, we're talking here, and there's-- you've given so many wonderful interviews. 30:40 But I, you know, I come back to this onePart of a, uh, I think it was MIT who just in sort of a throwaway line said, 30:47 despite or I guess probably because you're a, a working data scientist and journalist, you spend a lot of time thinking about problems that mathematics can't solve. 30:58 And I guess as always, you know, with anything, we could say satellites or whatever it might be, or, or, uh, carbon removal, we can obviously throw the word yet at the end of that because things do change and we, we pursue them whether or not we're paying the cost along the way. 31:12 But I wonder if you can help me understand that and how you do that while still not stepping fully into pessimism or doomerism or what it may be. How do we practically push forward, but 31:26 help people understand, like you were saying, this is not the thing yet, especially for something as, as, uh, with so many stakes like healthcare? Well, there's a certain way that privilege operates inside mathematics, 31:40 uh, and computer science is a descendant of mathematics. Uh, and this, uh, this kind of... 31:49 This, this privilege, the way it operates is that when you are an elite mathematician or an elite computer scientist, nobody tells you no, right? Which is super weird because everywhere else in the world- Yeah... 32:03 people tell you no all the time. But for some reason, people, uh, I mean, probably because mathematicians have, have, like, 32:13 put out this idea for centuries, like they've put out this idea that math is superior to- Sure... uh, to other disciplines, and that- Yeah... 32:22 you know, to be a really good mathematician, you deserve to, like, go off and kind of not be bothered with the petty concerns of real life, that you just need to, like, live this life of the mind, uh, in the ivory tower. 32:38 And, you know, it's really great if you can do that, but, you know, not everybody can. And- Mm-hmm... uh, it's also just not, it's not very responsible, uh, at a certain point. 32:50 So this, this kind of, kind of self-obsession that mathematics has turns into something that I call techno-chauvinism, the idea that technological solutions, mathematical solutions are superior to others. 33:06 And what I, I would argue is that we need to add nuance to that. 33:09 We need to, uh, think about using the right tool for the task, because sometimes the right tool for the task is undoubtedly a computer, and sometimes it's something simple like a book in the hands of a child sitting on a parent's lap. 33:22 You know, one is not inherently better than the other. So, uh, because mathematicians are rarely told, uh, no, uh, 33:32 you know, computer scientists are also rarely told no, uh, you know, you start to think when you're in math or computer science or physics or, you know, these related disciplines, you start to think that, "Oh, well, I am always going to have the answer for everything." 33:49 Uh, and that is not a really safe way to operate, you know? I mean- Sure... let's think about Shakespeare. 33:58 Like, there's this thing called hubris, and we can then look at artificial intelligence systems, and we can look at the ways that AI systems are actually harming people right now. 34:10 So take the case of facial recognition, for example. Sure. Uh, facial recognition is a kind of AI. Well, what, uh, was discovered is that facial recognition is better at recognizing men than women. 34:22 Uh, it's better at recognizing light skin than dark skin. It is best of all at recognizing men with light skin. It's worst of all at recognizing women with dark skin. 34:31 Uh, and trans and non-binary folks are generally not recognized at all, right? So this is a really biased technology. 34:39 The reason for this bias, uh, sometimes has to do with the training data that is, uh, that is used to train the facial recognition system, right? 34:50 So if you don't have a sufficient variety of skin tones in the training data, then the system itself is not going to, uh, you know, is not going to recognize a wide range of skin tones. 35:03 Because again, we're talking about mathematical systems here that are constructed, and they are constructed by people. 35:10 The way we make a machine learning system like a facial recognition system is the same every single time. 35:17 What we do is we take a whole bunch of data, as much data as we can find, and we plunk it into the computer, and we say, "Computer, make a model." Computer makes a model. 35:25 The model shows the mathematical patterns in the data, and then we can use that model to do all kinds of cool things, to make predictions or decisions or generate new text or images or audio or video, right? 35:38 Uh, so the facial recognition system is trained on data, you know, that shows human faces, and then it recognizes human faces. 35:47 And again, the problem can be that does not recognize a sufficient range of skin tones because of the training data, right? 35:55 So most people would look at this and say, "Well, okay, let's just put in, you know, better training data," right? And yes, this would make a facial recognition system more accurate. 36:05 But this is not going to solve the problem entirely, because then we have to think about the ways that facial recognition is used. So we have low risk and high risk uses of facial recognition. 36:17 Now, a low risk use of facial recognition is probably something like using facial recognition on my phone, uh, to unlock it. 36:24 Usually doesn't work for me, but it's not really a big deal because I just put in my passcode and, you know, I go on about my day. Uh, so I would call that a low risk use. 36:33 But a high risk use of facial recognition might be police using facial recognition on real-time video surveillance feeds, uh, because what is that gonna do? 36:44 Well, it's gonna misidentify women and people of color more often and is going to get more people caught up in the justice system without cause.Right? So that's going to have extremely negative effects. 36:58 And I would argue that this is actually a case where we should not be using facial recognition in policing at all. Yeah. I mean, [sighs] it's easy to say that's one example, but, 37:11 uh, these, these systems that have been sold to police departments and the TSA and things like that, they're catastrophically bad at the job they're supposed to do. Like, we already know that. Uh-huh. 37:22 Um, but even then, ones that aren't, I guess, as, um, m- publicly, uh, evident how bad they are, again, like you said, these systems have been designed and implemented and used at scale for, for years now. 37:38 Um, again, you've got all the pre-crime type stuff everyone's tried to do with facial recognition or, um, uh, you know, just general policing where they... Uh, God, who's the guy who was, um... You wrote about him. 37:49 Uh, the fellow who was commissioner of police in both LA and New York that made the whole, um, sort of pre-crime type system? Yep. Yep. 37:58 Uh, one of the ways that this, uh, that this manifested is it actually ended up in Hollywood. So if you have watched The Wire, there's this, uh, season where- Right... 38:09 they focus on the police, and there's a system called COMPSTAT. Uh- Mm-hmm... this is a, uh... 38:15 I mean, for a, for a data nerd like me, I really liked this, uh, this plot line because you saw the absurdity of predictive policing, right? Mm-hmm. 38:24 So the idea behind predictive policing is that we take crime data and put it in the computer and say, "Okay, computer, predict based on crime patterns in the past, predict where there's going to be crime in the future." 38:37 Uh- Mm-hmm... and it creates this, uh, this really toxic feedback loop because- Sure... when you say, "All right, where has crime happened in the past?" 38:46 Well, crime data is actually not crime data, it's arrest data, right? Right. So cr- crimes, like drug crimes, for example, happen in, you know, in Black and white and every community, like, at about the same rate. 39:00 It's just that Black and brown communities are overpoliced and have been for a long, long, long time. What the data shows is that, uh, you know, there's a lot of arrests happening in Black and brown communities. 39:13 When you feed that into the computer, the computer says, "Oh, there's, uh, there's all of th- these arrests happening here, so we should send more police to the Black and brown communities," which exacerbates the problem. 39:25 And then what happens- Mm-hmm... when you deploy more police? Well- Mm-hmm... they find more crime, so there's more arrests. So then that gets fed back in, and the computer says, "Oh, need more police." 39:38 It's not a beneficial system. No. And, and it's very easy... Again, they're very different, but you can swap those out, and, and we've seen this, uh, over and over again with mortgages, with other types of loans. 39:49 Like, why do you think Black people are coded into the system as [chuckles] not having the, more mortgages in the past? Like, how much r- you know, historical research do we need to do on, on redlining and everything? 39:59 Uh, I don't want you to rehash your entire book, but you o- obviously talked about the incredible soap dispenser, uh, example, which seems so simple on its face and yet is so emblematic of everything else. 40:10 But there's one that really horrified me. 40:12 And look, my kids are young-ish, though apparently older than I thought at this point, um, every day, and they did remote school during COVID, like everyone else, incredibly privileged to have internet. 40:24 Uh, mom and dad could work from home. They had their, their, uh, whatever the little Google laptops that the school gave out were, you know, all of those things. 40:31 They were very lucky to have all that, which many children in the US don't, um, uh, much less internationally. But you told this story about a predictive grading nightmare, um, that 40:46 I, I'm truly, I guess, astonished that it was used in practice. 40:51 I understand everyone was really flailing, scraping the barrel, whatever [chuckles] you wanna call it in COVID to figure out how we do these things that, that, that we had taken for granted, and again, provided the baselines for college acceptance and all this stuff for years. 41:03 They had to figure out to do something. And then everybody ditches the SATs and the ACTs and all these things, and now it swings back. 41:10 But I wonder if we can talk a little bit about this predictive grading story 'cause, again, it's emblematic of workers being monitored, students being monitored, all of these different things that 41:21 once there's this type of strike on your record, again, like with the recidivism algorithms and mortgages, it's very difficult to make those go away. 41:31 Um, just because facial recognition, uh, gets somebody wrong, usually a, a, a Black man, I mean, you told a bunch of stories about that in there, and we've heard many of them, um, and he might be released a few hours or a few days later, um, that, that's, that's still on his record in, in some way, and this predictive grading thing was, was so terrifying to me. 41:51 Could you talk a little bit about how that happened and why, and I don't even know. I just... I, I was like, "I have to ta- we have to talk about this." [chuckles] It's so upsetting. It's totally horrifying, isn't it? 42:01 Uh, and I mean, I'm a pro- These are kids. Yeah, exactly. Like, I'm a professor. Like, I give students grades. Like, I think a lot about grading. 42:09 I think a lot about, uh, what is my duty of care as a professor, and this, uh, this particular episode seemed to violate everything that I believe about education. 42:21 Um, so what happened is during the, uh, pandemic, the International Baccalaureate Organization, which is this organization that awards a very prestigious secondary school diploma, uh, decided that they were not going to be able to administer in-person IB exams. 42:36 Uh, which, you know- And totally get it, right? Totally appropriate because no- Totally get it... we were not, like, putting children- Yeah... in rooms with each other at that point. 42:44 Because they couldn't administer the in-person exams, they decided they were gonna use an algorithm to assign imaginary grades to real students, right? 42:56 Now, what happened is they used a machine learning systemRight? So, uh, as I said before, the way the machine learning system works is you dump in the data and then you, uh, you know, it makes a model. 43:10 The model predicts the grade that the student would have gotten had they taken the exam that was canceled, right? 43:19 Which is already completely absurd, but, you know, it's the pandemic, people are making all kinds of wacky decisions. 43:25 Uh, but it ran into, uh, a particular, uh, characteristic of education statistics, which, you know, if you've studied education statistics, you know that the way that it shakes out in the US is that, uh, students from wealthy schools tend to do well, and students from poor schools tend to do poorly. 43:49 Right? And well, let's think about race and class. Who's at the wealthier schools? Well, it's white students. Who's at the poorer schools? It's Black and brown students. 43:58 So the machine goes in a- and, you know, looks at all of this data and assigns grades. 44:04 What it did was it assigned a prediction of a failing grade to students who went to poorer schools, and a prediction of passing grades to students who went to wealthier schools. 44:16 And well, then that broke down around, you know, race and class lines. 44:20 And so I wrote about the case of a young woman named Isabel Castaneda, who was a straight A student, a heritage Spanish speaker, and her teachers had predicted that she was going to, you know, pass with flying colors, get great grades on all of her IB exams. 44:41 And the, uh, the algorithm predicted that she was going to fail her Spanish exam, right? Which is completely absurd. 44:48 So I mean, it just illustrates one of the big, uh, points that I'm trying to make in the book, which is that machines do not always make good decisions, and we really need to be cautious about turning over decision-making to machines. 45:05 And if you are going to have a machine make a decision, one of the things that you need is you need a mechanism for redress, right? You need to- Mm-hmm... 45:15 design a process for when the machine goes wrong and does something wrong, and you need to, you know, have a human being who's empowered to go in, make a different decision, and change the outcome and update all of the records. 45:30 Which by the way, is an expensive and time-consuming process, and- Mm-hmm... 45:36 you might actually be better off not firing the person who made the decision in the first place and replacing them with a computer, 'cause then you're just gonna have to like hire a com- a person to clean up after the computer makes a mess. 45:49 Yeah. It's a hor- o- obviously, any of these conditions are a nightmare. 45:52 Anyone being arrested, being denied a mortgage or insurance or a loan or whatever it might be, um, you know, it's so easy to go down both broadly and be like, "We should never use it," um, and on the other hand, go down rabbit holes on each of these and think of the implications that come from these things. 46:08 Um, again, like just the number of Black men in prison, and like what has that done to not just their personal lives and their ability to vote and get loans, um, but also to redistrict- I mean, the, the whole thing. 46:20 Like we can do it, we can do it all day. The kids thing always, whether it's healthcare, um, 46:25 Medicaid, education, whatever it might be, my kids were part of the LA school system for a while, and they were very lucky to go to a, one of the few very good public elementary schools where the teachers and administration did everything they possibly could to make this a, a nourishing and a fruitful experience for these kids. 46:45 But I'm under no, uh... I mean, I, I, I got as engaged as someone can be for a, a school system that's, you know, depending on how many kids are actually going to school, five hundred to six hundred thousand students. 46:58 It's, it's a behemoth. And um, you know, it's, you don't get that time back, you know? 47:04 And this young woman doesn't get that test back, especially as you said, if it's, um, bandwidth-wise and, and profits-wise easier for the company or, or whatever it might be, um, the public agency to not fix the problem, right? 47:20 Whether it's that individual, which then they're opening up a whole can of worms to them, or just in general by changing, um, like you said, changing a field from Boolean to a, to an editable string. 47:31 Like I understand that's a lot of work, but sometimes we have to do this work and, and that's where we are. And again, it's easier for society to... 47:39 Um, it seems insane because we've had so many [laughs] society, uh, three steps forward, seven steps back. 47:45 But it, it seems sometimes that it's easier for the societal part to move ahead than, than some of these systems where it's still so hard-coded. 47:52 And like you said, the technical debt is just outrageous 'cause these, some of these programmers still speak, uh, ancient languages. 47:59 You know, if you've ever stood on the side of an, an airport desk and someone's trying to help you and you happen to peek at their monitor, and you go like, "Oh my God. Why is it black and green?" 48:07 Like, how is that still the thing we're using to keep everyone in the air? And that's the safest system we got. Yeah, no, I totally agree. And, uh, it's, it's amazing to me that these systems work as well as they do. 48:20 And- Mm-hmm... uh, and like as far as I'm concerned, they're just, they're barely functioning. Like they're held- Sure... 48:26 together with bubblegum and duct tape, and I just, I don't think that people should have quite as much faith in them as they do because 48:37 like there's not much standing between us and total chaos when it comes to computational- Yeah... systems. 48:44 And again, I appreciate even, even despite that, how you work so hard to not go all the way to doomerism on these things, you know, to, to luddite, because again, there are incredible examples of these things changing the world. 48:57 You know, the ability for the scientists in, in w- whenever they did it, was it January or February 2020 when they uploaded the, you know, the, the genetics of SARS-CoV-2 to the internet within a day or something? 49:09 You're like, tell that to our, to our grandparents. [laughs] You know, explain that one. Good luck. 49:13 Um, it's incredible, so that the whole world could access it, and we could build these revolutionary new-mRNA vaccines that so many other people worked for decades to make safe. Um, you know, it can do amazing things. 49:24 It will do amazing things, but like you said, the sort of the blind faith, um, if not by some venture capitalist eager, uh, disregard, intentional public disregard for trust and safety and things like that, um, 49:41 really feels like it just starts to benefit the same folks and hurt the same folks, uh, along the way, if, if not at, at more scale. So how can we help? Like, what is most actionable to you? 49:52 Um, what-- where can people get most educated on these things? Where can they, um, whatever their skills or vocation, uh, volunteer or donate, um, or participate in some way? 50:05 W-where can we actually start to make some progress on these things? Is it internally at their own work? Is it in other places? 50:11 I'm, I'm super curious 'cause, again, you seem keen on not just, "Hey, this isn't ready yet," but also saying like, "But this is-- these are the things we need to do." 50:18 I mean, you spent the whole last chapter of the book that way. Well, the first thing that you can do is you could buy my book. That is the- Yeah. Great... that is the most- Oh, yeah... important first step. 50:26 Moral of the story. Yes. Uh, so Artificial Unintelligence: How Computers Misunderstand the World was my previous book, uh, and that is about the inner workings and the outer limits of computers. 50:39 And then More Than a Glitch: Confronting Race, Gender, and Ability Bias in Tech, uh, picks up where Artificial Unintelligence left off and takes a deeper look at society and the way that very human problems play out inside computational systems. 50:55 Uh, and both books, uh, end on a high note because, you know, this is-- it's kind of a lot to deal with. Uh, I look at the technical realities of, uh, of modern computer systems. 51:07 I look at some of the disasters that have happened, uh, and I talk about how can we learn from these disasters, but I also talk about, okay, c- what can we do differently? 51:17 Uh, so it starts with empowering ourselves, uh, learning a little bit more about technology, uh, so that we can push back when algorithmic decisions are unjust or unfair. 51:29 Also, uh, we need some action on the policy front. You know, we need, uh, better data privacy laws. We need, uh, better, uh, you know, better tech policy overall. 51:43 Uh, there have been some really great strides made in Washington, uh, around this. 51:49 I, I point people to the blueprint for an AI Bill of Rights, which came out of the Biden White House Office of Science and Technology Policy. 51:56 Uh, President Biden's, uh, executive order on AI, uh, that came out recently is also a really great leap forward, and we've seen some motion in many government, uh, US government departments, uh, toward wrangling AI a little bit better. 52:12 So at NIST, we have the AI Risk Management Framework. Uh, the GAO has done, uh, an algorithmic audit and, uh, you know, put out some, uh, guidelines for using algorithms, you know, in government contexts. 52:28 So these are pretty impressive leaps forward. We still have a ways to go, though. 52:33 Uh, so there are-- there's legislation like the Algorithmic Accountability Act that has not been passed but, you know, ideally will be in the future. So it's about individual and collective action. 52:47 It starts with learning more, and then, uh, it's about making different decisions than we have in the past. Uh, yeah. It all sounds so easy when you put it that way. 52:58 Um, what other books do you recommend, uh, Weapons of Math Destruction? Any, anyone else or, I mean, it's twenty twenty-four, YouTube videos, anything that, that you really feel will help people get up to speed on this? 53:12 I designed the bibliography of both of my most recent books as, uh, as reading lists. Uh, so I would- [laughs]... I would go through the bibliographies. 53:22 Uh, ones that I usually point people to are, uh, Weapons of Math Destruction, as you said, uh, Safiya Noble's book, Algorithms of Oppression. Uh, Virginia Eubanks, uh, has a book. 53:35 So Ruha Benjamin's book, Race After Technology, uh, Charlton McIlwain's Black Software. 53:42 Uh, a couple of influencers who I like, uh, Avriel Epps, uh, is doing some really interesting stuff around, uh, understanding algorithms. 53:53 Uh, there's a guy named Joel Bervell who does, uh, social media stuff about, uh, racial bias in medicine. Uh, I've learned a lot that way. Uh, so there are many, many resources out there. Um, thank you. That's great. 54:09 And yeah, we'll definitely point folks towards your bibliographies. Is it Automating Inequality? Yes, that's the one. Perfect. Thank you. By Virginia Eubanks. Fantastic book. 54:17 We'll s- we'll use AI to sub that in for where you said- Yeah. No, I'm kidding. We'll- Great... we'll not do that. Um, yeah. Virginia, I'm so sorry. I lost the, lost the thread there for a second. 54:25 [laughs] You're, you're a monster. Um, okay, last couple questions, uh, we ask everyone, and then I'm gonna get you out of here if that's okay. Does that work for you? 54:33 Meredith, uh, when was the first time in your life, uh, when you realized you had the power of change or the power to do something meaningful? 54:43 And folks tell us about running for student council, uh, as, as a kid or, or, you know, inventing something the first time. It could be, could be anything. 54:52 So I think one of the first times that I realized that I had power, uh, was, uh, a story that I tell in the beginning of Artificial Unintelligence, which is, uh, when I got an Erector Set for, uh, Christmas or my birthday or something when I was little, and I wanted to build this robot. 55:09 And, like, I thought the robot was gonna be my new best friend, and we were gonna, like, dance and party together, and it was gonna, you know, be alive. 55:16 And I built this robot with my little, you know, littleTiny, teeny tiny kid sized wrenches and what have you, and it had a motor. 55:24 I was very excited about the motor, and I put the motor in, I flipped it on, and nothing happened. Like, I started tinkering with it. I got my mom, I was like, "Mom, the, you know, the robot is not working." 55:34 And she said, "Well, did you turn it off and turn it on again?" I said, "Yes." She said, "Did you flip the batteries?" I said, "Yes." And so then she went and did the same thing. She was just like, "Oh, it's broken." 55:44 So I realized at that point there was a really big gap between what I imagined the technology would do and what the technology could actually do. 55:54 And so I've gone back to this, uh, this moment many times over the years, uh, and I use it as a way of thinking through what the technology can and can't do, and, uh, as a way of thinking about the power that humans have, the power that humans have when they're assisted by tools like motors and, you know, little kid sized metal wrenches. 56:20 Uh, but also the, the limit to that, the fact that, you know, that power is not unlimited. I love that. 56:29 Um, it turns out most of being an adult is managing expectations, no-- almost no matter what your job is or whether you have kids- Yeah... kids or you're a teacher or not. 56:37 [laughs] It really, uh, boy, does that, does that, uh, does that come in handy. Um, thank you for sharing that. I love director sets so much. Those things were- They're so great... amazing. 56:49 But you're right, expectations out the window, uh, for, for who this little friend was gonna be. Um, Meredith, who is someone in your life that has positively impacted your work in the past six months? 57:01 I have been talking to a student group, they are called Encoding Justice. Uh, and they are doing a bunch of really great work about, uh, surveillance in schools. 57:16 They're opposed to surveillance in schools. Uh, and so I am really grateful that they have been sharing their wisdom with me. I am doing a project about, uh, school surveillance. 57:29 Uh, and I have, uh, I have just learned a lot from their efforts. 57:33 Uh, they have a really interesting platform that they have put out around things that, uh, that young people think can be done right now to, uh, to change our AI driven world for the better. 57:47 Uh, so I would definitely recommend checking out Encoding Justice. I, I will do that immediately. I think that's great and, uh, you know, just such a travesty. It is such an u-unforced error of what we need to do. 57:59 I, I mean, even just down to the mental health of these kids, it's such a nightmare. Um, Meredith, last one. 58:05 What is, uh, a book you've read in the past year or so that has changed you in some way, uh, changed your thinking or opened your mind to some topic you hadn't considered before? 58:15 Or frankly, I mean, I just basically read about dragons at night at this point because it turns my brain off, so it could be a coloring book, I don't care. 58:21 We have a whole list on Bookshop and, uh, and the pe- and the people love it, so. The most spectacular book that I've read recently is All-Star Chain Gang. Uh- Okay... 58:30 and it is a kind of post-apocalyptic, uh, book about, uh, the carceral state and, uh, about terrible possible future. And it just-- it moved me and it made me think differently about the world. I love that. 58:47 Thank you for sharing that. I'm so impressed you can remember the last book you read. I, I mean, it truly goes like [whistles] right through me. Um, if I don't write it down, it didn't happen. 58:55 Um, Meredith, where can our, uh, listeners and readers and such, uh, follow your, you and your team's work? Uh, I'm on all the social platforms @merrabroussard. 59:07 Uh, my website is meredithbroussard.com, and you can find More Than a Glitch or Artificial Unintelligence anywhere that books are sold. Perfect. 59:14 Um, thank you so much for all this and for writing these books and, and talking so honestly about your own experiences and how, again, they are, uh, part and parcel of, of everything else. 59:26 Not that they're not obviously, uh, damaging on their own, but this is, uh, this is, this is where we are and what we need to dig out of to use these tools, uh, in a much healthier, equitable, useful way for, for more folks. 59:41 So I really, really appreciate everything you do and for your time today. Thank you so much. [upbeat music] That's it for this week's conversation. 59:51 For more conversations, scroll back in the feed or visit podcast.importantnotimportant.com to search by name, topic, whatever. Thanks for sharing, thanks for leaving a review, and thanks for giving a shit. 1:00:06 [upbeat music]