The United States Supreme Court
The University of Michigan Admissions Lawsuits

Grutter v. Bollinger
THE FACTS

From the Cross-Examination of Grutter's statistician:
Are The Odds Ratios He Computed Valid Measures
of the Racial Preference in Michigan's Admissions?

Plaintiff's statistical expert, Dr. Kinley Larntz, employed several methods purporting to show the amount of preference that "underrepresented" minority applicants receive. One method presented the racial differences in the rate of admission for applicants with similar academic achievement (as measured by undergraduate grades and LSAT scores). At some levels of achievement, the admission rate for favored minority applicants was 20 times as high as the rate for other applicants. These data were not disputed in cross-examination. A second method compared racial differences in the odds of admission. For each of six years, the relative odds of acceptance were calculated for applicants from each racial/ethnic group as compared with Whites, controlling for grades and test scores. In 2000, the odds of admission for Black over White applicants reached 443 times as great. (The complete cross-examination is online here.)

Dr. Larntz's calculations were vigorously challenged in cross-examination by Stuart Delery, counsel for Defendant. If one group's rate of admission is 5%, we know that no group's rate can be more than 20 times that large, since no rate can be greater than 100%. What does it mean, then, to say that the odds of admission for one group are several hundred times the odds of admission for another? Controlling for grades and scores, the Black over White odds ratio in different years ranged from 53 to 443. When additional factors were controlled, the B/W odds ratio shot up even farther. Are ratios of such magnitude and fluctuation a valid measure of the law school's use of race?

1/17/01 - BENCH TRIAL - VOLUME II

[Page 116] CROSS-EXAMINATION BY MR. DELERY:

Q. Good afternoon, Dr. Larntz.

A. Good afternoon.

Q. We have met before, isn't that right?

A. That's correct.

Q. I took your deposition in this case back in February of 1999? A long time ago.

A. I'm sure that's right.

Q. You said this morning, I believe that the purpose of your analysis was to look at the role that race plays in admission to the Law School, is that a fair statement of your approach? [117]

A. Role in a statistical sense, yes.

Q. In your view, is that the same as extent to which race is considered in admission?

A. Well, I'm not sure I understand an awful lot of difference. What I'm saying is that what I did was try to understand and describe Admissions decisions and understand to the best that I could, I call it role. And I'm not sure, I guess I should be an English major to understand the extent of the difference.

Q. The reason I ask is, you maybe you haven't been told this, it's one of the questions the Court has put to the parties for trial here, is the extent to which race is used in the Law School admission process, are you aware of that?

A. I wasn't aware of the specific questions, no.

Q. All right. You believe that your analysis quantifies the role that race plays in the admissions process, is that right?

A. That's exactly what I think I did, was to try to describe the Admissions decisions, and the role in a statistical sense; and again, I'm doing this in a statistical sense.

* * * *

Q. You believe that your odds ratios give an accurate picture of the role that race is playing in admission?

A. With respect to the individuals with similar credentials, GPA and LSAT and residency and gender [120] and fee waiver status, I think the odds ratio gives a good composite measure which summarizes the extent to which--extent, I use your word, to which race is being used, yes.

* * * *

[Inserted here is a comparison of probabilities and odds, initially presented by the witness under direct examination, which is pertinent to the ensuing dialogue. Ed.]

Q. Now, this morning Judge Friedman asked whether--actually if we could put up slide 30 from this morning. I think that's what was up when the question came. This morning Judge Friedman asked whether this 81 odds ratio for Mexican Americans here, meant that Mexican Americans were 81 times more likely to be admitted than white students, do you remember that discussion?

A. Yes.

Q. And, in fact, the answer is no, that does not mean that the Mexican Americans are 81 times more likely to be admitted than the white students, isn't that correct?

A. If we mean times likely in terms of 81 times probability, if that's what you're meaning, then that's correct. It's, in fact, an odds multiplier [122] and so it, in fact, multiplies the odds depending on what the base line odds would be.

Q. Okay. Let's go back to this, your drawing from this morning, if we could. I think you said that this 81 for Mexican Americans approximates basically the lower half here of what you sketched out, is that right?

A. That's correct. 81, yes.

Q. So, the 81 odds ratio translates to a ratio of probabilities of .9 to .1, is that right?

A. Ten percent probability using an odds ratio of 81 would become 9 to1 probability, that's true.

Q. So, to place that information into the sentence that we used before, for these numbers we would say that the group with the 90 percent probability of admission is nine times as likely to be admitted as the group with the ten percent probability, is that right?

A. If you're using it in terms of probability, that's in terms of probability.

Q. Okay. So, the likely language refers to probability, in your view, as commonly used by statisticians?

A. No, as commonly used by statisticians, statisticians will work in terms of odds. If I might say, I'm [123] sorry. With respect to analysis of binary response we work in terms of odds and odd multipliers or odds ratios.

Q. You work in terms of odds, but let me go back to the sentence earlier. Is it fair to say that this example here with the 81 odds ratio, indicates that the group with the 90 percent probability is nine times as likely to be admitted, say, the group with the ten percent probability?

A. What I would say: it's nine times the probability; that's what I would say. We particularly don't use likely, because it's subject to all kinds of misinterpretation.

Q. All right. And just to take another example, the one that you have up here at the top half, the 75 percent chance or 75 percent probability versus 25 percent, the odds ratio comes out to nine. But we would say that the probability is only three times greater for the group of 75 percent chance, isn't that right?

A. In terms of probability, that's right.

Q. Okay. I think I just want to get this clear, because as you say there's room for misinterpretation. And you would agree that we should try to be precise in the language that we [124] use. So when we're talking about odds, we should use odds type of language?

A. As best I can, I will try to be specific as that, yes.

Q. Let me do another example just to get a sense of the relationship between probabilities and odds. If one group has a 99 percent probability of getting admitted, and a second group has a 90 percent probability of getting admitted, so .99 versus .90. Am I correct that the odds ratio for that is eleven, or about eleven?

A. Well, we can do the math.

Q. Let me get a pen here. .99 versus .90, let me just make sure I have this right. The formula would be for the odds ratio .99 over .01 all divided by .90 over .10, is that right?

A. That looks good.

Q. Okay. And this comes out to about eleven, doesn't it? You can check me with your calculator, if you like? [ .99 / .01 = 99; .9 / .1 = 9 99 / 9 = 11 Ed. ]

A. I think eleven is a good number.

Q. So, we get an odds ratio of eleven even though the probabilities are very close, right?

A. The probability of acceptance is close, and the [125] probability of denial is quite different.

Q. The probability of acceptance is--let me just put it this way. Both groups are highly likely to be admitted in that case?

A. Highly likely to be admitted?

Q. Yes. .90 versus .99?

A. Yes.

Q. Okay. And yet you end up with an odds ratio of eleven?

A. That's true.

Q. When you indicated earlier that two or three in your experience is a very large odds ratio?

A. Yes. And if the Court permits I can explain. The reason, of course, is that you also can look at the chance of denial. And the chances of being denied admission for these are one percent versus ten percent. And so there's a symmetry with respect to that, and that's why we use odds in statistics. And so if you look at the chances of denial, it's one percent versus ten percent which is quite discrepant.

* * * *

Q. Now, you have mentioned a couple of times, and certainly many times this morning, the idea that you wanted to look at applicants with similar credentials. Is it fair to say that that was one of the basic principals of your analysis, you try to identify the students with similar credentials and [130] compare those?

A. I think I took as a basic principal of my analysis that I would use, the groupings as set by the Law School itself to define groups like that, and that's where I started from.

Q. And when you say groupings as defined by the Law School itself, you mean in the grids in Exhibit 16 that you were given?

A. That's correct.

Q. You didn't get those groupings from any other source?

A. The groupings came directly out of Exhibit 16.

Q. For example, you didn't review the deposition testimony of the Admissions officers who actually make the decision when deciding on the structure for analysis, did you?

A. I don't recall such review.

Q. And when you use the term credentials here today, you generally mean GPA and LSAT scores, isn't that right?

A. For the most of our analysis GPA, LSAT. In some sense we did analysis involving residence, gender, fee waiver.

Q. Did you consider residence or gender or fee waiver to be credentials when you used the term? [131]

A. They're characteristics of the applicants.

Q. But when you talk about credentials, you're talking about grades and test scores, isn't that right?

A. That's what I'm talking about.

* * * *

[175]
Q. All right. I believe you said a number of times that your goal in this analysis was as much as you could, to look at students who were similarly situated in terms of credentials as you call it, correct?

A. That's the goal of statistical, I call it the principle of statistical fair comparison, that's what I would call it.

Q. Okay. And if we could put up slide 37 from this morning.

Q. If you look at the African American line . . . here on the left, the left side is your model that controlled only for grade point [176] average and LSAT scores, correct?

A. That's correct.

Q. And if we look at the line for African Americans, your odds ratio there is 257.03, right?

A. Well, I'm not going to worry about the decimal, you can't.

Q. 257, okay. On the right is your second model where you added in some additional factors in addition to GPA and test scores, right?

A. That's correct.

Q. And so you were controlling for more factors in the model reflected on the right?

A. That's correct.

Q. So, in your view, the applicants being compared in the model on the right, were even more similarly situated than the applicants in the model on the left?

A. I believe that's true, yes.

Q. And the African American odds ratio for the model goes up from 257 on the left to 513 on the right, correct?

A. Correct. I mean those are both giant numbers and I don't want to say they're very different as far as factors goes.

Q. So, when you control for more factors you're getting [177] a larger odds ratio, right?

A. In this particular case we control for these additional factors. I got a larger odds ratio in--my expectation was that it may be that they could go down. And in many analyses where we do control for additional factors, they go the other direction.

Q. In other contexts in your experience they go down?

A. In statistical contexts that I have worked on, which is a greater variety of contexts, yes, they can go in either direction. I didn't say they would go down, I said they could go either direction.

Q. In this case in each year the odds ratios go up, is that right?

A. I'm not sure that they're is uniformity, I think that's probably the case.

Q. At least in 1995?

A. Well, certainly in '95 we have that, and we can look at the reports if you want to for the other years.

Q. Suppose that you had numerical information on all of the factors that the Admissions Office considers, so that you can bill all of the factors in the model and control for them. In that situation, wouldn't you [178] expect the odds ratio to approach infinity?

A. In what sense? I guess if I had additional numerical factors, additional factors that we control for?

Q. Yes.

A. You know, I just don't know.

Q. In other words, you wouldn't expect that in your model as you add in other factors so that you got to the point where you were controlling for everything the Admissions Office considers other than race, that the resulting odds ratios would not be infinity?

A. I think the odds ratios may get large in this case. I don't know if I had--you're talking in a very hypothetical way, since we can't do this analysis in only a hypothetical way. But, in fact, boy if you ever have such models, you've described the process perfectly, that's right. So everything would be infinity in sense of odds ratio.

Q. And the reason for that, and I'm just trying to understand the way your approach works. The reason for that is if you control for all the other factors and are looking at people who are identical except for the fact that some are minorities and some are [179] not, then any difference in the admission in those two groups, the model attributes to race, isn't that right?

A. If that's the only factor left and that's how decisions were made, then that's what would happen.

Q. And the resulting odds ratio in that context would be infinity, correct?

A. It would be large, it could be infinity in our hypothetical if we could make all the odds ratio infinity for everything.

Q. Okay. And the reason it would be large or infinity, is that the only factor left to explain any of the difference would be race, right?

A. If that were the deciding factor, sure.

Q. And the odds ratio in that situation would be the same, no matter how much race had been taken into account by the person actually making the decision, isn't that right?

A. If we have all the other factors that went into the process, if we had that, which we don't here certainly. But if we did have that, then I think, in this hypothetical example you would wind up in that situation.

Q. Okay. And this odds ratio analysis then, can't tell us about how much race is taken into account by the [180] people making the decision, right?

A. It can't, is that what you're saying?

Q. I'm asking you if whether I'm correct that it cannot?

A. It measures with respect to just what we have here. It measures that when we take in account grade point average, LSAT grid cells, how much race is taken into account with respect to explained decisions beyond those. That's what it explains, no more than that. It's a description of the Admissions process.

Q. You think that this model is saying something about how heavily race is being weighed by the person who sits down and reads the admission file?

A. What I think is the aggregate effect of the decisions made in the Law School with respect to admissions. The aggregate effect is--well, for instance if we look at what we can't see there, the effect of residence, as far as making decisions. That the effect of race, for instance, is much greater and has a stronger effect than Michigan residence. And when we compare non-resident minority applicants to resident majority applicants, you can see that, in fact, decisions were made strongly in favor of the minority applicants. So, [181] with respect to that, which is that factor, I could say.

* * * *

Q. What would the odds ratio be if, instead of admitting about a third of the minority applicants, the University or the Law School admitted about half of the minority applicants?

A. In the sense that they would consistently, and I'm going to ask you, because I want to make sure I understand how you're thinking. That they would take individuals in the grid and then go further down in the grid for admission in the way, and then result in about half of the minority students being admitted?

Q. They may have to go to some extent further down the grid, or maybe they would take more minority students from cells, you know, comparatively higher up?

A. Sure. And so in general what would happen in that case, I think, and I'll wait to see how you follow up to decide whether I agree with that. In general the odds ratios would go up.

Q. Okay. So, they're already as large as you've ever seen, and if we admitted 20 percent more minority students they would get--

A. (Interposing) I think they would get bigger. They [186] would get bigger in the sense that if the same pattern of admission goes on with respect to using grade point average and LSAT which is clearly done, then they would get bigger.

* * * *

[187]

Q. Before I do that I should ask you, as you understand it the Admissions policy that you have reviewed adopted in 1992, has been in effect throughout this period, 1995 through 2000, is that right?

A. I don't know if anyone told me any different, I have not seen another Admissions policy. So I don't know anything to the contrary. . . . . . .

[188]
Q. Would you agree also that the number of applications from members of the various racial and ethnic groups was fairly stable over this period?

A. I think it is fairly stable, that's true.

Q. There's some fluctuation, but it's fairly stable?

A. These are data. If there's not fluctuation, then we wouldn't believe the data.

Q. So, over the same time period as you calculate them, the relative odds have varied quite substantially for members of the minority groups?

A. I think that's the way I would expect, they would be very substantial.

Q. Okay. If we could put up slide, I think it's 35, which is the 2000 Relative Odds of Acceptance. We have for African Americans 443 are the odds that you calculate, is that right?

[Here are the slide data for 2000 and 1997, along with 1995. Ed.]

A. That's right.

Q. Okay. And if we then go to slide 32 for 1997, instead of 443 you have 53.49?

A. That's correct.

Q. So, that means that the odds ratio for African Americans in 2000 was about eight time greater than it was in 1997, is that right?

A. That's correct. I have to say once odds ratios are [189] high they're high. And 50 is high, 400 is high. And I would have to say very clearly that I think that there's not a substantive difference when odds ratios get as high as those.

Q. So you don't think there's a substantive difference between 53 and what was it, 443?

A. I think they're both big, okay. And I think the substance which is what we are talking about, is we have to very careful and statisticians don't get bogged down in numbers and, in fact, allow us to understand the substance. In a substantive way, these are both big numbers.

Q. Interesting to hear that statisticians say don't get bogged down in numbers. But did you do anything to look at whether the differences in the odds ratios for these two years 443 versus 53, were statistically significant?

A. I did not make a formal comparison.

Q. There is a formal test that could be done to test the significance of that difference, is that right?

A. It's possible to do such a test.

Q. And you haven't done that?

A. I did not carry that test out.

Q. Okay. Would you be surprised to find out that the difference between these two numbers, 443 and 53 is [190] more than eleven times the standard error of the difference?

A. Yes, I would be surprised. I can't imagine given the size of these that that difference is there. I can do the calculation, if you would like.

Q. If it turned out that the difference between those two odds ratios is more than eleven times the standard error of the difference, would that concern you as a statistician?

A. Would it concern me?

Q. Yes.

A. In saying that there is a statistically significant difference between the Admissions policies in these two years?

Q. Yes.

A. And thus say if I were concluding that I would say that in 1997 they didn't give so much preference to African Americans, and in 2000 they decided they would give more preference?

Q. Exactly.

A. Would that concern me?

Q. Yes.

A. In the substantive part of this case, that doesn't concern me because these are all large preferences.

Q. Okay. You don't think that that kind of instability [191] in the odds ratios would call the validity in your model to question?

A. Actually the fact that we got large ratios no matter if they're as different as you said, and the consistency of them in the cross years actually makes me feel very comfortable with the substantive conclusions that we have drawn into these models.

Q. I believe you said near the end of your testimony this morning, that you thought you had quantified the role that race plays in Admissions, did I hear that right?

A. Did I say I had quantified it?

Q. Yes.

A. I'm not sure I said that as a direct quote, I may have said something to that effect.

Q. Do you believe that you have quantified the role that race plays in Admissions?

A. I think what I have shown is that race plays--the individual selected minority groups are given a large allowance with respect to Admissions decisions.

Q. How big is that allowance?

A. How big is that allowance?

Q. Yes. Can you put a number on it?

A. I can give you examples of odds ratios for similarly [192] situated individuals at various levels of GPA and LSAT.

Q. But my question is, is it 443, or is it 53 for African Americans?

A. And I would say that that specific number is in the sense not important statistically, because they both represent large allowance.

Q. So, we can't hang on any one of these numbers as representing what the allowance that you say you found actually is?

A. I mean I think it's difficult from year to year, it's estimated to be different from year to year and I think that's the nature of the process.

Copyright by the Regents of the University of Michigan