Physics GRE Considered Harmful
“As presently constituted, it’s quite possible that the GRE physics subject test does more harm than good, and we should either fix it, or seriously consider getting rid of it altogether,”
A quote from Jennifer Siders in this article at aps.org, that really we ought to take seriously. I doubt we will, though, because the Physics GRE is well entrenched at most graduate programs across the country, and making changes like that is always tough. Indeed, the article I linked to (as a result of seeing it in Pamela Gay’s Facebook status) was written 13 years ago, and yet the Physics GRE is still going strong.
I’ve been grouchy about standardized tests for some time. When it comes to things like the general GREs and the SATs, I believe that it does correlate with overall academic performance. Whether or not it’s testing the right stuff, there seems to be some correlation between what it tests and what we’d really want to test. But, it’s not perfect. That is, for (say– I’m making this number up) 80% of students, the SAT and general GRE might a good indicator of how successful they’ll be in college. As such, from a mercenary college admissions’ point of view, it’s worth keeping using them. Most of the time, they get the right students, and damn but it’s really easy to cut down on the number of applications you actually have to put work into thinking about by sorting on a simple number. Of course, from an individual fairness and a humanity point of view, it’s pretty sad to think that the other 20% (or whatever) who would have thrived at a certain college aren’t even considered because of a bad test….
The Physics GRE, however, has bothered me since I started as an assistant professor. Now, mind you, this is not personal sour grapes. My Physics GRE score back in 1990 was 89th percentile. At the time, I felt a little bad about that; I was one of those geeks who always did well on standardized tests, and thought that I should get over 90% on anything math/science related. Much later, I realized that 89th percentile is damn good for the Physics GRE. I did not personally suffer as a result of the Physics GRE, so I’m not posting this out of bitterness.
But, there is evidence that the Physics GRE does not correlate very well with how you do in Physics grad school. It seems completely unsurprising. In grad school, you do well by doing well at research. Yeah, you have to pass your classes, but even there it’s very different from what the Physics GRE tests. The Physics GRE tests your ability to think uberfast (which may be relevant in conference arguments, but is not terribly relevant for most research), your ability to recall things you’ve memorized, and your ability to quickly go through canned problems about basic physics. It’s not completely irrelevant, but it’s not testing what is most important about graduate school.
Of course, all the hand-wavy justifications for why it’s the wrong test only mean so much. As I said, there is evidence that the Physics GRE does not correlate very well with how you do in Physics grad school. What’s more, there’s evidence that women who do just as well as men in grad school on average score lower on the Physics GRE. In other words, either because of societal conditioning or because of intrinsic differences, the Physics GRE is more unfair for women, on average, than it is for men. Given that we’ve got a recruiting and retention problem for women in Physics, we should take this very seriously.
September 3rd, 2009 at 10:23 am
I concur. The Physics GRE felt like more of a trivia recall test (and a very hard one, at that) than anything else. Perhaps that’s sour grapes on my part, as I didn’t get into *any* grad schools. (Graduating during a recession didn’t help.) That test is the only standardized test where I got less than 98th percentile – I was in the mid-70s.
I’d like to think that I would have done well in grad school, and your analysis suggests I might have. 10% of my undergrad credits were in research, and I really enjoyed it. The impact on my GPA (entirely positive) suggests that I didn’t suck at it.
I don’t know if my GRE score had anything to do with it, but the whole experience certainly changed the entire direction of my life.
September 3rd, 2009 at 10:34 am
Which grad schools did you apply to? Mid-70’s is damn good on the Physics GRE. When I was on the graduate admissions committee at Vanderbilt, anybody who scored over 70 on the Physics GRE was viewed has having a perfect score, effectively…. We would only really get nervous about it if it was below 30th percentile or so, and even then it wasn’t necessarily a veto criterion.
Grad school admissions in Physics can be weird. Katie Chynoweth, who is a grad student at Vandy and who worked with me before I scattered out, is an excellent grad student, and doing very well; she’ll probably graduate this academic year. I don’t think she was admitted to any schools the first year she tried; she was admitted to Vanderbilt her second year out. That same year that Katie got in nowhere, I had a truly excellent undergrad who almost didn’t get in anywhere; in the end, after he had been rejected from lots of places, we started making some calls to try to figure out what was going on, and he was admitted to U. Colorado. Afterwards, I talked to some other folks I knew at top schools, and it just randomly happened that that year there was a huge surge in the number of grad school applicants. E.g., a guy at UCLA told me that people who would have been obvious admits the year before didn’t get an offer.
September 3rd, 2009 at 11:31 am
I’m a woman who scored poorly (in that 30th percentile range) on the physics GRE. I got into grad school anyway, though not my first choice (which was CU Boulder.) I’m conflicted: on the one hand, I don’t really think the physics GRE tests much of anything worthwhile. On the other, do I just think that because I didn’t do well on it? But I have done reasonably well in grad school and, so far, in industry…
Your article is making me ask myself again why I didn’t do well on the physics GRE, then. I can come up with a lot of reasons for me personally, the biggest one being that I went to a small liberal arts school and took a lot of things like Latin and art history while others at big research schools were doing all physics all the time. I also studied abroad one year out of my four, which did screw up the sequencing of my physics classes a bit — I took E&M there that I did *not* have the background for, and then never took it at home. I also tested out of the intro physics (and calculus) course with AP credit earned in high school. I took an AP calculus class to prepare for that test and I was fine, but the physics test I took and prepared for on my own initiative, and by the time the GRE rolled around, I’d forgotten all of the stuff about inelastic collisions and pully systems that I’d taught myself five years before just well enough to get a 4 on the AP exam. I took it for granted that they were “easy” problems — but not when you’re trying to think fast, they aren’t. I skipped discussion sections because I had a job… Etc.
None of these is really an excuse, but combined I wonder if they don’t in part account for some of the gender disparity on these tests. I believe that there are more women in liberal arts colleges — mine was 66% female in 1999 — and art history classes (I have no idea about study abroad.) And the AP exam? Is there a gender correlation there?
Then there’s “stereotype threat” which is well documented. And a gender difference which I think may be real, based on anecdotal experience, but doesn’t necessarily have to do with ability: I think men may be hormonally biased to be more competive, leading them to place more importance on high percentile test scores and thus to score higher… (If women are just slightly less competive by nature, that might account for a large part of our relative absense from boardrooms and, say, racecar driving).
September 3rd, 2009 at 12:21 pm
Not sure what the use of the GRE is at all. For all scientific graduate programs, how you do in the lab is going to determine your success or failure. Not your ability to provide correct answers on a standardized test. I’d say decent grades in college (especially in lab classes) are as good an indicator (if not better) to predict graduate school success than any standardized test.
Just my two cents.
September 3rd, 2009 at 12:26 pm
I had a god-awful migrane on the day of my physics GRE, and it’s only available on one day. So I got something like 39th percentile. Despite nearly perfect grades and an impressive resume, I was rejected from every graduate school physics department. The only school I got into was University of Illinois in Urbana-Champaign’s engineering department, which doesn’t look at the physics GRE. It also happens to be one of the best engineering colleges in the world, so I guessed it worked out.
So yeah, I’m a little bitter about the physics GRE.
September 3rd, 2009 at 2:02 pm
Rob, when you were at Vandy, how did you evaluate international students from schools you might not have heard of before? A physics/astronomy PhD program gets applications from students in schools all over the world, some well-known, others obscure. Surely some sort of standardized testing is necessary, even if much more weight is put on grades, etc.
Having said that, I think the Physics GRE definitely needs to be changed if it is to be more useful. It places far too much emphasis on speed. I, for one, could never get through all the questions. But that doesn’t mean I knew less physics than someone who did, just that I don’t think/work problems so fast. My four years of grad school experience so far tell me that speed/quickness on your feet might be important during oral/qualifying exams and for impressing people during a talk, but pretty irrelevant to doing good research and writing a good paper, which are the main requirements of a PhD. I would say that curiosity, creative and innovative thinking, and perseverance are the most important skills for a good researcher.
So in short, I think there is a place for the Physics GRE, but a small one. And it needs to test some skills other than speed. Of course, unlike Rob, for me all this could be just sour grapes since I didn’t do so hot.
September 3rd, 2009 at 4:50 pm
I think a different number was used the year I was applying: GPA. There was a huge flood of applicants, and no one left grad school unless forced to. (Gotta love a recession.) The acceptance criteria seemed to be GPA > 3.9 and from an Ivy league school (or technical equivalent.) My undergrad GPA was hardly stellar. By the time the next year had rolled around I was gainfully employed and didn’t even think to apply. Probably should have – but didn’t.
September 3rd, 2009 at 4:59 pm
Naved — the truth is, different people on the committee weighed different things different ways– which is probably a good thing. Myself, I have to admit that I would do a “first cut” based on grades and GREs. That was very rough; that is, it would say “almost certainly in” or “almost certainly disaster”. When it came to actually thinking about students, I personally weighted the reference letters the most. This, of course, is hard, because it’s not something that’s easy to quantify and sort. But, by the same token, it says the most about the student.
As for international students– well, we divided the world into three groups. USA, international-not-China, and China. We usually had 2x to 3x as many applicants from China as we did from everywhere else put together. And, many or most of them had high scores. The positive spin on this is that they are very good at learning how to do standardized tests. The negative, but almost certainly true, spin on this is that cheating on these things is institutionalized over there. (You’d see students with a high %ile on the Verbal GRE, and a letter of intent that indicated that they were only barely fluent in English….) What’s more, the reference letters tend to be very uniform, with all students receiving good and similar-sounding reports. So, for China specifically, we had no way of telling the students apart *unless* somebody in our department had a connection with someone who knew an individual student’s capabilities. As a result, only those with an “in” would be admitted from China. I don’t know if they realize how badly they’re hurting their students over there by “helping” them get such uniformly high scores….
For the rest of the world, we looked at everything. GPA and GREs. Truthfully, scores on the Physics GRE were almost always better for international students than for most USA students. We would generally give them a pass on the Verbal GRE if English wasn’t their native language. (In extreme cases, where we’d worry about TAs the students couldn’t understand, we’d think about what extra help they might need to be a TA.) But, myself, even there, the reference letters were what was most important.