Thursday, March 10, 2022


The complaint was about standardised testing, and how relying on it just encourages kids to parrot information back. Because usual suspects. It makes the schools teach to the test...the wealthy kids can get test doesn't test how you think, really... Someone has a startup developing a new test that is more like a problem-solving video game. They interviewed lots of people. The last interview was with the president of the American Psychological Association who told his own story about having test anxiety as a boy, then getting a C in his first college psychology course, and comparing notes with Philip Zimbardo, who also got a C in his first course, and then announcing that the tests were destructive and limiting and it was important for kids to know that you can be anything you want if you really desire it and are willing to work for it.  

That's when I suddenly thought "You have simply exchanged parroting information for parroting the correct attitude."  I thought of Freddie deBoer's work, which I have discussed a few times, that this "you can do anything" myth is so central to the mindset of education, and educational psychology, that they cannot endure to have it undermined.  

I realised that everyone else they had been interviewing was doing the same thing, deploring the parroting of information while applauding the parroting of the correct attitudes. What is your job as the president of the APA (remember there are at least two APA's), after all? You have to be the face of the organisation, making sure that you look like the real experts.  You have to make sure it keeps up with trends and doesn't fall out of step with the Zeitgeist. It is a job of social and cultural sensitivity more than intellectual attainment. The softer the science, the more the PhD (PsyD, EdD, etc) is a measure of perseverance and reading the times and individuals in authority over you. 

Test anxiety might be a real thing, but it has proven quite difficult to identify and measure. I know at least some people i have known personally at work who have claimed it, and I have come to think "No, you just ain't that smart." But anxiety is real, and screwing up a test with it sounds plausible, so I don't toss it completely.

The new test design begins by claiming that the old paper and pencil tests only measure one small part of full intelligence, with an implication that it's not a very important part either. They claim that employers want something that measures the other kinds of intelligence as well, like empathy, and creativity, and collaboration. These qualities kept coming up. I thought "Really? Employers are telling you they want to know more about empathy?" I didn't later have the inspiration that HR departments might indeed be telling them that, because they are forever dealing with difficult people who are making life difficult for everyone else, and HR believes that if they only had some empathy it would fix that.  That could be. Good luck measuring it on a test. Same for creativity. Lots of people have tried to dream up ways to measure that over the years. I don't know of any tests that are successful.  I took one once, of listing as many uses for a brick as you could think of in one minute.  I can also speak fast and blew off the scale.  Highest score they'd seen. And your point? That is truly a "So what?" moment.

Collaboration is increasingly important, even in hard sciences where the ideal used to be the lone genius. I entirely agree. Of all the work that I did over the years, the times I was able to take dysfunctional teams with talented people and get them working together with joy and camaraderie were my proudest moments. I got very good at it. (I had more fun at the really brilliant team that already had joy and camaraderie and I only had to do some basic bits to keep things moving, but that's a different story.) It is important, and may indeed be more important than the (supposedly) narrow type of intelligence the SAT measures. I just don't see how you are going to measure that on a test. 

And why would you want to, when you can get real-world observations of whether someone is playing nice or not? Isn't that better?  I think it is the same as above about empathy, that people see that they are spending far too much time getting talented people to just f-in' get along and not be dicks, and want to have some number or short eval they can look at on an application to stay away from them.  Some threshold number for hiring. 

Well fine.  But now you are just trying to measure a bunch of stuff at once and blaming the one test you have in the arsenal that measures a clear straightforward thing and does it pretty well. The SAT achievement tests in specialties actually are achievement as much as aptitude, showing how much Spanish or Physics you know.  The SAT remains an aptitude test, whatever people call it. If you want to say that your college has decided that collaboration and citizenship are more important, I think that's fine.  You may be right.  Go for it. But then own that, that you are dropping one type of measurable excellence because you don't care about it.

If you really think that how you play the game is more important, then don't keep score. And don't complain that others are keeping score.


james said...

I'm not sure how you "score" empathy. That leaves the evaluation arbitrary, and we're back to "who you know." Which is a huge factor already...

Christopher B said...

HR may desire to easily identify difficult people but they'd much rather do it with a test than make the call about who is being an ass in a real world situation.

Donna B. said...

When my ACT scores were sent back to my high school, I was called to the counselor's office and her first question was "How did you cheat on this test?" Frankly, I was shocked at my scores too, especially that math was my highest score -- a subject I'd been barely passing since 9th grade. So, I understood why she was questioning it, but I was highly offended that she thought I'd cheated. I didn't think I was smart enough to figure out how to cheat!

She thought she was vindicated when my SAT scores came back -- barely average for that high school. I didn't tell her that I didn't quite remember taking the test. One piece of advice I have for parents and students is DO NOT spend the weekend before such a test in a college dorm with a "trusted" family friend introducing you to the wonders of alcohol and partying all night.

So, my scores are evidence for both sides of the argument: the high one pointed to an aptitude that didn't result in achievement though I did manage to achieve much more than the lower score predicted.

I never experienced anxiety over a written test. I liked taking tests; that's possibly related to why I like trivia games so much.

Assistant Village Idiot said...

Excellent examples of exactly what yous say. It is not that likely you will score much over your head. There aren't going to be that many "Slumdog Millionaire" questions that you just happen to know. But it is possible to screw up anything, including your SAT's, especially as it is an all-morning test and one of the things being measured is your ability to maintain mostly-sustained focus over that time period. Whether that is truly part of intelligence is an interesting debate. One could argue that there are brilliant people who can only keep focus for brief periods, but the other side is also quite plausible. The harder questions take at least a few seconds of toggling back and forth between partial solutions to get to a final answer. If you can't do that, you're not actually brilliant.

And trivia games are not an infallible measure of IQ, but there is an overlap. It's a collection of things that you paid at least brief attention to at one point in your life and put efficiently into storage for retrieval.

David Foster said...

Test anxiety....if someone wants to be a pilot, of any kind, then they will need to take both a written test and a practical test (flying with the examiner.) Would anyone seriously propose that a candidate be excused from these tests on the grounds of 'test anxiety'?

Cranberry said...

One reason to move away from a paper and pencil test has been the very slipshod approach to test security on the part of the testing agencies.

The New York Times article lists multiple cheating scandals discovered over the years, including the Operation Varsity Blues cases, which included bribing test centers and sending in ringers.

There is expensive test prep. I don't know how effective it is, but any test can be prepared for, especially if the organization reuses test sections, and doesn't do a good job securing its exam. As a comparison, I believe the MCAT (since 2007) and the LSAT (since 2019) are administered online.

As to the SAT achievement tests, while 950,000 students took the SAT in 2021, far fewer students sat for the achievement tests. Only 4,606 took the Korean achievement test; as a score of 800 was the 64th percentile, I'm not sure it added anything to the application. As for physics, students probably think there's more value in taking one of the AP exams in physics, which specialize in different fields, rather than the SAT achievement test. (Looking online, it seems there is now AP physics 1, 2, and C.)

Grim said...

"You have simply exchanged parroting information for parroting the correct attitude."

Excellent insight.

Let me tell you, it's fun getting a Ph.D. without being willing to parrot the correct attitudes. It's fun trying to make a living, too. This is the skill most valued in corporate America, as in academia, at this time.