Thursday, October 26, 2023

Gaming The System

Rehash of previous posts of a few years ago.

When the town I lived in moved up in the rankings of test scores for highschoolers I was suspicious.  There wasn't any obvious reason for the change, no sudden increase in Asian doctors moving in or new high-tech firm that had hired a hundred new engineers. It also seemed to be confined to the high school, not all the grades. I made a point of getting into a conversation about it with the principal, and learned that prior to testing day the teachers who were going to be proctoring the exams met to discuss the more difficult students and whose room they would be in. "I have a pretty good rapport with him.  I can get him to stay on task."  Getting the kid who was going to was going to give up half an hour in, whether obviously or quietly, nets you more points than years of finding ways to hunch your A- students up to an A.

The Finns do something similar in the PISA tests, where how well the nation scores is a matter of honor, which gives even their most helpless students some incentive to at least try. In both cases, even though it games the system a bit in terms of measuring how good the education (or more likely, the natural intelligence) is, I would be reluctant to call it out, because it is a very good societal value: everyone helps the weakest move up. The school reputation or national honor motive behind it makes it not quite altruistic, but it ain't bad. 

Crime is reduced not by inspiring the young people who have only a 1% chance of committing a serious crime to be a little bit better, but by identifying those with a 30% chance and finding ways to cut that in half, whether in seriousness or frequency.

In terms of technical improvements, the opposite may be true.  Creating the circumstances for one person to create a steam engine is better than a hundred tiny improvements in water mills. The latter is nice, certainly, and much of human progress has followed that path. But printing presses, axled wheels, or biremed ships increased advantage much more, and quite quickly.

2 comments:

David Foster said...

Steam engine wasn't really created by one person, though. Newcomen's engine worked, but was a coal hog. James Watt improved it, but couldn't have done it without Wilkinson's boring machine. And there were thousands of improvements, major and minor, over time..higher pressures (which Watt opposed), compounding, lubrication systems, better boilers, etc etc.

Someone at X was saying the other day that she wasn't worried by slipping standardized test scores because what really matters for progress is just a handful of very brilliant people. I think this is wrong.

UDee said...

Regarding standardized testing, I suggested to my team teachers we hold test taking practice during elective time & schedule 10-12 students each session. We provided snacks & soda & were very low key in our approach. Students were chosen who were close to moving to the next level & about 4 sessions were held with a different group each time. As I was the only one on the team to keep track of test results, I was pleasantly surprised to see improvement in their scores. Was my evaluation scientific? Not really, but it was worth a small investment of our time & money. These were middle schoolers.