January 6, 2010

The Lost Art of Comparison Studies

Filed under: academia — Joshua @ 6:10 am

Russ Roberts has found an interesting article in the New York Times about weight gain in females. Apparently, women who have children or have long-term partners gain more weight than single women.

After adjusting for other variables, the 10-year weight gain for an average 140-pound woman was 20 pounds if she had a baby and a partner, 15 if she had a partner but no baby, and only 11 pounds if she was childless with no partner.

Roberts thinks this article is interesting because it can’t seem to stumble on an explanation for this. It tries out all sorts of hugely implausible theories (for example: women with partners tend to eat out more, and restaurants serve portions that are too big for them) but misses what should really be obvious to everyone: unhitched women have more of an incentive to stay thin. And they have more of an incentive to stay thin because thinner women are (generally speaking) more attractive. Once in a committed relationship, this incentive goes away, or at least diminishes in strength. EVERYONE knows this. We even know its converse: if we meet a thin woman in a “committed” relationship, we immediately put shock quotes around “committed” - like I just did. Roberts thinks it says a lot about academia’s assumptions about obesity that paid researchers can’t discover what everyone knows is the cause.

OK, sure, that’s interesting, and sure, he has a point about it being revealing about researchers’ assumptions about obesity. But you know what I think is even more interesting and more revealing? That the obvious followup study about the relative weights of single vs. committed men was apparently never performed. In fact, why do an all-female version of this study at all? Even if your primary professional interest is in female weight gain, surely the numbers on male weight gain are of interest to you - if only as a way of isolating how much the variable “female” contributes to the phenomenon? It’s not like it’s difficult to account for the control group in this case; there are, as far as I know, only two physical sexes.

What this is is yet another illustration of yet another human statistical perception blind spot. This one works like this: you take a population you’re (politically) interested in, do a study on it, establish some kind of trend, and then, by merely publishing your results, you give the impression that whatever trend you’ve documented is exclusive to that population. But in fact all this does is establish that the trend exists in the study population. It doesn’t say anything about whether it is exclusive to, or even more pronounced in, this population. To establish that, you of course have to do comparison studies. Like, in this case, quantifying the amount of weight gain in men with the same dependent variables over the same period of time.

I would put lots of good money on there being a similar trend among men. I think - and forgive the boldness of this assertion, gentle reader, as I am not an expert in gender-based weight gain - that if you did such a study you would come to the astounding conclusion that married men with children and men in committed relationships also tend to weigh more than unattached men. AND, I think the reason for it is also exactly the same: men who are single care more about their physical appearance and thus have more of an incentive to slim down than men who have already achieved the “be fruitful and multiply” goal (where “fruitful” isn’t to be confused with … well, you know). Unfortunately, documenting this would be inconvenient for people who make their living studying “female-specific” phenomena, and so they don’t bother.

May 15, 2009

We Don’t Need No Thought Control

Filed under: academia — Joshua @ 7:23 pm

A recent entry on Mr. Tweedy’s blog reminds me that I don’t like grades.

No, really - and this isn’t a even a new opinion that I formed in my many years as a teacher. I’ve thought this way since at least high school - and no, it isn’t because I get bad grades. (I get good grades, but not perfect grades - and that’s because I am one of those people who so totally lacks self-discipline that I only turn in homework in classes I’m really interested in. True story: I got an A in intro Spanish in junior high without turning in a single assignment - because I was so far ahead of the class that it seemed pointless, and the teacher agreed. The condition was that I not take Spanish I the next year because she didn’t want to be in that position again.) I think it’s sorta the same reason I’m against the death penalty. You think of what a thing is for, and you try to use it to that purpose only. If it’s being widely abused, you shut it down.

So take the death penalty, for instance. I don’t think there’s really a coherent moral case against it. Rights in society are a kind of contract (you respect mine and I’ll respect yours) - and if you violate that contract to some appropriately egregious degree, you lose your rights. If we reserve the right to put people in cages for the rest of their existence (as clear a violation of any “rights” they would otherwise have as any), I don’t really see the blazing white line that would allow one to say that the death penalty is going “too far.” So there’s no coherent moral case against it for me. But there are two persuasive practical ones. First - there’s the obvious problem that once you’ve killed someone you can’t unkill them, and given the staggering number of wrongful convictions in this country, it seems like a good idea to hedge our bets here. Second - and relevant to the grading issue - is the fact that I hear WAY too many arguments for the death penalty on the grounds that it brings “closure to the family of the victim,” or some similar bromide. And that really irritates me, because the purpose of the law is NOT to minister to people’s feelings! The law is not an instrument of vengeance, and neither is it a personal grievance counsellor. It should not be in the business of “punishment” at all. Penalities serve two purposes: (1) deterrence and more importantly (2) giving asocial people some time out - perhaps permanently if they’re beyond redemption. Crimes can’t be “made right,” so why try?

I think grades have been similarly perverted. Really, what a grade is supposed to do is evaluate a person’s knowledge of a subject. If you can demonstrate that you’ve grasped the subject, you get a passing grade. If not, you don’t. It SHOULD be that simple - but on the ground what I find is that grades aren’t used to that purpose at all. What happens instead seems to be that instructors make the assumption that everyone passes ahead of time, and they only ever fail people in those really off-the-charts cases where the douche just didn’t turn in any homework (cough, cough), or signs his name on the exam and then just doodles - or something. But all else equal, everyone passes. The bulk of the task seems to be calibration - that is, making sure that the distribution of grades reflects some expected ranking of students based on the instructor’s impression of how well they all did relative to one another. The point that makes it all so horrible is that “relative to one another” bit. Ideally, it should just be “relative to class expectations,” and the only interesting question should be “did or did not student X gain working knowledge of the subject taught?” But unfortunately, in reality grades seem to function more like a currency system in a closed economy: you hand out points based on who the model citizens are, and solely for the purpose of giving them a way to compare and rank themselves relative to one another (and so metaphors like “grade inflation” are much more apt than most people realize). Instructors use grades for two purposes, neither of which are the original intention: (a) as a proxy discipline system and (b) as a proxy recommendation system - becuase those students who get perfect grades get to go to the good schools.

My proposal is we do away with the miserable ranking system as switch to a pass-fail system - where “pass” is roughly our current A, and everything else is “fail.” In other words, students just accumulate a list of subjects they know about on their transcripts. The key to making the system work, of course, would be removing the taboo on “fail,” and I think we could do that by simply not recording “fails” on transcripts. So grades become a kind of certification instead, and since there is no record of how many times a student applied for this certification, there is no problem taking and retaking a subject until you get it right. An employer should only be concerned with whether the applicant crossed the finish line, after all, not how long it took him to get there.

The reason I’m suspicious about using grades as a proxy recommendation system, by the way, is that it has the same problem that all attempts to quantify “talent” have: it’s too crude a measure for real talent, and so all it accomplishes is allowing people who are really good at jumping through hoops pass themselves off as more qualified than they are. Don’t get me wrong - jumping through hoops is a talent of its own, appropriate to certain kinds of jobs for sure. But it’s only one of many, and the current grading system gives too much weight to it at the expense of people who are actually good at what they do. No - identifying talent is one of those things that simply can’t be automated. You know it when you see it, and instructors who recognize it can instead volunteer to write the student in question extra special recommendation letters or something. Trying to be “fair” about how we rate talent is just futile - because the distribution of real talent is itself not fair. We have an education system that rewards ambition rather than real effectiveness, and it shows.

May 28, 2008

When Language Log isn’t about Language

Filed under: academia, linguistics, politics — Joshua @ 8:09 am

There’s a deliciously disingenuous post on Language Log today about Obama’s past sins that is illustrative both of the kinds of acrobatics his supporters go through to defend him on his past associations as well as the gratuitous leftism of much of academia.

The argument goes like this:

Everyone (well, almost everyone) knows that context is important when we try to understand meaning. … And when you look at language in the context of time (when something was said in the past), it’s relatively easy to find fault with silly people like us who change our minds about some wacky position we once held. Years, months, weeks, or even minutes ago most of us said, sometimes even believed, something that we wouldn’t want to support any longer. But there are bullies out there who ignore such time contexts so if you said it once, you’re stuck with it forever.

You see where this is going, right? People are allowed to change their minds on things. Well, right, fair enough. But it doesn’t take us long to get from there to here:

I’m coming clean at this late date because political events have made these wrenched-from-context-gotchas the game du jour. For example, we all know that Senator Barack Obama’s membership in the Chicago church where Reverend Jeremiah Wright once preached marks him negatively in the minds of a huge number of U.S. voters. No matter that he hasn’t regularly attended that church for decades. He used to attend, so he’s guilty, even if he didn’t hear the allegedly offending sermons, and even though he has since renounced Reverend Wright’s offensive positions. Forget the context of time. It happened. So off with his head.

What Shuy (the author) is “coming clean” about, by the way, is his three-year membership in a sexist gentleman’s club way back in the 1960s. I think we can all agree that he’s allowed to have changed his mind with the changing times in the last 40 years, especially if it’s true, as he says, that the primary reason for resigning his membership in the club was its exclusion of women.

What, one wonders, does this have to do with Obama? That’s the illustration of the second point: a leftist academic will find a way to smuggle shallow and off-topic political commentary into just about any discussion. It’s annoying, to put it mildly, and made all the more so in this case since Shuy cowardly closed comments on the post in question. Language Log generally allows them - but not when they’re pontificating about Obama.

It takes only a cursory familiarity with the facts to know that Obama doesn’t qualify for the “changed his mind” exemption in the way Shuy does. First and foremost, that’s because to qualfiy you have to actually have changed your mind, and Obama’s little conversion rings more than just a bit insincere. Association with Wright isn’t something that Obama did in the 1960s and then decided against on his own before he was in the public spotlight. Quite the contrary - he titled the book that doubles as his campaign manifesto after a line in a Wright sermon as recently as 2 years ago. He remains a member of Wright’s church to this day and has no intentions of disassociating himself from it. Furthermore, he has made inconsistent statements on the controversy that make it clear he’s not being entirely truthful about his motivations, initially claiming in news interviews that he had never heard Wright making such controversial statements but later admitting in campaign speeches he had. Perhaps richest of all is Obama’s on-again-off-again invitation to Wright to bless his campaign launch and act as his campaign’s spiritual advisor. Obama called Wright the night before the official campaign launch event to disinvite him, and that apparently on the basis of a Rolling Stone article about him that devotes a lot of column space to Wright. And what did Obama say to Wright to disinvite him? The following:

“You can get kind of rough in the sermons, so what we’ve decided is that it’s best for you not to be out there in public.”

If there’s any reading of that that doesn’t amount to “my association with you could be poltically embarrassing,” I’d really like to hear it. Certainly there’s no way to look at that and conclude that Obama was blissfully unaware all this time that his pastor had been saying all these controversial things, as he initially claimed in television interviews. And neither is there any way to look at that and conclude that Obama was starting to develop real personal misgivings about Wright’s positions. Nope, this was all politics all the time.

As for Obama’s eventual “renunciation” of Wright that Shuy is at pains to make sure we’re all aware of, does anyone really need to remind Shuy that it came a month after a popular speech in which Obama defended his association with Wright, and only came because Wright not only deliberately made more such comments in press interviews, but also implied that Obama had been lying about his opinions on Wright all along?

So yeah, people can talk about changing their minds, but they only get a cookie for it if they’re not lying for political expedience. If Roger Shuy was a member of a sexist organization in the 1960s which he dropped out of to protest its sexist policies, that’s fine - laudable, even. I think we can all agree that he’s allowed to “change his mind” about his membership under these circumstances (although, one hastens to add, the way he spins it in his Language Log column, he didn’t actually change his mind since he supposedly tried to use his membership to change organizational policies from withing, etc. etc., but never mind). Just as I think we can all agree that we wouldn’t buy it if he’d never actually left the organization, claimed on one occasion that he was unaware it was sexist and then on another that he knew but it didn’t matter because it had other redeeming qualities, and then only disowned its leader a month after he defended him, and that only because the leader impled in public that Shuy was insincere in his opposition to the group’s sexism. Obviously, in such a case, we wouldn’t believe for a minute that Shuy had actually changed his mind about anything.

Just as no one who’s been paying attention honestly believes that Obama has changed his mind about Wright.

Shuy’s column is disingenuous, his mention of Barack Obama gratuitous, and really, the whole thing, as a political and personal apologia, is inappropriate for a blog like Language Log that is supposed to be about Linguistics.