Opening my latest e-mail edition of NCTE, I noted the second of what surely are many articles and blurbs citing a new study about correlations between data gathered at RateMyProfessors.com and data gathered in the IDEA Student Evaluation System, which is used by around 275 colleges (http://www.insidehighered.com/news/2008/04/25/rmp).

As I read this report, I have the same feeling I’ve had each time I read the findings: something does not line up.  I know that the argument people are tempted to make based on the findings - that RateMyProfessors.com is a pretty accurate measurement of a course and a professor/instructor because it is getting feedback that is similar to course evaluations - is somehow completely bogus, but I cannot articulate why using statistics. 

Thinking about my unease with the conclusion that RMP is a good measuring tool of the quality of our teaching, I think about the course evaluations I filled out as an undergraduate, and of my peers’ evaluations in those courses.  I ponder the RMP reviews of instructors and professors whose work I know well, remembering my initial response as to whether the website’s contributors were accurate reviewers. 

Maybe students have gotten more honest and hard-nosed in their reviews of professors and courses since I was an undergraduate, I think.  Then, without even a second’s pause: I think not! 

I remember the course evaluation being used as revenge against a professor for any number of perceived infractions:

A) Course perceived as too damned boring for the student’s liking (generally due to subject matter, not teaching method),

B)  Professor perceived as unreasonably demanding - in other words, professor encouraged actual learning through course activities, and honestly measured whether it looked like learning was taking place,

C)  Professor perceived as politically divergent from student’s own cherished belief system: student believed this was biasing their grade, and in turn this gave students grounds on which to completely ignore any feedback from professor during the term.

D)  Professor is not white

E)  Professor is not male, and is not apologetic about holding high course standards, is not perceived as attractive enough to be objectified (and summarily dismissed because they have been inserted into a sexist paradigm).  Professor is perceived as either taking revenge on the world for her perceived unattractiveness through insisting on high academic standards, or is perceived as too attractive to have achieved her position through expertise in her field.  The former is punished by sexist reviews for being too much "like a man" (e.g. possessing too much brainpower or expertise in a field for their limited notion of what a woman "should" be like), but the latter is punished for being too much "like a woman" (e.g. despite anything she says, she is perceived by sexist reviewers as not having anything of intellectual or disciplinary value to say).

I am thinking more specifically about two professors I had as an undergraduate, and how my peers and subsequent students rated them both in course evaluations and on RMP. 

The first, Dr. R, was a white male professor whose course consisted of lectures and discussions over assigned readings.  Students were required to pass several tests on both lecture and reading material, and were required to write a course term paper.

The second, Dr. W, was a female professor from China whose course consisted of lectures and discussions over assigned readings.  Students had to pass take-home essay exams over the readings, and were required to write a term paper and present a panel session on a topic of their choice to the class and professor.

Students reported that they evaluated Dr. R. as being unreasonably demanding - the scuttlebutt about his courses was that they were hyper-intellectual contests of doom for students.  The students claimed that you could easily fail his courses, that his tests were voluminous, unnecessarily difficult, and designed to trick you.  Dr. R. was loathed by many students because he unapologetically demanded that students know the material that the course covered.  If they did not demonstrate knowledge of the material, they failed.  He also demanded that students be on time to class, and locked the classroom door to prevent the interruptions of students coming in 20 minutes late and walking past the lectern to take their seats.

In my second class with Dr. W., I was sociable with my fellow undergraduates and was privy to their thoughts and reviews of Dr. W.  They thought she was far too demanding, but more than that, they claimed that they could not understand her when she lectured due to her (very slight) accent.  I knew this to be bullshit - her speech was clear and easy to understand, and the students complaining were the worst slackers in the class, never doing the readings, never speaking in class, rolling their eyes behind her back as she wrote on the board.

When I found out about RMP and first visited the site, I looked both professors up, and lo and behold, there were many reviews online that echoed what these students reported in the course evaluations. 

There were (and still are) about three kinds of reviews on RMP:

A) Slacker students slam professor because they believe they should get ‘A’s in courses they have not successfully completed,

B) Decent students praise professor because they recognize the quality of her/his teaching of the subject.

C) Fraternally-Motivated students rate the course’s difficulty in terms of how much work, participation, and attendance it took to earn a good grade.

So what were my evals of these professors, and what reviews did I leave on RMP?  I gave Dr. R. highest marks - his lectures were brilliant syntheses of what had been said and what was known and speculated on his topics of specialization.  I loved the degree to which his exams measured exactly what had been covered in class, reinforcing what I had learned.  He was more than fair in his grading of my earnest but problematized term paper, noting where in the argument I was weak, and where my argument gained strength.

I defended Dr. W.’s intelligibility in my review of her courses - she was easy to understand for those who wanted to listen.  Her grading was demanding, but I learned more in this kind of course than in the courses that gave me a good grade for little work.  The courses I remember most, in hindsight, are the hardass professors’ courses.  The information I remember most is from these well-constructed courses.  Knowing more after having attended college and after having earned a degree in something is a good feeling, at least for me. 

This brings me back to the false correlation between the similarities of the course evaluation results and the RMP results: the same pool of students is reviewing professors through both venues, but does either venue really measure the quality of learning that goes on in a given course?  I have always found that they measure other things: how much the students like that subject, whether the student is comfortable and open to learning from women or international professors, and the student’s overall attitude toward what they want to get out of their college classes, be it specialized knowledge or easy ‘A’s toward a degree that they hope will get them a good salary.