Tuesday, May 5, 2009

Teaching Evaluations: Ignore Them

I spent an hour or so this morning talking to a young colleague who had just made the mistake of reading her teaching evaluations for the semester. She was upset, antagonized, demoralized, and stricken with anxiety. The student criticisms ranged from silly and irrelevant (e.g., comments about the instructor's shoes) to simply mean. No doubt she'll carry all of this into her classes next semester, thereby enabling the all too familiar self-fulfilling downward spiral from mere unpopularity to truly ineffective (because fixated on the evaluations rather than the subject-matter) teaching. Why do we put faculty through this exercise in intimidation and humiliation?

So when she asked me what she should do, I told her that the answer was simple: stop reading your evaluations. Ignore them. Seriously: Has anyone has learned a damn thing of value from student evaluations?

139 comments:

Clayton said...

SMU instituted a new policy this semester. To encourage students to leave evaluations, evaluation forms won't be distributed in class but instead will be filled out online. (That way students who don't attend class can still give give input.) Students can't see their grades until they complete an evaluation. This includes students failed for plagiarism. I can't wait to read what they have to say.

Anonymous said...

My thoughts on this are kinda rough. I think the Profs that think they ought to read them don't really need to (aka, they already internalize the perspective of UG students). Those Profs that do not think they ought to read them, probably ought to.

I know many Profs that really need to read some blunt criticism about their methods. But for ones that already are concerned enough to already read them seem to still have some regard for the quality of their teaching.

This is of course predicated on the notion that students fill out evals in good faith. Which is, alas, a big assumption.

If anything, Profs shouldn't care what their GS evals say.

Krinos said...

What I learned from my most recent evaluations: 1. I'm bald (huh, who knew?), 2. I say 'fuck' a whole lot (I knew that already), 3. "nobody cares about his cute kids" (that *is* news to me), and 4. my philosophy class is "waaaaayyy to (sic) theoretical" (well, I'll be darned!).

So next semester: I'll wear a hairpiece to keep my baldness from being a distraction, use the word "fudge," not let anyone know anything about my personal life, and be sure that we talk about nothing but how philosophy applies to "real life".

Oh, and Clayton, hope the plagiarists cut and paste reviews from my evaluations for yours.

Jon Cogburn said...

Psychologists have statistically analyzed Student Evaluations and determined, among other things: (1) all else being equal, the best predictor on how well students rate the choice of textbooks question is whether or not you bring donuts to class, (2) all else being equal, female professors do much better if they dress and act in more traditional feminine ways, (3) for male professors the ways in which you modulate your voice and move your hands is highly predictive of a number of scores.

These are basically consumer satisfaction surveys. Unfortunately a non-trivial percentage of students think of their grade and compliments as the product they are purchasing. So if you don't make them feel smart it's as if their hamburger is overdone. And even more unfortunately a decent portion of students are the kind of people who are really superciliously rude to waiters.

The really pathetic thing is that administrators will use these numbers in totally unjustified (statistically) ways. (1) Everybody should score above their departmental average. (2) If professor x scores five hundredths of a percentage above professor y, then x must be better than y.

If you do reliably horribly then there is a problem; probably you do need to make your class more entertaining. Fair enough, we shouldn't have attitudes about that. If it's a low level class, then play bits of funny DVDs in each class to illustrate the points (Monty Python, Kids in the Hall, etc.)

If you always do excellently then there may be a problem too. But everything in between is just noise. They just give administrators and senior colleagues something to thump you with, should they so desire.

I predict, given the research mentioned above (the Chronicle had an article) a someone is going to win an influential lawsuit in the next few decades. The thing with gender roles and the statistical absurdities are both really problematic.

Anonymous said...

Change "colleague" to "president," "students" to "citizens," and "teaching evaluations" to "public opinion polls," and this post would sound a lot like Cheney's advice to Bush circa 2007-2008.

Also, your advice contradicts empirical research. See for instance Ken Bain's conclusion in WHAT THE BEST COLLEGE TEACHERS DO (Harvard, 2004): "All the teachers we studied have some systematic program . . . to assess their own efforts and make appropriate changes. Furthermore, because they are checking their own efforts when they evaluate their students, they avoid judging them on arbitrary standards." (19) Of course this doesn't mean you need to rely on university or departmental evaluations, but you do need some way of gauging what improvements you need to make. Personally I like to do an informal anonynous "what can we do to improve things?" about 8 weeks through the course. Students see that I take their advice seriously, and even though it is all perfectly anonymous (I encourage them to write all in caps or left-handed if they are worried about handwriting detection), I have yet to receive a single hostile comment.

Anonymous said...

I learned that if I brought cookies on the day of the evaluations, I got better evaluations.

Clayton- that sounds like a more poorly thought out version of what many law schools have. There the evaluations also must be done on-line, but they are done before exams, there is an option for "no answer/no opinion" on every class, and you must at least choose that answer to get the exam number you need to take exams. Whether the information is any good I can't say, but the mechanism is better.

Mike Almeida said...

The value of the information you get is obviously just a statistical question. If you're using an even mildly sophisticated questionaire and you've got something that approaches a decent sample, the silly answers get discounted quickly. We use one that is in wide use, and the information returned compares rhe evaluated class to classes of similar size and content nationally. The results don't strike me as wildly unfair or inaccurate. And it beats the hell out of having colleagues evaluating classes!

Jon Cogburn said...

Mike,

The issue wasn't with sample size or silliness of the question.

Rather, student answers have been shown to track things that have nothing to do with the specific questions.

Another way to reliably raise the numbers is to tell students that your rehiring is partly based on the evaluations. When students are told this, the numbers are better.

I'll dig up article with this data and post it here in a little bit.

Anon 4:49-

That's a great point about something that really works. I've done the same thing (have all students turn in anonymous questions and suggestions at times in the semester), and it is a really good idea for lots of reasons. First, you do get some good ideas to improve the class. Second, if done right the students feel you respect their opinions more.

I still think it makes sense for a junior person to try to make the class more entertaining as a way to raise the numbers. It's one thing you can do that does not require dumbing down the material and that brings the scores up. Given the ridiculous uses the scores are put, I think it's good advice to non-tenured profs.

Jon

w said...

I can related to the cookies. I was in a grad seminar, way back when, that didn't go very well. Really, it was brutal. Not the prof's fault really, just a mismatch between her interests and those of most of my peers. Anyway, our prof showed up to the last meeting of the semester with the course evaluation forms and a suitcase of beer (you know, a 24-pack). She put this stuff in the middle of the table and looked at the eight of us. "I won't say it was fun," she said, then she turned and walked out. I kinda miss her.

imipolex-g_unit said...

Three things:

1. Notice how no one has yet risen to Spiros's challenge to state which damn thing of value they've ever learned from student evaluations.

2. The philosopher Clark Glymour has written a terrific essay, " "Why the University Should Abolish Faculty Course Evaluations." link: http://www.hss.cmu.edu/philosophy/glymour/glymour-universityFCE2003.pdf

3. There are plenty of ways to tell if people with no expertise in philosophy or its teaching have been appropriately instructed in a philosophy course, and it sure the fuck ain't by asking them. Fuck student evaluations.

English Jerk said...

I used to think that student evaluations could be useful to the professor for thinking about how he or she might improve the course, but that they should not be used by the department or administration to evaluate the quality of our teaching. My reasoning was as follows: I am not a plumber. I am doctor of philosophy. So when my plumber wants to become a Master Plumber, they don’t contact me to find out if my plumber did a good job. I know fuckall about plumbing. The crapper works, but maybe the plumber didn’t fix it to last and a month from now I’ll be knee-deep in shit. I don’t know, so I’m not the person to ask. If you want to be a Master Plumber, you’re evaluated by a Master Plumber, who is a doctor of pipes and poop. My students—wonderful as they are—are doctors of fucking nothing. They’re in no position to evaluate my teaching because they don’t know any more about teaching than I know about plumbing. But they know things like whether or not they got what they expected, and knowing about things like that, I thought, could help me reformulate my course (at the very least, I could make clear what they should expect).

I was young and foolish then, and now I see that evaluations have no value of any kind whatsoever. My job is to help them learn. Nearly every course I teach changes every semester, sometimes dramatically, so that they learn more or learn better. I know what changes to make because I can see how they perform: if they suck at x, then next time I help them get better at x. Asking them does no good: if they knew they sucked at x, then they wouldn’t suck at it. And my experience has been that sometimes I get evaluations that say I’m the most inspiring professor they’ve ever had and sometimes I get evaluations that say I’m an insane confusing windbag who expects way too much of mere undergraduates, and these responses have never correlated in any way whatsoever to their performance and have never provided me with any useful information of any kind. So their encomia and their denunciations only waste my time and the institution’s time and their own time.

That said, it’s worth remembering that this is really the only moronic bureaucratic mindsplutter that is inflicted on us and not on secondary school teachers. Despite all the other humiliating rubbish they are subjected to, in this they do as the plumbers do. And so should we: the more robust our system for evaluating one another is, the less weight administrators will be able to give to these customer satisfaction cards. But in the meantime, follow Cogburn’s sensible advice: for tenure, make jokes, give candy.

Glaucon said...

My favorite experience with student "evaluations" was a pre-tenure review in which a colleague whose alleged pedagogical ineffectivemess is rivalled only by his pompous jacksssery proceeded to flip through a stack of those worthless things, reading random comments and asking me to respond. "This student thinks the class shouldn't have met at 8am -- care to respond?". What a fucking idiot.

I'm not sure if I've learned anything from these things, though they have occasionally confirmed what I already knew or suspected (e.g., that such-and-such a text didn't
really work). The best way to deal with student "evaluations" is to swap with a colleague and share only the very few with comic value -- e.g., "this wasn't an English coarse".

Mike Almeida said...

Jon,

Rather, student answers have been shown to track things that have nothing to do with the specific questions.I'm pretty sure I took the point to be the reliability of the information evaluations afford faculty. That's not the point? No doubt there are all sorts of data that correlate in unappealing ways. So what? My point is fairly simple. If you've got a reasonably sophisticated instrument, these correlations are still possible, but entirely lose interest. Even the most accurate instruments allow for fully misleading correlations.

Setting this aside, the important question is whether it is statistically possible to generate a reliable questionaire for student evaluations, given the typical motives and aims of students/faculty. The answer is (obviously, I think) yes.

Michael Cholbi said...

I'm afraid I have to be a dissenter here. Teaching evaluations have been thoroughly studied. An excellent summary of the relevant research can be found by Googling "Doyle evaluating teacher effectiveness summary." No, these evaluations are not perfect (none are), they can be badly used and interpreted (as can any evaluative instrument), and yes, students are not always mature and fixate on trivial things. But a well-crafted student evaluation is as good a gauge as any in assessing instructor performance. As the research summary I mentioned says it:
"Research indicates that students are the most qualified sources to report on the extent to which the learning experience was productive, informative, satisfying, or worthwhile. ... Research on student evaluation of teaching generally concludes that student ratings tend to be reliable, valid, relatively unbiased and useful."

Again, they're not perfect, but unless teaching effectiveness simply cannot be measured (a claim I reject), then student evaluations deserve to be taken seriously and ought not be ignored.

Dan said...

i've read one that says, verbatim: I didn't really learn anything from this class, because all the answers to philosophy are in the bible. I would recommend this class to some of my friends because they don't believe in evolution and the professor does

i learned something from that.

729 said...

Spiros asks: Has anyone has learned a damn thing of value from student evaluations?

Okay. I'll bite. I have learned some things of value from my student evaluations, although I need to qualify this. From time to time, I've commented utilizing advice I've received from friends who are (successful) professional actors and my own experiences in the past as a performer. So, from this perspective I've taken information from my student evaluations to better gauge my classroom performance and refine my classroom persona. I do not look at the evaluations as critical reviews (from professional critics), but something more like the sort of talk actors engage in after performances with respect to how certain lines of dialogue in a play are running. When a script makes particular demands on actors and the audience, they and the director take stock of what they are doing with the lines--what's worked and what hasn't with audiences.

I guess, for me, while I have a clear set of learning objectives and things I prepare long in advance, and things I try experimentally, I am genuinely interested in understanding for better or worse how this all comes across and the impression students tend to have--what my performance qua performance is like, so that I can control it as best I can. An example of how this sort of reading of evaluations has been valuable was noting things in comments where many positives like "high energy and enthusiasm" are matched with negatives "too much, too fast." This was an instance in which splitting the difference in terms of my performance was effective. Over time, I've crafted my classroom persona, which does not necessitate pandering to "the audience" or baking cookies, but means knowing and gaining control of how I generally appear to students, how I generally communicate material and becoming very consistent at being this persona. I have a certain distance from myself and my classroom persona, and I like it this way.

Besides written comments, certain statistical differences between my colleagues and me on how students rated our overall clarity made for enlightening discussions between us about what we were doing--how we were laying out arguments, defining terms, and, ultimately, whether we were on the same page about very basic skills. When a colleague is consistently rated a bit more clear than I am, I really want to know what she's doing. A whole bunch of questions on the evaluation will turn on intangibles and stuff none of us can do much about, but since the student evaluations will remain a part of our lives, we try to get whatever might be useful from them.

Of course, as English Jerk mentioned, my department is pushing increasingly towards more extensive peer-observation. This helps enormously, especially with respect to administrative evaluations--that is, give administrators more fine-grained qualitative assessments of our teaching so as to de-emphasize their use of student evaluations. This requires actually having peers willing to do this and take the time to write up their reports, but it's worthwhile.

All this being said, I also ignore student evaluations on a certain level, too. One of my horrifying realizations a long time ago was that I (at least) tend to over-overemphasize whatever stands out as negative. Out of the however many nice or neutral evaluations I receive, the sheer mean, crazy and sometimes downright derogatory comments are all I remember. The nasty wheel gets my attention. So, I don't look at my evaluations immediately after the semester. I get around to looking at them when it suits me, pour a shot of whiskey, and read them to get my own impressions.

Jon Cogburn said...

Cholbi and Almeida,

O.K. Put your money where your mouth is. What research "generally shows" them to be good. In particular, does this research in any way contradict the very good survey of the research in the article http://www.vccaedu.org/inquiry/inquiry-fall97/i12-adam.html ?

There are fifteen correlations they came up with and most of them have *nothing* to do with the actual questions the students are answering. Many reflect sexist attitudes of the student population, and many have no connection to good teaching.

To show that student evaluations are "effective" you would need to correlate the responses with the truth. So, for example, you would have to correlate students scoring textbook highly with the textbook being a good textbook. Nobody has done anything remotely like this.

Instead the kind of inane correlations shown in the above research have been established. They are sexist and (as the above article shows) in all likelihood actionable.

Please also read Michael Huemer's nice piece ( http://home.sprynet.com/~owl1/sef.htm ) which also actually cites the extant research. Also see the note by the American Psychological association on Gillmore and Ceci's research here http://www.apa.org/monitor/feb98/studs.html .

The good news is that with the above articles you can effectively improve your evaluations without doing much. The bad news is that fully doing this would require female (but not male) professors to socialize with students and instantiate traditional gender roles. Oh well, as long as they're "effective."

Jean K. said...

I think teaching evaluations reveal (of course!) perceptions. They don't reveal the truth about whether you are informative, insightful, fair etc. However, the perceptions actually do matter. For example, being perceived as boring just ain't good. If you are boring, students will learn less, retain less, and the class will leave less of a lasting impression. If you make people feel angry or disrespected...again, feelings actually do matter, even if they aren't justified. Teachers do have to pay some attention to what students are feeling. New instructors really should read evaluations, I think, because in the beginning you can really can have inaccurate perceptions of what students are thinking and feeling. You can think you're being scintillating and amusing, while students see you otherwise (and the like). After many years, though, I think patterns develop, and the benefit of reading evaluations diminishes. At that point, it can be just demoralizing to subject yourself to the ravings of outliers who think you're the worst teacher on earth.

PA said...

Over the years I've heard it both said that studies show that student evaluations reliably track teaching quality and that studies show that they do not (and I have always been, and remain, too lazy to look at the studies myself). My question is what is the reliable measure of teaching quality against which both sorts of studies compare student evaluations (and why don't we this instead of student evaluations).

Robert said...

I wait at least a term, and usually a year, to read evaluations from a class. That way, if there is anything of value in them, I'll be better able to see it (being further emotionally from the events themselves.)

I do learn something from written comments, at least sometimes. I may notice a comment repeated by several students. That's usually a sign that something could change for the better.

Mike Almeida said...

I have no idea what this touted research is supposed to show. No doubt, there are lots of bad questionaires used for student research. I don't need a research report to inform me that such evaluations are around and that departments use them. If that's your point, I concede it.

The ridiculous conclusion to draw from this is that there could not be statistically good questionaires developed for this (i.e. college student) population. It's not credible to suggest that this particular population actually defies good statistical assessment.

To turn to actual assessments, we've used IDEA surveys for the last 6 years or so, and they don't seem especially unfair nor inaccurate. But even supposing they were interestingly flawed, certainly it's possible to develop a survey for this population that has these virtues. And that's the point I think I've been trying to make. It would be stunning if reliable statistical conclusions became unavailable right at the campus entrance.

Anonymous said...

I’m curious to know what people think of the following question. Let’s suppose that the scale for standard questions on the teaching evaluation is 1 through 5 (1-very bad, 5- very good). Consider two assistant professors, A and B, who have taught a comparable number of students (say a few hundred students) over the past few years in similar classes in terms of level/topic. Suppose that one of the standard questions on the evaluation form is “rate the professor’s overall quality” (or something like that) and let us suppose further that A has an overall average of 4.5 on this questions and that B has an overall average of 1.5.

Do you think that in the absence of any other information about their teaching abilities we should conclude that (1) A is probably a better teacher than B; (2) B is probably a better teacher than A; (3) we have no reason to think that one professor is a better teacher than the other?

Anonymous said...

Almeida,

I take it that Cogburn's point is that student evaluations are reliable indicators of student opinions about teaching quality, but student opinions about teaching quality are not good indicators of teaching quality itself. If students are not good at evaluating teaching, then why would it be surprising that no questionnaire could be developed to turn student opinions into a good measure of teaching ability? No doubt, one can construct reliable statistical models of how many books on baseball history will be checked out of the New York public library next year, but why would one expect such a model to have any connection to, say, upcoming elections in the UK?

The point of the research Cogburn references is just that student evaluations often track irrelevant attributes of teachers, as opposed to tracking quality of teaching.

Robert said...

If there's a large enough sample (I don't know exactly what that is), I'd certainly take that as a reason to look into it. That's a sign of some problem, but usually such problems can be solved with help. A good opportunity to help a fellow colleague improve, that's what I would think.

Neil said...

The claims that there is a bunch of research showing that teaching evaluation are good trackers of quality and that there is a bunch of research showing the opposite are both true. The first bunch comes from work in educational psychology, which is the neanderthal cousin of academic psychology. I encourage you to go and look at some: it really is jawdroppingly bad.

You can learn stuff from these surveys, but your knowledge will be Gettiered. You can design surveys to produce useful knowledge, but first you will need to find a way to replace your student body with a different one whose perceptions are more relevant.

Dr. Killjoy said...

You guys are nuts. Of course student evals matter. Here is just a smidgeon of what I have learned from them over the years.

1) I need to purchase a different style of trouser.
2) I should "judo chop all the haters".
3) I should allow at least one particular student to sit on my face.
4) Booyah!
5) I am both full of myself and smarmy (or at least full enough of myself so as to come off as smarmy).

QED.

Glaucon said...

I try to avoid calling these surveys "evaluations." Everything I've ever read on them insists that they provide data but should not be thought of as themselves evaluative. But a recent discussion has reminded me that I shouldn't call them "student surveys" either, since there are no students anymore. They are learners now. Not students, not pupils, and not -- alas -- stupils.

PA said...

@ DR. Killjoy

I've learned

1) that I walk like a vampire
2) that I'm less interesting than a doorknob
3) that I lack common sense and a personality
and
4) that I have good water retention

I'm still not sure what (4) means.

KateNorlock said...

PA, item #4 means you can apparently hold a long class without ever taking a bathroom break, or suggesting one to the class.

I've taken some students' suggestions seriously to my benefit, so I don't discount all the feedback as worthless, and I don't ignore them. I have a lot of students, and wide patterns are telling. But yes, one must ignore the rare negatives. They'll just stick to you for months if you don't shake them off.

J.P. said...

Jon Cogburn, you present some thoughtful criticisms of teaching evaluations. I respect you for giving references to the relevant research to back up your claims. I am also grateful for your reference to research on possible sexism, which I agree is important and which I plan to look into.

I value teaching evaluations more than you do, but concede I will not be able to rebut all your criticisms in a blog comment. Rather than try to change your mind on all points, I will try to demonstrate why I think the empirical research on this is not as cut and dried as you suggest, by zeroing in on just one source you invoke, namely the summary essay by Huemer (whose book on intuitionism I loved, my bashing of him here notwithstanding).

Huemer cites the famous Dr. Fox experiments, in which an actor was hired to deliver a high-energy, low content lecture, which then got positive audience evaluations. Huemer concludes that this and many follow up studies show that "audience ratings of a lecture are more strongly influenced by superficial stylistic matters than by content."

However, there is evidence that expressive delivery helps learning. As a different summary of the Dr. Fox research notes, "Students who watched the highly expressive lectures performed better on a multiple-choice recall test than students who watched the less expressive lectures. " (See
http://ctl.stanford.edu/Newsletter/memorable_lectures.pdf)

If expressiveness makes a lecture more memorable, then Huemer is jumping to conclusions by dismissing good delivery as "superficial." Indeed, it seems obvious to me that, all else being equal, good delivery is a plus. A high-content, high energy lecture is surely better than one that is all content, no delivery.

Huemer also cites a study by Ceci and Williams in which an instructor (Ceci) participated in a teaching workshop that taught him how to give more expressive lectures, and subsequently saw his teaching evaluations improve across a host of areas.

As Huemer writes, "Students in the spring semester believed they had learned far more (this rating increased from 2.93 to 4.05), even though, according to Ceci, they had not in fact learned any more, as measured by their test scores."

This is a potentially devastating criticism of teaching evaluations. If the instructors ratings went up when he improved his delivery, but it made no difference to student learning, then it would seem the evaluations are indeed tracking something unrelated to education.

However, it is never mentioned exactly how Ceci's students were evaluated. All we get is a vague reference to "test scores." This key shortcoming was noted by two other researchers who wrote in to the journal that published Ceci and Williams study, criticizing their experiment on many points:

"An additional shortcoming is the failure to describe the psychometric properties of the learning measure. For example, the learning measure may have included a large number of items based on reading assignments rather than on the content of the lectures. Failure to find a significant effect of instructor expressiveness on student learning may merely indicate the poor validity of the achievement test. It is also possible that students compensated for poor and uninspiring instruction by devoting additional effort to out-of-class study.
"

The letter presents all sorts of reasons to take the original study with a grain of salt, but the basic punchline is again that giving an engaging lecture does not necessarily hamper learning, and may well help it. If so, Ceci's 4.05 evaluation was tracking something related to learning after all.

The bottom line re Huemer is that his presentation of the education literature is too selective and misleading to be convincing. To my mind, this raises the possibility that if we went through other summaries that made the whole question sound shut and dried from an empirical point of view, we'd find similar ambiguities and problems, and again be reminded of the need to carefully re-read all the original studies in questions.

I concede however that this is just a possibility, and that demonstrating it is the case would require going on at even greater length than I already have, which I doubt anyone wants.

If any of my fellow teaching nuts are reading this I strongly recommend the Stanford newsletter on teaching cited above, which contains some solid teaching tips.

J.P. said...

Not sure my link came through cleanly. It is:

http://ctl.stanford.edu/Newsletter/memorable_lectures.pdf

Mike Almeida said...

(Some) Anonymous (or other) writes,

. . . but student opinions about teaching quality are not good indicators of teaching quality itself.It is ridiculous to claim that every student is a bad evaluator of teaching on the basis of extant questionaires. The fact that students are influenced by factors irrelevant to teaching performance is not news. Every population is influence by factors irrelevant to what we want to learn. This is what statistically good questionaires do: they diminsh the significance of these irrelevant factors.

Here's an analogy. When taking tests for psychological disorder/health, lots of people try to "fake healthy". They will answer questions in ways they think will make them come out on the healthier end of the psychological scale. So, these people are clearly trying to skew the test. But every well-designed test for disorder indicates when people are faking healthy. If the test is designed right, the influence of irrelevant factors is diminished. In these cases we have a population that is trying it's best to skew our results, and the tests are sufficiently sensitive to indicate this. When it comes to student evaluations, we can certainly develop questionaires that diminish the influence of factors that are irrelevant.

English Jerk said...

Mike:

You seem to be missing the point of the statement you quote from Anon@6:48. The point is about an actual state of affairs, not about the limitations of our methods for inquiring into that state of affairs. For example, there is no way to construct a questionnaire that will elicit the details of quantum mechanics from people who know nothing about it; the opinions of such people will simply have no relationship with the details of quantum mechanics. If you want to know about quantum mechanics, you should not ask people who don't know anything about it. If they know nothing about it, it doesn't matter how you go about asking them.

So the corresponding claim is that students' opinions have no connection with the actual quality of teaching. One could certainly dispute that claim on factual grounds, but the claim has nothing whatsoever to do with our methods of inquiry. It's not that students do have the ability to assess our teaching but are distracted by extraneous details; it's that those extraneous details are all they have access to. The most perfect questionnaire imaginable cannot elicit information from a person who does not possess that information.

Neil said...

Mike, of course no one is claiming that every student is a bad evaluator of teaching quality. The claim is about the reliability of scores as a guide to teaching quality. Good evaluators have their scores washed out by bad. The only method that could be used to avoid this happening is to weight the scores of the good evaluators. Since there are no extant methods of identifying these people, there is no way to implement this methodology. Not saying it can't be done, just saying it hasn't been done. The analogy isn't with trying to elicit information about quantum mechanics from a kindergarten class. A better analogy would be trying to elicit information about quantum mechanics by asking a group of 100 people, some small percentage of whom might have relevant information, whether they agree with various statements about quantum mechanics.

Rufus Doofus said...

To promote excellence in leadership, we should distribute Dean evaluations (and eval's of other administrators) to all members of the university community, making sure that administrators' continued employment depends on favorable ratings. This will facilitate an even excellenter learning experience for all stakeholders!

Anonymous said...

As a person still looking for permanent employment, it gives me great hope that there are so many of you with terrible opinions of your students and their abilities to track good teaching. I've learned plenty from my course evaluations. I design and distribute my own evaluations for most courses in order to supplement the generic forms that my SLAC uses. Of course, no evaluative tool can track all that is worth evaluating. One has to read the results with a critical eye. And, I'm not defending what any of your deans and institutions do with the evals. But, that wasn't the original question.

The question was what we learn from the evals. Here are some useful things I've learned, mainly from my supplemental evaluations: which specific topics in a class the students find interesting; how much of the reading (the students are willing to report) they actually did; whether the students found a secondary text useful; how they ranked the value of different assignments. Of course, I get some of this information personally. But, students are often hesitant to tell instructors anything critical about their courses in person. The cloak of anonymity can give them the freedom to be honest without fear of consequences. Is this not obvious?

I had heard of the Glymour piece, but hadn't seen it before imipolex... posted it; I'm grateful. But, the piece is disappointing, not terrific. Some of the morals he draws from anecdotal evidence are unsupported. I won't go into detail about each one. In what I take to be the most compelling case, Glymour reports that he changed the format of his logic course. Students learned more, but his evaluations were poor. He concludes: "Student evaluations are more influenced by formats meeting their expectations than by how much they and their classmates learned." Read the story. The students decided that he had changed the format of the course to avoid lecturing. Did he not discuss the format of the course with the students? Was he not available to them? I taught a course that I think was similar to the one he taught; the students would never have accused me of trying to avoid lectures since it was like running 60 concurrent independent studies. Glymour's story is suspect, and self-serving.

More importantly, is our goal, as teachers, to impart the most content? Do we really think that success as a teacher is measured by the quantity of arguments the students can recite? Glymour's logic course sounds ambitious, and the achievements seem impressive. But, is a logic course better if the students are unhappily pushed to, say, Kalmar-style completeness proofs than if they are enthusiastically embracing translations and derivations? Which students will be more likely to pursue further work? Which students are more likely to apply what they learned to other courses?

I concede that students are often tracking worrisome factors. But, studies that denigrate course evaluations by comparing high-content boring delivery with low-content exciting delivery are just setting up false dilemmas. Student course evaluations may be limited in scope and effectiveness, and may be designed well or poorly, and may be misused. But, I really like them.

Then again, my evals tend to be fucking great. If they weren't, I'd probably be railing against the injustice of it all, like so many of you.

Glaucon said...

I propose that we institute Poster Evaluations. Such instruments would allow to to assess, in an objective way, the quality of the various posts and posters -- e.g., Anon @9:22.

Here's my first question:
"Poster appears to be a smug, self-righteous, condescending jackass."
a. Strongly Agree
b. Agree
c. Unsure
d. Disagree
e. Strongly Disagree

Second question:
"I find poster's reasoning -- e.g., student evaluations must be valid because mine are so good -- compelling."
a. Strongly Agree
b. Agree
c. Unsure
d. Disagree
e. Strongly Disagree

Third question:
"The cloak of anonymity can give [other posters] the freedom to be honest without fear of consequences. Is this not obvious?"
a. Strongly Agree
b. Agree
c. Unsure
d. Disagree
e. Strongly Disagree

Fourth question:
"Cram it up your cramhole, LaFleur."
a. Strongly Agree
b. Agree
c. Unsure
d. Disagree
e. Strongly Disagree

anon 9:22 said...

I'll happily cop to the appearance of smarm, and take the 'Fuck you'. But, come on, Glaucon. Do you not believe that griping about evaluations correlates with being disappointed with the content of one's evaluations?

Glaucon said...

Anon 9:22 --

You're a good sport!

While grousing about student "evaluations" may be correlated with getting low numbers, I think that's often not the case. My scores tend to be pretty good, but that doesn't keep me from thinking that these surveys aren't good measures of anything but student satisfaction -- not learning or teaching quality -- and that they shouldn't play the role or have the weight they do at the institutions I've been at, which aren't atypical.

I just saw a terrific chamber music concert; I really liked it. Am I qualified to evaluate the playing? No, at least not beyond basics like "they seemed to be in tune.". I'm not a competent judge of excellent musicianship. I doubt that many if any students are competent judges of excellent teaching, either.

Neil said...

Anonymous at 9.22 am; psychological methodology - double-blinding, statistical controlling, and so on - exist because you can't know the things you think you know, just by inspection. The properly controlled work shows that if the conclusions you think you have learnt from the evaluations come from the averaged scores (this doesn't apply to comments), then if they are true it is by accident. I also advise you t look up "ad hominem" in a intro logic textbook. As a matter of fact, I do think my teaching evaluations are wrong: I get well above average scores and I suspect myself of being no better than average.

mvr said...

FWIW, I think it worthwhile to separate the value of evaluations as measures of comparative faculty quality (where I think they're not good) and as a feedback mechanism about something that you can use to teach better (where I have found them helpful).

The points about sexism is well confirmed, so any comparison of males and females using the numbers on evaluations seem highly suspect to me. I'm willing to believe that cookies can skew things in similar ways and think Jon Cogburn is probably right about correlations with irrelevant factors of other sorts. So I think that at best their use in evaluating faculty merit has to be corrected by all sorts of other data. (If in written evaluations half the class says that the instructor was hostile to students, I don't think that could be written off entirely without more info, so I don't want to say they are never relevant to this purpose.)

On the other hand, I read mine and pay attention to them. For me the raw rate the instructor number is not what I look at. I look at patterns in the written comments. If five students in a class of thirty make similar comments I try to figure out why they might be doing so. Then I adjust how I teach if I have a hypothesis and see what happens in future classes. Since my gender does not change from semester to semester and since I don't bring cookies on evaluation day at least some of the irrelevant factors are controlled for. So changes in my evaluation patterns over time can give me info, even if those correlations skew comparisons between faculty.

As someone above remarked, evaluations can give you good evidence of student perceptions of what you are doing. That is not a good all in measure of the quality of your teaching, but it does tell one about something that effects how effective one's teaching can be. If several students in the class say they were afraid to speak up for fear of being criticized harshly, I need to adjust something.

In response to the request was for examples of cases where evaluations were helpful, I recall that when I first started teaching my large lecture classes I was regularly perceived as unapproachable and distant. This caused me to readjust some of the tactics I used to try and project a no-nonsense atmosphere in those classes and my evaluations changed in this respect. That's not good enough evidence for science, but I do think that the information helped me to adjust things in a positive direction.

I'm writing this mostly because I think people tend to get too defensive in the face of negative evaluations. I do think there is good reason to oppose using them as the main basis for evaluation of teaching quality for bureaucratic purposes. But that isn't a reason not to use them at all to improve one's own teaching. The fact that we know enough to know that comments about our clothing are irrelevant should also allow us to pick out the more relevant feedback and use it in helpful ways.

Rufus Doofus said...

I designed my own questionnaire for students, which I distribute about 1/3 of the way into the course. I started doing this because my university stopped asking students for comments in the standard eval's, opting instead to have students just check dem boxes in the on-line questionnaire (since administrators don't read -- 'ceptin numbers -- they get all Pythagorean fer numbers).

Also, a nice lady in the Learning & Teaching Office told me that giving your own mid-term eval's has been shown to improve your end-of-term evaluation scores.

J.P.: What if it were shown that students pay more attention and learn more when the instructor is attractive? Would universities then be justified in stating that applicants for teaching positions must be attractive? So that academics would be even attractiver than news anchors?

A big point in Huemer's item was the alleged correlation between good eval's and high grades. Grade inflation and course eval's as a gauge of customer satisfaction ... makes sense. The customer is always right, after all.

Anonymous said...

I have taught philosophy for years, and have sat in on other instructors' courses many times out of interest. Most of the teaching I have seen is absolute crap, and I'm often surprised how much of it students can take.

Those of us who teach philosophy generally receive what most people would consider a very decent income, and yet we break our backs trying to avoid any kind of criticism of our most important work. In just about any other job (particularly those that pay so well), we would be out on our asses if we did a crappy job repeatedly, and there is considerable oversight. And what do we have in philosophy? Almost no oversight, superiors and colleagues who would rather tiptoe around us for years than suggest that we're not doing a very good job and need to improve or get out of the business, and smug attitudes galore (e.g. many people who have posted in this thread) about those whose interests we are paid to _serve_ and how inferior they are to us.

Here's something interesting: how many of the whiners about student evaluations have anything better to suggest? How many of them propose even a single thoughtful (or even unthoughtful) alternative way to ensure that those who can't teach well either shape up or ship out? Of course they don't, because they're not really interested in finding better ways of getting the relevant information. They're just crappy teachers, or friends of crappy teachers, looking to safeguard their privilege at the public expense, or are touting their membership in a club whose hallmarks are a cynical scorn for their professional duties and an unhealthy desire to circle the wagons in the face of any threat to the status quo.

Without a viable alternative suggestion, these petty carpings are pretty obvious smoke screens for the professional incompetence one would expect from the relative privilege we enjoy under the banner of a phony 'academic freedom'. Please -- either accept that your bad reviews are a reflection of your bad teaching, or propose an alternative, falsifiable way of testing your worth, or shut the f*** up.

Thanks.

Anonymous said...

Okay, Anon 1:52 AM, why not pre-test and post-test over material or skill-knowledge that the relevant faculty suppose ought to be learned by students taking classes in the relevant department? (Something like this is suggested in Glymour's essay. He gave his students selected problems from the LSAT at the beginning and at the end of a year of logic, if memory serves.)

Granted this doesn't answer the interesting point raised earlier that "amount learned" might not be the right thing to care about -- perhaps "interest in material going forward" is important as well. I'm not sure how to test for this, but maybe looking at graduation rates and/or retention rates generally with respect to enrollment in specified professors' classes might be informative enough.

Another suggestion already made a few times on this thread -- more careful faculty/mentor observation -- might also be deployed. One might even give this responsibility (maybe only in part?) to education faculty who are supposed to know something about pedagogy.

We could also add hoops similar to those most states ask high school teachers to jump through, e.g. require lesson plans be submitted (to whom I'm not sure) a week in advance, although most high school teachers I've talked to think these hoops are not too helpful.

In any event, the "whiners" aren't the ones trying to preserve the status quo, since student evaluations *are* the status quo.

Anonymous said...

Those sound like good suggestions, Anonymous 3:58am. I'd be interested in making a try of them. Is this what you are proposing we institute instead of, or in addition to, student reviews?

As for whether the whiners are pro- or anti- the status quo: I see the matter somewhat differently than how you do. The status quo as I've seen it is that, no matter how poor a job instructors do teaching, they cannot be fired on that basis. That is the case, at least, at the great majority -- and in fact, I think all -- the schools with which I have direct experience. I certainly expect that that is the situation at the great majority of schools. How many people can say that about their job? And what does it say about the concern we have for our students that we allow that state of affairs to persist?

Moreover, the status quo at many schools -- including one where I was a graduate student until recently -- is that instructors with poor reviews, poor enrollments, and what any reasonable, objective measure would show up as poor teaching cannot even be reprimanded and sent to remedial teaching training on that basis. It was once mentioned at a department meeting that there are schools at which people who are consistently bad at their work are forced to improve by spending six hours in remedial classes. Some people hearing this almost fainted away.

What I would like to see, and what I meant by _not_ the status quo, is a situation where instructors who do a bad job consistently are first warned, then trained to do a better job (if the warning doesn't work), and then fired (if the retraining doesn't work). Naturally, it would be good if the instructor's merits (or lack thereof) were measured in many different ways, including a) the sorts of tests you mention; b) student evaluations; c) peer reviews; and d) enrollment levels. If this seems extreme to anyone, I would suggest that that itself is a measure of the extent of the problem. If salespeople, lawyers, contractors, entertainers, etc. etc. etc. consistently do not perform quality work, they get fired. Why should we be any different? Why should we force students to tolerate it?

Tenure should not be a protection from this retraining or firing, for any reason I can see. There might, it is true, be cases where a department finds it desirable to retain the services of a faculty member who clearly is a failure at teaching, owing to his or her consistently good research. In that event, I think that faculty member should either be given a full-time research position or (if financial factors demand it) a part-time research position with much less compensation.

We all know how many promising recent graduates there are out there who would be great at teaching if only an opportunity opened up for them. Retaining the services of the failures at teaching -- or refusing to monitor the teaching acumen of faculty members for reasons of delicacy -- while great would-be teachers languish away and yet another generation of students is forced to endure 'teaching' at the hands of incompetents seems misguided and unfair. And yet, it seems to me, this is the status quo.

Neil said...

That's right anon; if I can't suggest a better method of measuring teaching quality, I can't say that existing methods are no good. Presumably I can't point out that this argument is indicative of idiocy either, since I can't suggest a way of curing your idiocy. So I won't,

Anonymous said...

Neil, of course you can legitimately raise -- as a matter of abstract discussion or a preliminary factor -- some concerns about whether the current methods of teaching evaluation are no good without proposing an alternative. However, let's be honest: the context in which these comments generally appear, and the context in which these comments seem to appear here also, involves the implication that actual policy decisions be made on that basis alone (i.e. the complainers want the evaluations to be got rid of, or not taken seriously, without the institution of anything better). Do you disagree? Have I mistaken a purely idle discussion here, or a preliminary discussion undertaken by people who are suggesting problems tentatively while seeking out better ways by which they and their peers be held accountable, for what I take it to be (i.e. an attempt to brush aside teaching evaluations, full stop)?

As for 'curing my idiocy', let's see: while most working people are held to standards for the main aspects of their work on pain of losing their jobs, philosophy teachers are not and are paid handsomely regardless. But philosophy teachers still exhibit such scorn for those whose education they are amply compensated for improving that they bristle even at the minimal efforts made to hold them slightly accountable (even if this more or less never carries with it the risk of the philosophy teacher losing his/her job). And I, though a member of this ridiculously privileged elite, express my exasperations at other members of the privileged elite who are helping themselves to the remarkable compensations of our work while refusing to be held accountable for poor work standards.

Yeah, it sure sounds as though I'm an idiot.

Neil said...

Dear Idiot,

First off, let me offer half an apology for being an idiot. I offer any sort of apology because you replied in reasonable tones, but only half because you made an insinuation which made the name-calling justified. The insinuation was that my attack on evaluations was motivated by my free-loading ways (or my being friend with free-loaders). You don't know me, you don't know my friends and you don't even know what I do (my primary activity, as it happens, is not teaching).Nor do you have any reason to draw such an implication. So only half an apology.

Now to the substantive issue: you are right that there is a practical issue here. But you are very confused about the nature of that issue. You are right that the implication of the claim that teaching evaluations are worse than useless is that they ought not to be used. You seem to think that the claim that teaching evaluations should be withdrawn can only be made by someone who can suggest a better alternative method of evaluating teaching quality. But that's very badly confused. Suppose I discover tha treatment X (say the application of leeches for the treatment of cancer) is worse than useless. I point this out to you. You, I suppose (after you have told me that I am a friend of cancer, I guess) will tell me that I have no right to suggest the withdrawal of the treatment of the treatment unless I have a better alternative. But this is clearly wrong: the fact that I can't treat cancer does not invalidate either my claim that leeches are worse than nothing as a treatment, nor does it invalidate the implication that even in the absence of an alternative treatment, we should not use leeches. Got it?

Anonymous said...

Fuck you, Neil. Calling me an idiot again in your smarmy tone demonstrates nothing other than your lack of maturity. If you want to descend to that kind of discussion, please leave your name and home address so that we can continue it in person.

If I may return to the actual point, briefly: I have already explained why my position does NOT entail that nobody can offer a critique of something without offering a viable alternative. AGAIN, what I am saying is that, given the context, this discussion looks suspiciously like what I said it looks like, and not at all like the idle, intellectual consideration you suggest it is. Please go back and read my comments again, and you will see this.

You would have me believe, it seems, that your (and others`) remarkable concern over this particular issue among all other issues can and should be considered without any particular attention to its context or to what other issues you are neglecting to mention. The fact that this is exactly the issue that an elitist who wants to avoid censure for bad work, or preserve a snotty attitude toward students, would raise is not, you suggest, something I can rightly draw attention to. It`s as though I were to muse endlessly and scornfully about the fact that people of Ethnicity X have this, that, and the other negative trait and then, when others call me a racist because of my endless whining about those of Ethnicity X, I were to respond by calling them idiots because I never said that people of all other Ethnicities didn`t also have negative character traits. This sort of behaviour is sophmoric at best, and I have no time for it or for your ilk. I may not know who you are, but I`ve seen enough to end things here.

I hope your juvenile reaction to our serious objections is helpful to others in seeing the kind of thoughtlessness that typically characterizes, in my experience, your side of the debate.

Neil said...

Dear fucking idiot (who hides behind the tag anonymous, and then has the cheek to demand my real name. I am using my real name). I offered you half an apology, and told you why it was only half: because you started the name calling (whiners, crappy teachers). You respond by more name calling. You are a fucking idiot. You also can't read. I agreed that there was a practical issue: ie, that this I am not, as you say, pretending this 'an idle, intellectual consideration'. Because teaching evaluations are worse than useless, they should not be used. Got idiot? It's a practical syllogism (fill in the premises for yourself). There may be plenty of bad teachers out there; I frankly don't know. There may be methods of improving teaching; I don't know that either. I do know that because teaching evaluations are worse than useless, they should not be used. How this is supposed to commit me to elitism or condescension is beyond me. I suppose you, with your powers of idiocy, you can see into my soul. Perhaps you are a good teacher: I can't tell. It is hard to believe you are a good philosopher, though.

om said...

Neil - it seems it's you who's making a very simple fallacy here:

Anon implied that MANY (not all) of those who complain about student evaluations do so (at least partly) because they don't care enough about teaching to begin with. He also said that this motivation explains why they often fail to accompany their criticism with constructive suggestions about alternatives. (From anecdotal, though by no means scientific, observation, these claims seem true enough -- again, not about all such critics, but about a considerable number of them.)

He neither said nor implied that ALL criticism of student evals that's unaccompanied by alternative suggestions is NECESSARILY so motivated. In other words, he didn't say that such criticism COMMITS you to any further view or attitude; he merely said that it's often accompanied and motivated by certain attitudes.

Nor did he say or imply anything about the attitudes and motivations of any particular person, e.g. one Neil; so your complaint about his alleged pretense to "look into your soul" is completely off.

Anonymous said...

Thanks very much, Om, for helping to clarify where Neil has gone wrong in trying to understand what is at issue in my comment. I don't see much point in discussing things with him any further, for what I take to be obvious reasons; but Glaucon made a typical sort of comment earlier that is more thoughtful (though I disagree with it), and I'd like to address that.

Glaucon's comment had to do with attending a chamber music concert and wondering whether he is in any position to judge the musicianship of the performers to any serious level. The analogy, I take it, is meant to hold between the musicians and us (those who have good grounds for knowledge of whether we are any good, since the musicians/we have studied music/philosophy for years); and between the audience and our students (whose knowledge of the subject the musicians/we are trained in is negligible, at best).

However, this comparison, and many other complaints I have heard over the years, seem to involve a confusion about what it is student evaluations are meant to assess. Students are not being asked to assess whether our philosophical views are correct or well-considered, but about their impressions about our skill at _teaching_ philosophy. And by and large, this is an area in which we tend to have no more formal training than our students do. So the chamber music analogy collapses.

If the data really do clearly suggest (and as previous posters in this thread have shown, this is quite doubtful) that one should not rely on one`s student evaluations as a guide to one`s teaching merits, then what about the alternatives?

Is there any good reason for us to rely on our own introspective judgments about how well we are teaching? Is there any good reason for us to think that our colleagues are better-equipped to evaluate our effectiveness as teachers than our students are? Where`s the evidence for that? It is clearly an error to assume from the fact that X is a good judge of philosophical merit that X is a good judge of philosophical _teaching_ merit (since one can easily imagine people who are very good at doing philosophy but can`t keep clear in their heads what students need to hear, and which order they need to hear it in, for them to understand the issue. Actually, I`ve met many people who seem to have that characteristic).

So what is one to rely on, if not teaching evaluations, introspective judgments, peer reviews, or a recollection of how one was taught by one`s own teachers (all of which are ruled out if we adopt the criterion that nobody with specific and advanced training in a particular field should be appealed to as having useful information as to one`s merits in that field, i.e. _teaching_?), or the criterion that we should only make these decisions on what has been tested with double-blinding?

One could, as suggested, adjust one`s teaching strictly and solely on the basis of purely objective, double-blinded tests regarding whether students have learned a range of key points or developed new skills over the class. I think such measurements of effectiveness would be good to look at, but I don`t see how they could tell us all that is important. The problem is not only that what is of value is difficult to measure accurately and objectively (e.g. how much the student has learned from taking your course, rather than others taken simultaneously or in the previous term, about whether a technical side issue is relevant enough to discuss in a certain paper), but also because there seems to be disagreement among instructors about what being a good instructor amounts to.

For instance, suppose that, during a typical term, Instructor X gets 3 out of 10 of her students to become 40% better at philosophy (supposing we find a way to measure that accurately), while the other 7 show only a 5-10% increase; Instructor Y gets all of his students to become 25% better at philosophy, but 4 out of 10 students decide not to take further philosophy courses because they found the course boring; and Instructor Z gets 2 of his students to have 50% improvement in philosophy, and 8 to improve during that course by only 10%; but of those final 8, 3 students who would otherwise have never taken more philosophy decide to take further courses and 2 of these end up making an improvement of 40% in the next year because they come out to Instructor Z`s extracurricular philosophy discussion group. Who is better: X, Y, or Z? Does anyone here have a clear, objective answer to this question? Has anyone even tried seriously to work it out, up to now?

So we seem to have two main options: we can take these sorts of sceptical worries about common sense seriously, and be led into a position of scepticism regarding all criteria (including our own introspective assessment) of how well we are teaching; or we can adopt a common-sense approach where we try to be suitably well-informed but muddle through as best we can. In the first case, we should ignore student evaluations only because we should also ignore anything else, and nobody seems better-placed than anyone else to have any knowledge about whether anyone is any good at teaching. In the second, it is unclear why we should not take our evaluations seriously.

Puffnstuff said...

Yeah, right, Om. Anon @ 1:52 AM didn't insult Neil specifically. S/he instead smeared quite generally most who criticize teaching evaluations (w/o offering alternatives) as “just crappy teachers, or friends of crappy teachers, looking to safeguard their privilege at the public expense."

Anon then proceeded to smear quite generally those who teach philosophy as people who "exhibit such scorn for those whose education they are amply compensated for improving that they bristle even at the minimal efforts made to hold them slightly accountable...." We're also said to belong to a "ridiculously privileged elite," many of the members of which "are helping themselves to the remarkable compensations of our work while refusing to be held accountable for poor work standards."

Since Anon speaks anecdotally, I will, too. At my school, un-tenured academics in all disciplines live in perpetual fear of teaching evaluations. Every term, senior professors sit in on our classes to evaluate our teaching. But it doesn't matter if you get rave reviews from every one of those senior academics -- if you get even some poor teaching evaluations from students, you very quickly learn that your tenure prospects are shaky. You learn quickly to start giving higher marks, to accept any student excuse for missing tests, to stop trying to be a gadfly in class or to provoke the students in any way. Otherwise, enough students will conclude you're mean, in which case, you're toast.

You still haven't answered Neil's straightforward point -- these evaluations are not just ineffective but neutral. They are positively harmful. Rampant grade inflation, the weeding out of 'nerdy' teachers, sexist evaluation procedures, etc.

And since you like to project hateful attitudes into other people based on 'context', I'll do likewise. In my experience, people who speak of academics as a 'privileged elite' that helps itself to its perks at 'public expense' and that stands in marked contrast to 'most working people' have no idea of what academic work involves and, so, are probably not themselves academics. They are, typically, Fox-news dolts. (Let me be clear -- I'm not saying this specifically about Anon @ 1:52 AM.)

Anonymous said...

Anon at 12:40 PM,

You raise an interesting point with respect to evaluating teachers like X, Y, and Z. And in light of your concerns, I would think the best approach would be to settle on a standard (probably somewhat, though not completely, arbitrary) and then develop tests to compare teachers to that standard. That seems like a useful thing to do, and I would be happy to work under the threat of losing my job for failing to meet such a standard.

However, I don't understand how student evaluations are supposed to solve, even in part, the problem raised by your example of X, Y, and Z. On the contrary, I expect that students have as many diverse opinions about what makes a teacher good, interesting, or effective as we do. So, they probably have different opinions with respect to X, Y, and Z.

Since we haven't yet settled on a standard, we don't know whether student evaluations give us a measure against that standard. However, I think we all agree (correct me if I'm wrong) that things like being physically attractive, fitting a gender-role stereotype, and artificially inflating grades are *not* marks of effective teachers, whether they fit the mold of X, Y, or Z otherwise. The main problem with student evaluations is that they track these attributes as much as or more than attributes that we might want to measure, e.g. how much did students learn and how much have they been inspired to pursue the discipline further. At least, that is the way I see the evidence pointing. Perhaps you don't agree with the findings. That's fair. But then it seems to me that before putting anyone's career on the line on the basis of student evaluations, we should be more sure, and that means conducting more, better designed experiments. Or perhaps you are optimistic (as a previous poster appears to be) that a good enough questionnaire or a good enough statistical model will be able to control for the noise. I am not.

I hope I am not being condescending here (either to you or to our students). I don't think any of this reflects badly on students. Even very smart people often cannot correctly say why they act or judge the way they do. If we could introspect like that, there would be little need for experimental psychology.

Also (in closing ... these are getting long-winded, eh?) since I recommended a pre-test/post-test model in answer to an earlier question about alternatives to student evaluations, let me be clear that I am in favor of *replacing* student evaluations not simply supplementing them.

Anonymous said...

Dear Puffinstuff and Anon 12:40:

Thanks for your interesting comments.

Puffinstuff, that is interesting that your school takes student evaluations so seriously in making its tenure decisions. Schools I am familiar with do not, so it's good for me to hear about this. I agree with you that there does not seem to be, prima facie, any good reason for putting more weight on student evaluations than on peer evaluations (though I could imagine some defeaters, such as if it turns out that instructors tend to go easier on fellow faculty members' weaknesses in the 'I'll scratch your back, you scratch mine' approach, or if it turns out that, for some reason, instructors have a more difficult time than students in identifying instructors who 'lose' otherwise intelligent and hard-working students by not explaining core concepts clearly enough).

Incidentally, I am famous among students at my school for being a tough marker who demands a great deal of work from my students. And my student evaluations mention this with warm praise. Colleagues of mine who dumb down their courses, by contrast, seem to lose students. My logic professor back in my undergraduate years used to fail at least half his students, but he was great at teaching and his reviews were always outstanding. So I question the correlation between high reviews and easy grades. Students, in my experience, love it when a course presents difficult material and offers them a chance to acheive high standards, but (here's the important point) the instructor puts in whatever work is necessary to help students to achieve those high goals.

Anon 12:40, I should clarify: I agree that my example of Instructors X, Y and Z does not at all show the superiority of student evaluations over other evaluations. I, like you, doubt that students would have any better insights into these sorts of issues than we do. But my overall point was not that student evaluations are a particularly effective method of determining instructor skill. The target of my annoyance is, rather, those individuals who (out of personal fear or elitism or whatever) want to promote a system in which their failures at instruction, no matter how acute, cannot be grounds for their dismissal. That seems to me to be a clearly bad system for any job worth doing.

If it turns out that there are other methods of making these evaluations that are more effective than student evaluations, then I really am all for switching to that instead. However, I wanted to show (in my last post) that many of the critiques given against student evaluations so far are, for all we know, equally effective against any other alternatives we can think of.

That being the case, it seems at least somewhat strange that there is such a fixation on worrying about student evaluations in particular. Perhaps, I might suggest, these complainers are by and large not so much interested in improving the checks on bad teaching, but rather opposed to them because they think there should be no such checks.

Anonymous said...

Om, here's the paragraph from anonymous in question:

Here's something interesting: how many of the whiners about student evaluations have anything better to suggest? How many of them propose even a single thoughtful (or even unthoughtful) alternative way to ensure that those who can't teach well either shape up or ship out? Of course they don't, because they're not really interested in finding better ways of getting the relevant information. They're just crappy teachers, or friends of crappy teachers, looking to safeguard their privilege at the public expense, or are touting their membership in a club whose hallmarks are a cynical scorn for their professional duties and an unhealthy desire to circle the wagons in the face of any threat to the status quo.

Note the absence of scope restrictions: all of us 'whiners' are 'crappy teachers' or friends of same.

Second, suppose you were right (perhaps you have a special ability to detect scope restrictions like Anonymous's ability to see into souls). Suppose that anonymous was only able to see that *most* whiners are crappy teachers. He is still attributing to himself an ability to see into souls, just not one at a time. He is still an idiot.

He didn't teach you informal logic, did he?

Neil

Anonymous said...

Neil, I am apparently not as troubled as you are by these (as yet undefended) sceptical worries about 'looking into others' souls', if you insist on putting it that way. I feel that I can, in some instances, quite reliably discern characteristics and intentions in other people on the basis of how they act. For instance, given how you have conducted yourself so far, it would be rather surprising to me (and presumably to many others reading this as well) if you turned out _not_ to be a misanthrope, an asshole, and a generally juvenile person. If I observed you acting like this a great deal of the time, no doubt I would have even better grounds for that conclusion.

And yes, of course what I meant to say was that _most_ of those whiners who suggest no alternatives seem motivated by socially undesirable motives. I acknowledge the lack of clarity in my phrasing. Sorry about that.

Neil said...

Anonymous, thanks for the graceful admission of the lack of clarity in what you say. Apparently being a whiner can me motivated by other things. Note again the scope of 'whiner' - not, I think you will acknowledge, a neutrally descriptive word. Now some questions: how is it whining to say that teaching evaluations are a waste of time bc they do more harm than good? Isn't it in fact a duty of any academic to point to the existence of a wasteful (of time and resources) and harmful practice? What is the evidence that there is some (you now say defeasible) connection between pointing out that teaching evaluations and socially undesirable motivations? Third, have you considered that your confidence in detecting my qualities on the basis of a few pieces of evidence is misplaced, and indeed a reflection of the very kind of problem (inability to track the quality of interest, combined with a false confidence that one can track the quality) that are in fact characteristic of teaching evaluations?

Anonymous said...

To answer those three questions:

1) Looking over the posts of this thread, I see much more than people merely asserting or arguing, on empirical grounds, that student evaluations are ineffective. Many of the posters are engaging in a whine-and-sympathy fest. It was never my claim that everyone who holds or argues for the ineffectiveness of student evaluations is ipso facto whining. You really need to learn to read more carefully. This is the second time you've gone out of control attacking a straw man by misunderstanding what I said. Perhaps you think you can look into _my_ soul and see that I hold the proposition "All critics of student evaluations are whiners", even when I never said that...?

2) My evidence for holding that those who whine about student evaluations tend to have socially undesirable characteristics is as follows: I have listened to an endless (not literally endless, Neil -- it's hyperbole. Get it?) string of rants from different instructors about student evaluations. After awhile, finding it tiresome, I started suggesting that, one way or another, we ought to have in place _some_ reliable way of measuring which instructors are bad teachers so that they can be retrained or got rid of, since I'm sick of the profession being brought down by bad teachers. The response, in almost all cases, has been shock that anyone would even suggest that those of us who don't do their work well, even after warnings, should not continue to be employed. When this is not stated explicitly, I ask them point-blank whether they think that bad teachers should be let go of. They nearly always say no. Now, people who earn money at the public expense (and at the expense of our students), and who do not want to be held accountable for their work, lack the virtues of civic-mindedness and responsibility. These people also tend to adopt a very snobby attitude toward their students, and to think that they are somehow better human beings than their students are, while taking the students' money and putting forth little consideration toward them (in my experience). I take it to be clear that these are socially undesirable traits. I also expect that the people I have seen are not unusual among those who produce similar rants. Therefore, by induction, I suspect that whiners about student evaluations tend to have undesirable social traits.

3) I don't think the worry you suggest is a serious one -- at least, not as you present it (and without any supporting evidence or argument).

Glaucon said...

Wow! Spend a couple of days grading and you miss all the fun.

Anon @ 12:40pm suggests that I am "confus[ed] about what it is student evaluations are meant to assess." Luckily, Anon is here to set me straight! "Students are not being asked to assess whether our philosophical views are correct or well-considered, but about their impressions about our skill at _teaching_ philosophy. And by and large, this is an area in which we tend to have no more formal training than our students do. So the chamber music analogy collapses."

Why in the world would you take me to have suggested that students are being asked to evaluate anything but our teaching? The analogy was between evaluating musical performance and evaluating teaching. The point was that most of us are in a position to say that we like or dislike x, but fewer of us are in a position to say whether x is good or not. No doubt students are pretty good at saying what they like and don't like, but that's different from their being good at evaluating teaching. Why would you have taken me to be suggesting anything else?

Does my doubting that students are by and large not qualified to do more than express their likes and dislikes about teaching make me a whiner and a crap teacher who possesses "cynical scorn for [his] professional duties and an unhealthy desire to circle the wagons in the face of any threat to the status quo"?

Hmmm. As my sainted Irish grandmother would have said, "Póg ma thoin!"

Neil said...

I'm going to end this exchange now (yes, I'll let you have the last word: I don't doubt you'll take it) with just a few reminders of things you ought to know but apparently don't. You can't know the things you think you know. The scientific method - yes, the method that demonstrates that student evaluations are worse than useless - was invented because of the everpresent danger of confirmation bias, salience effects, and so on - the kinds of things which bring it about that the 'evidence' you cite isn't evidence at all. That's why the worry I suggest is a serious one. That's why your insults are not merely juvenile but misplaced. That's why you quite literally don't know what you're talking about. Unfortunately I have no doubt that this exchange has left you more deeply entrenched not in your false views, but also in your confidence that you can detect the secret motivations of those who disagree with them. This would be merely sad were it not for the fact that you're a teacher, and therefore are probably passing on your nonsense.

Anonymous said...

Hang on, Glaucon. You need to think the thing through more carefully.

You say that your analogy is "between evaluating musical performance and evaluating teaching." Yes?

Well, think about it: the musicians are well-trained in musical performance. Right? That's their central area of training. Philosophy instructors, by contrast, tend not to have a background in _teaching_ philosophy. That is not their central area of training, and is often not an area in which they've been trained at all. Their background is in _philosophy_.

And that was my point: we philosophy teachers, unlike musical performers, do not seem to be in a much better position than our students at evaluating what sorts of teaching strategies and practices are good. Hence, as I argued, the analogy breaks down: while those of us amateurs who attend a chamber music concert are speaking to trained experts on something outside of our area of expertise if we were to comment on their performance style, the same does not seem to hold for students evaluating us.

Your grandmother, incidentally, is welcome to pog her own thoin.

Anonymous said...

Neil, how can I take the last word alone, when you've done such a good job yourself? Let me share this final comment with you. These two statements from your posts resonate together so beautifully:

"I suppose you, with your powers of idiocy, you can see into my soul...You can't know the things you think you know."

-and-

"I have NO DOUBT that this exchange has left you more deeply entrenched not in your false views, but also in your confidence that you can detect the secret motivations of those who disagree with them." [emphasis added to accentuate the hilarious, but consistent, hypocricy].

Glaucon said...

Anon,

I'm puzzled. Then your point seems to be that teacher evaluations of teaching and student evaluations of teaching are equally worthless -- hardly a robust defense of the latter or justification of the judgment that most philosopher teachers are "crappy teachers" etc.

You write: "we philosophy teachers, unlike musical performers, do not seem to be in a much better position than our students at evaluating what sorts of teaching strategies and practices are good."

I've received very little formal training in teaching, but I think I can get a sense from a batch of essays that a particular strategy worked or not, that I should've emphasized a distinction more than I did -- or that I couldn't have emphasized it any more than I did but that most of these students didn't seem to get it, perhaps for reasons that concern them rather than me, or perhaps because I could've done better.

om said...

Neil: "perhaps you have a special ability to detect scope restrictions"

I do. It's called "applying the principle of charity."

Neil: "He is still attributing to himself an ability to see into souls, just not one at a time."

umm... it's called basic psychological interpretation. You know, where people interpret each other's unstated attitudes and motivations. I hear normal people engage in this activity on a regular basis. I also hear that certain kinds of psychopaths are incapable of it.

Anonymous said...

Glaucon wrote: "Then your point seems to be that teacher evaluations of teaching and student evaluations of teaching are equally worthless -- hardly a robust defense of the latter or justification of the judgment that most philosopher teachers are "crappy teachers" etc."

First off, Glaucon, I'm perplexed as to why you think this was my argument: did you really think I was arguing that _since_ we have no reason to think that professional philosophers are more knowledgeable about teaching than students, it _follows_ that most philosophy teachers are bad at what they do? Whatever made you think that I was arguing that way? My reasons for thinking that many philosophy teachers are bad are quite different.

Here's what I'm actually saying. I take it (please correct me if I'm wrong) that your chamber music analogy was brought in in support of the main thesis of the thread (i.e. that instructors should ignore their teaching evaluations). I wanted to show that that argument by analogy is not successful here.

My way of showing that there is a problem with your argument by analogy is to point out that there is a relevant difference between the two cases. In the case of the chamber music concert, the performers have, as a result of their training, a great deal of technical knowledge of the subject in question (i.e. how to play chamber music well). For that reason, if I (someone without any such training) were to approach a musician after the concert and say "I think you need to work on your embouchure", that musician -- once she determines that I have no musical training -- can rightly ignore my comment as mere presumptuousness.

By contrast, we philosophy instructors tend not to have much, if any, specialized training in how to teach effectively. Hence, the thing that made it perhaps reasonable for the chamber musician to ignore the comment from the uninformed audience member -- the great disparity in relevant training -- does not occur in the context of teaching evaluations.

Now, that is not, of course, to say that there cannot be special reasons that would justify ignoring _particular_ comments by students. Obviously, a student who only mentions that an instructor is bald, or any of the other ridiculous things that some people mentioned above, should not be taken seriously. And even the comments that seem more worthy of consideration (e.g. that the instructor talks too quickly, or seems to stumble around too much and to be unprepared for class, or loses the attention of the class by reading from notes rather than making eye contact, or doesn't seem to give due consideration to alternative views, or ought to stop letting slackers turn in late assignments without penalty because it saps others' motivations) should not be accepted thoughtlessly. Obviously, instructors need to use their heads. But the situation is not analogous to the chamber music one, and the advice to _ignore_ one's teaching evaluations outright is typical of the horribly irresponsible, snotty attitude that is too pervasive in academia and that fosters bad teaching habits.


You write: "we philosophy teachers, unlike musical performers, do not seem to be in a much better position than our students at evaluating what sorts of teaching strategies and practices are good."

I've received very little formal training in teaching, but I think I can get a sense from a batch of essays that a particular strategy worked or not, that I should've emphasized a distinction more than I did -- or that I couldn't have emphasized it any more than I did but that most of these students didn't seem to get it, perhaps for reasons that concern them rather than me, or perhaps because I could've done better.

Anonymous said...

Sorry, the last two paragraphs were from your (Glaucon's) post and weren't meant to be at the end of mine.

Anonymous said...

Anon 3:23 PM,

I think you've missed Glaucon's point, which is simply this:

Premiss 1: Philosophers are no better at evaluating teaching ability than are students
Premiss 2: Students are poor evaluators of teaching ability.
Conclusion: Philosophers are poor (or worse than poor) evaluators of teaching ability.

You have argued for the first premiss. Glaucon thinks the studies discussed earlier support the second premiss. (And I agree with him.) There is no claim here about the quality of philosophy education, only a claim about evaluation.

In any event, I'd like to hear again, as slowly and carefully as you can, your reasons for skepticism about the second premiss and your reasons for thinking that accepting the second premiss carries with it some more general indictment of common sense.

Anonymous said...

I'm a little confused, Anon 6:19. As you would have it, Glaucon's conclusion is that philosophers tend not to be good at evaluating their own, or others', teaching ability. Is that really what Glaucon is arguing for? If so, then I agree! We tend not to be good at doing that, and tend to need all the help we can get. We also need to be held accountable for doing a bad job when we do a bad job.

I took it that the aim of Glaucon's chamber music analogy was to support the central claim made by the originating post of this thread: namely, the claim that philosophy instructors should ignore, and in fact _not even read_, their teaching evaluations. I've been arguing against that.

As for your request, well... however good or bad the average student evaluation is (and there seems to be a difference of opinion here about which conclusion the research supports on that), it seems undoubtable that at least _some_ students are good -- at least, good enough to listen to -- at pointing out areas of difficulty or strength in one's teaching. Does this mean that student evaluations should be accepted as true uncritically? Of course not. But throwing them away unread just because they might be wrong (when the wrong ones will, it seems from the silly examples given, often be reasonably easy to detect) seems unwarranted and irresponsible.

The part about common sense is this: I argued that, if one thinks that one has absolutely nothing to learn about one's own teaching ability (or the teaching ability of someone who is under review) from student evaluations on the grounds that students do not have good training in the relevant area and/or are biased and unreliable, then I think there are also good grounds for thinking that the same thing is true of peer reviews or self-reviews. And then it seems to follow that we must either accept complete scepticism (i.e. grounds for absolute dismissal with no consideration) about student reviews, but also about self-reviews or peer reviews. And the conclusion then seems to be that the profession of teaching philosophy has the remarkable property that there is no basis whatsoever for determining whether anyone is doing it well, or badly, or even better or worse than anyone else. That seems, to me, to fly in the face of common sense.

Anonymous said...

Ah. I take it the chamber music example *does* do what you and Glaucon say it does -- it undercuts the value of student evaluations. Glaucon goes on to point out that your criticism of the analogy (at best) only goes to show that peer evaluations are no better than student evaluations. But this doesn't show that the analogy fails. After all, the analogy was aimed at showing that students aren't specially equipped to evaluate teaching ability.

I am happy with that conclusion, since I think that better measures are available, though more difficult to design and more time-consuming to implement. I don't trust introspection in philosophy at large, so I think I'm perfectly consistent not to trust it with respect to teaching evaluations. If that isn't consistent with common sense, then so much the worse for common sense.

Your reply about why you reject Premiss 2 makes me think we are talking past one another. I take it that what is at stake is average score on a five or seven point scale for questions like "Rate the overall quality of the instructor." You seem to have in mind individual responses in essay form, where (I admit) students *do* sometimes have useful things to say. But if you or anyone else has given good reasons for doubting the empirical work on average scores on the numerical parts of evaluations, then I must have missed them. If you have such reasons, I would like to hear them. If not, then I would like to know why you think those average numbers should be used for making hiring/firing decisions. And if you don't think the averages should be used, then how do you propose to make use of individual student comments?

Anonymous said...

Hi again,

As I understood it, the aim of Glaucon's argument by analogy with chamber music was that, just as chamber musicians have special training that justifies them in utterly disregarding comments on their playing by their lay audiences, we philosophy instructors have special training that justifies us in utterly disregarding comments on our teaching by our students. My objection to the argument was that we do not in fact have this relevant training (i.e. training in teaching). Hence, his argument fails.

You suggest that individual comments by students, of the sort I mentioned, appear significantly more useful and worthy of consideration than the numerical scores they offer, which (I imagine, because they are quantitative) are the scores that some of the literature has found to be biased or otherwise unreliable. OK -- if that's correct, then maybe that's a good reason for paying more attention to the verbal ratings.

I am not arguing that one rely on the numerical scores for hiring/firing decisions.

The reviews could be used by instructors, or by those deciding whether to retain the services of an instructor, in a number of ways. Again, I certainly don't recommend that anyone use them uncritically. However, at the very least, they could be used to help identify key areas of strength or difficulty. For instance, if I were on a committee that had to evaluate the teaching strengths of Professor A, and a sizable number of the student reviews I looked at said that Professor A presents materials in such a confusing manner and ill-thought-through manner that everyone gets put to sleep, and if (moreover) students complain that they actually unlearn information because Professor A is so confused, then that would presumably give me a good reason to look into the matter in whatever way would be most appropriate. Professor A might also, I venture, benefit from reading these critiques. Professor A might also benefit from reading that his students feel they learn more from his interactive lectures than from his attempts to have the class break up into small groups, which some students find frustrating.

Is there a secret formula for translating these sorts of comments into practical guidance, or a hiring decision? Surely not. But to ignore them altogether -- to not even _read_ them, when assessing yourself or your colleague -- strikes me as extremely arrogant and irresponsible.

Glaucon said...

My chamber music analogy was fueled by my just having been to a chamber music concert, and that seems to have implied something I didn't mean to imply, for my point had nothing to do with the special training chamber musicians have. The point was that there's a distinction between what I like and what's good, and I don't think many students--and of course not just students--are very good at drawing that distinction. I don't think many if any students are competent judges (in Hume's sense) of teaching quality, so I don't think they're competent to evaluate teaching in a meaningful sense. Nothing I've read in my toe-dipping into the relevant literature suggests otherwise. Student feedback can provide data that someone competent to evaluate teaching can take into account. And, as I said before, I don't think that it thereby follows that I am a crappy teacher who shirks his professional duties. (Of course, I may be a crappy teacher and duty-shirker, but the argument that anyone who doubts the value of student "evaluations" of teaching is a crappy teacher is quite plainly unsound.)

Anonymous said...

Glaucon, you're still attacking a straw man. I thought I (and Om) had explained this point enough times already in this thread, but apparently not.

It is _NOT_ my argument (and incidentally, it isn't an argument at all, but a general statement) that "anyone who doubts the value of student 'evaluations' of teaching is a crappy teacher".

As Om concisely put it back when Neil was having trouble reading, what I'm actually saying is "that MANY (not all) of those who complain about student evaluations do so (at least partly) because they don't care enough about teaching to begin with... [and] that this motivation explains why they often fail to accompany their criticism with constructive suggestions about alternatives."

Anonymous said...

Jeez! Can we stop with the name-calling?? If you're going to say that many or most or all (doesn't much matter which one given the context) people who criticize the use of student evaluations are crappy teachers, you really ought to have something more than anecdotal evidence. Admittedly, it's an empirical question, which we could put some money on if you want. But until evidence comes in, it's just insulting.

And on the other side, grow some thicker skins! So somebody posted a claim that you are or are likely to be or might possibly be a crappy teacher. So what? It's abusive, but get over it.

So, let's drop the name-calling and stick to substance.

Anonymous said...

Fair enough. There does seem to be good (as good as such evidence can be) anecdotal evidence to support the view that many people who are motivated to complain at length about teaching evaluations without offering any alternatives, and who suggest that instructors just discart their evaluations without reading them, tend to be inferior teachers and/or to be generally in favour of a system that does not hold teachers accountable for their merits at that job. It also would not be surprising if this were the case, since there is clearly a strong motivation for bad teachers and/or people who don't want to be held accountable to want to avoid them.

Admittedly, this evidence is not conclusive; but it does seem strong enough (as far as I've seen) to support at least a prima facie assumption that this trend exists.

It would therefore, I venture, be the position to hold unless there is good empirical evidence to the contrary. Does anyone here know of any such evidence? Have any studies been done investigating the link between opposition to teaching evaluations and bad teaching and/or elitism and/or a preference toward unaccountability?

Anonymous said...

On the basis of careful observation (though I admit that the evidence is only anecdotal), I have come to the conclusion that people who complain about people who complain about teaching evaluations are stupid, pompous arseholes. Of course there is some small possibility that I am wrong. Does anyone know of any studies which might bear on this?

Anonymous said...

I can think of a number. They're filed up your ass.

English Jerk said...

Anon. @ 3:59:

1) It would be much easier for others to keep track of your various posts, and thus to work out exactly what your claims are, if you chose a name.

2) Your original claim (which you have since repeated) was that “the whiners about student evaluations” do not have “a single thoughtful (or even unthoughtful) alternative way to ensure that those who can't teach well either shape up or ship out” (5/15 1:52a). This claim is demonstrably false with respect to those who have commented here, as you can discover by reading the posts above yours (or by reading them more carefully). My post, for example, was quite explicit in proposing that peer evaluation should replace student evaluation when it comes to deciding whether to tenure, to promote, or to fire someone.

3) You then claimed that the critics of student evaluations fail to propose alternatives “because they're not really interested in finding better ways of getting the relevant information. They're just crappy teachers, or friends of crappy teachers, looking to safeguard their privilege at the public expense” (ibid.). This is an empirical claim, as you sometimes acknowledge, and the only evidence you have provided is “anecdotal,” i.e., your own testimony about your experience, which you think gives you good prima facie reason to suppose your explanation to be true (5/18 4:55p). But it can’t possibly convince anyone else; even if they have the same experience themselves, it is their own experience that justifies their belief not your testimony about your experience. And many of us, after all, don’t have your experience. I, for example, don’t think student evaluations provide any real basis for evaluating the quality of teaching (for the reasons I gave above), but I consistently score higher than my department average on evaluations (which should, if you’re right about evaluations, prove that I’m not a “crappy teacher”), I am not paid “at the public expense” since I work at a private institution, and I have no interest in protecting “crappy teachers.” I think people who are incompetent in any area of their professional duties should not be hired in the first place; and if they’re already employed, then I’m happy to concur with your suggestion that “instructors who do a bad job consistently [be] first warned, then trained to do a better job (if the warning doesn't work), and then fired (if the retraining doesn't work)” (5/15 5:16a). So based on my own anecdotal evidence, I have good prima facie reason to deny your causal claim. The only way you’ll convince me is if you can provide empirical evidence that at least meets the most basic standards of rational inquiry (which your anonymous testimony does not).

4) But are you really interested in convincing people like me? The clear purpose of your original post was to discredit a view you disagree with on the grounds that its advocates have moral defects of various kinds (whining, poor teaching, elitism, etc., etc.). As a philosopher, you should recognize that this is an ad hominem argument (however implicit) and commits you to a fallacious inference. Others have already pointed this out to you, but you insist on repeating the claim, so perhaps more explanation is required. The moral virtues or defects of a person do not have any connection with the truth or falsity of that person’s claims. If it is true that “Murder is wrong,” then it is true regardless of who says it (even if the person saying it is a serial killer, or a liar, or whatever); that’s just what it means to be true. But surely you know this. And surely you also know that you’re not going to convince your interlocutors by insulting them. So why do you keep posting, and posting at such length? I find this genuinely mysterious.

5) The above-cited line of reasoning is ad hominem, but it’s also a red herring. The question at issue is not whether incompetent teachers should be fired (surely many of us would agree with you about that). The question Spiros originally posed, and that the posts before yours discuss, is whether teaching evaluations are good for anything, and thus worth reading. As I see it, three rational positions have been defended thus far:

A) Student evaluations both provide useful feedback to teachers about what students like or dislike and they provide a substantive source of information on the basis of which one can determine the quality of someone’s teaching (especially when combined with other sources of information like peer evaluation). This is Mike A.’s position, as I understand it.

B) Student evaluations may not provide a substantive evaluation of the objective quality of teaching, but they do provide information about what students like and dislike and thus allow one to judge the effectiveness of (some aspects of) one’s performance. They may not be useful for administrative decisions, but they are still worth reading for the instructor’s own purposes. This is 729’s position, as I understand it.

C) Student evaluations at best reflect only what students like or don’t like, and they do not measure teaching competence in any way, because students don’t know enough about teaching to make informed judgments about it. Making students happy may help them learn, but we are paid to help them learn not to make them happy; it is thus the learning, and not the instrument of learning (liking/disliking), that should be measured if one wants to evaluate our performance. Moreover, it is at least possible that in some circumstances making students unhappy might help them learn: they might benefit from being made uncomfortable, or from experiencing confusion, or from having definitive solutions endlessly withheld. Administrative reliance on teaching evaluations thus needlessly limits the range of pedagogical methods at our disposal (not to mention encouraging grade inflation, etc.). This is my position (and, if I’m lucky, Glaucon’s).

Your position (whose target seems to be position C) involves claiming that students and professors have exactly equal competence in assessing teaching, and that we therefore must choose either (a) not evaluating teaching in any way whatsoever, or (b) relying on student evaluations to evaluate teaching (5/16 12:40p). This is a false dilemma, since there are obviously other options that are consistent with your initial premise (e.g., having someone with a doctorate in education evaluate our teaching).

6) More importantly, your initial premise (the equation of student and teacher competence in evaluating teaching) is, I think, both profoundly counter-intuitive and clearly false. Glaucon’s given some good reasons to doubt it, to which I’ll add in a moment. But let’s recall your original argument in support of this premise: “It is clearly an error to assume from the fact that X is a good judge of philosophical merit that X is a good judge of philosophical _teaching_ merit (since one can easily imagine people who are very good at doing philosophy but can’t keep clear in their heads what students need to hear, and which order they need to hear it in, for them to understand the issue. Actually, I’ve met many people who seem to have that characteristic)” (5/16 12:40p). You are quite right that this inference is false, but it seems to me that this is another red herring. I could be mistaken (this is a long thread), but I don’t recall anyone suggesting that philosophers are in a position to judge teaching purely by virtue of their training in philosophy. So this is an argument against an implausible defense of the contrary position, but it is not an argument in favor of your position.

7) And there are good reasons to think that your position is wrong. First, your position depends on all professors having no training in teaching, which is patently not the case. For example, the graduate program in which I was trained (at a large Ivy) involved substantial training in teaching. But even if we suppose that some professor P has had no formal training in teaching, professor P surely has experience teaching, which students presumably do not. (To simplify matters, let’s assume we’re talking about students who are not education majors or somesuch, since they’d be at best a minority.) So the question comes down to whether experience doing X makes one better able to assess a given performance of X than a person who has neither experience nor training in X. For example, say that our friend Pete has received no formal training in plumbing, but has, over the course of ten years, plumbed widely and plumbed much. Will Pete be better able to judge a good plumbing job than a plumbing ignoramus like me? Surely he will: people get better with practice, and one of the things that getting better at performing X depends on is the ability to recognize the defects in our past performances of X (i.e., to evaluate performances of X). So my point is that, at a bare minimum, professors will be better judges of teaching than students because professors have experience at teaching. On top of that, many professors do have at least some formal training in teaching. And some aspects of their research training will also surely apply to evaluating teaching (e.g., assessing the clarity, coherence, or soundness of a line of reasoning presented in a lecture). Professors are thus, ceteris paribus, in a better position than students to evaluate teaching. And since I see no reason to think that aggregates of students have any reasonable basis at all for evaluating teaching, professors, not students, should be doing the evaluating. Where members of a department are unwilling or unable to adequately evaluate one another, surely the solution is not to rely on student evaluations but to seek outside faculty evaluators (people from other departments or from other schools). Peer evaluation, combined with observing the successes and failures of our students in performing the coursework in which they are to demonstrate the skills and knowledge we’re trying to help them learn, seems to me to be perfectly adequate for any conceivable evaluative purpose.

Anonymous said...

Thanks for your thoughtful comment, Englishjerk. I will see about getting a blog name. In the meantime, to address your most salient points:

I) Again, I must ask that the principle of charity be applied to what I said, particularly since I have since clarified it many times. Quoting again from Om's good, concise summary of what I have said: "MANY (not all) of those who complain about student evaluations do so (at least partly) because they don't care enough about teaching to begin with... [and] that this motivation explains why they often fail to accompany their criticism with constructive suggestions about alternatives." As you can see, this claim is consistent with there being people who complain about teaching evaluations who do not have these negative character traits. I apologize again for failing to write in explicitly the qualifiers that I felt an application of the principle of charity would make clear.

Hence -- counter to what you say in 2) -- the fact that some people have suggested alternatives does not render anything I have said 'demonstraby false'.

II) My original comment was not, once again, an argument: it was a critical _comment_ about the elitist and irresponsible attitudes I saw exhibited by many of those who contributed to this thread, and many others I encounter in academia. I don't know how I can make this any clearer: I wasn't trying to present an argument in my original comment. I made the comment I made to express my disapproval of the bad attitude I was seeing. I hoped that raising the issue I raised would help some of these people see that they were acting and speaking in a shameful manner, and that they were taking for granted (and it still seems to me that many of them are doing this) that they are somehow owed a living, whether or not they are doing a good job, and that they should exhibit such contempt for the opinions of those they are paid (by whomever) to educate that they should not even listen to any comments from them (see the originating post of the thread).

Since I was not making an argument, I was also not committing an ad hominem fallacy or a red herring fallacy. I was not attempting to demonstrate the falsity of an argument by attacking the character of the person who presented it, nor was I attempting to demonstrate the truth of a proposition by presenting irrelevant reasons.

III) You say in 5) and 6) that I claimed, against Glaucon, that teachers and students are equally good at judging teaching ability. However, that is not what I said. What I said is that, whereas Glaucon's argument by analogy depends on our having a reason to believe that we instructors have special expertise on the relevant areas, and that students do not, we do not, by and large, seem to have those reasons. There is a great difference, Englishjerk, between stating a)that we have no grounds to think that X is greater than Y and b)stating that X and Y are completely equal. It is a) that I said, not b).

Therefore, I have not constructed the false dichotomy you accuse me of, and the statement I _actually_ made does not seem "clearly false".

IV) Finally, to address your plumber analogy: I agree with you that a plumber who has worked successfully on a fair number of leaky pipes over the years has good grounds for thinking he has expertise in that area. However, he has this expertise because he can see that the pipes he works on have stopped leaking; that they do not start leaking again before he leaves the home of his customers; and that his customers do not call him back a week later to fix them again.

Many philosophy instructors, by contrast, do not have this sort of avenue for self-evaluation. They blame (who will stop them?) the failings of their students on their students; they don't keep up with students to see whether they improve, and whether this is correlated with their own teaching or someone else's (just imagine how hard it would be for a plumber to assess his or own work if ten other plumbers always worked on the same pipes at the same time!); and then, on top of it, they cut off the only remaining line of feedback from their work by refusing to listen to comments from their students. That's the problem.

Anonymous said...

Anonymous, I don't know how many times you've posted, but you *still* don't even understand what the debate is about. You talk about instructors "refusing to listen to comments from their students". The literature on evaluations is about scores, not comments. At many universities, scores are used in tenure decisions and for promotions: the literature shows that these scores do not track teaching quality. As Neil said (May 10), "this doesn't apply to comments". Maybe you've been talking at cross purposes the entire time.

Mostly Anonymous said...

There is charity and then there is charity. Are you seriously denying that your intent was to insult and discredit opponents of student evaluations?

I'm also not sure that I buy the "critical comment" dodge. Let me illustrate. Suppose there is a town-hall meeting (for a small city, say) at which a proposed restructuring of the local bureaucracy is presented to the public. Several council members argue that the restructuring would make the city government more effective. Then a certain citizen stands and says, "Politicians are out to line their own pockets and the pockets of their friends. They don't care anything about the public but are only interested in preserving their position of power." How should the citizen's statement be understood? Is it merely a critical comment about "some" politicians, or is it an implicit argument that *these* politicians should not be listened to and hence, that the restructuring not be supported?

Further, I don't know what you mean when you say, "many philosophy instructors ... do not have this sort of avenue for self-evaluation." Don't they ask their students to write papers and/or give their students examinations? If so, then the grades the students receive are indicative in the way leaks are indicative to a plumber, aren't they? It seems to me that the argument that supposes that instructors blame their students for their low grades won't serve to distinguish between philosophy instructors and plumbers, since (bad) plumbers are surely able to blame leaks on things other than their lack of ability (bad pipes, bad seals, poor customer maintenance, poor tools, too much customer interference, too little time, etc.) just as (bad) instructors are able to blame their students. Or is this maybe what you had in mind with the qualifier "many"? I really find that confusing.

Incidentally, I would have expected the experience argument in favor of peer evaluations to run like this. Experience as a student (recipient of teaching) improves one's ability to evaluate teaching performances. Since one's peers have (much) more experience *as students* than do one's own students, one's peers are better judges of one's teaching than are one's students.

Anonymous said...

Anonymous is moronic idiot.

Please interpret this charitably.

Anonymous said...

Anonymous 12:01: you accuse me of having missed the point of this thread, because I talk about instructors who refuse to listen to comments from their students. However, please look at the title of this thread. It is 'Teaching Evaluations: Ignore Them'. If you read the original post, you will find that its central advice to instructors is "stop reading your evaluations. Ignore them". The writer does not say to igore the numerical scores and to read the comments. And actually, the only reference to the content of the evaluations in that post is to the comments, not to numerical scores, seen by an instructor when she "made the mistake of reading her evaluations for the semester". Many, if not all, of the following posts also discuss comments, not numerical scores, and this seems to have been a general theme. So I cannot agree with you that I have missed the point of the thread, that brief and isolated comment by Neil notwithstanding.

Mostly Anonymous, you ask whether I'm "seriously denying that your intent was to insult and discredit opponents of student evaluations". My original post (May 15, 1:52am) did aim to raise doubts about the motivations, and the legitimacy for complaint, of those who make a special point about complaining ad nauseam about student evaluations, and who display along the way a scornful and arrogant attitude toward their students. Did I mean to be insulting, and to discredit these people? Of course. Did I mean the scope to involve _all_ opponents of student evaluations? No (and we've been over that about ten times now). But the big point is that I was not seeking to present an argument for a conclusion by discrediting those who present an argument for the opposite conclusion (i.e. I was not committing an ad hominem fallacy). I was making ad hominem comments, but not in the course of arguing for a conclusion. The ad hominem part was the end, not a means to an end, of my original post. Insulting someone, in itself, is not sufficient for committing the ad hominem fallacy.

As for the plumber analogy: it still seems, as I explained earlier, that there are factors in the philosophy instruction case that allow instructors to dodge an acknowledgement of their own failure as teachers in ways that plumbers cannot. For one thing, as I said before, there are many instructors who teach a student in a given term, and the effects of being taught well sometimes are not immediately evident; so it is presumably always difficult to tell whether to take credit or blame for the degree to which one's students have (or have not) improved over taking a course with you. For another, leaky pipes will never help to fix themselves, while even students who are taught badly by all their instructors will often learn _something_ just by reading the texts and writing essays. Third, there are several extraneous factors in students' lives that will influence their progress (or lack thereof) beyond the quality of the teaching they receive. All these things together make it very murky for an instructor to be sure of how successful his/her teaching is, even with a sincere effort. Factor in the obvious bias on the part of the instructor toward believing he/she is doing a fine job, and the prospects for a sincere and accurate self-evaluation start to look pretty bleak.

A fourth difference is that plumbers who refuses to take seriously the possibility that they are not very good at plumbing, and who blame all failures on the pipes, the climate, etc., will sooner or later get feedback of a different kind: their customers won't have them back.

Glaucon said...

Anon, your remarks in the posts subsequent to your original blast are a bit hard to take.

Initially (5/15 1:52am) you write:

[H]ow many of the whiners about student evaluations have anything better to suggest? How many of them propose even a single thoughtful (or even unthoughtful) alternative way to ensure that those who can't teach well either shape up or ship out? Of course they don't, because they're not really interested in finding better ways of getting the relevant information. They're just crappy teachers, or friends of crappy teachers, looking to safeguard their privilege at the public expense, or are touting their membership in a club whose hallmarks are a cynical scorn for their professional duties and an unhealthy desire to circle the wagons in the face of any threat to the status quo. In reponse to English Jerk (5/18 @ 8:30PM) you write:

I hoped that raising the issue I raised would help some of these people see that they were acting and speaking in a shameful manner, and that they were taking for granted (and it still seems to me that many of them are doing this) that they are somehow owed a living, whether or not they are doing a good job, and that they should exhibit such contempt for the opinions of those they are paid (by whomever) to educate that they should not even listen to any comments from them (see the originating post of the thread).I'm going to go out on a limb here and suggest that I'm not the only participant in this discussion who finds this disingenuous at best. Your initial, vitriolic post was meant to help others see the errors of their ways? I don't believe it for a second, and I don't believe that you believe it. Nor do I believe that all you've been saying all along is the innocuous point that some people who complain about teaching evaluations are are at least partly motivated by their not caring about their teaching. If that's all you meant, you chose a rhetorically odd way of making such a weak point.

I guess I'll have to take you at your word when you say that your student evaluations are cloaked in "warm praise" for your rigor. But frankly, given your demeanor here -- humorless, condescending, hostile, demanding charity of interpretation but not giving it, hyperbolic, judgmental -- I'm a bit skeptical. They're typically not traits I associate with good teachers / teaching.

For what it's worth, I think Spiros's advice in the particular circumstances in which it was offered seems sound to me. If I had a junior colleague who'd just be savaged as he describes, I'd suggest the same thing. Your vitriol is ironic, really, given that he offered the advice in order to prevent "the all too familiar self-fulfilling downward spiral from mere unpopularity to truly ineffective (because fixated on the evaluations rather than the subject-matter) teaching." I'm not sure where the base motivation is there. I doubt a charitable interpretation will yield one.

Anonymous said...

All right, enough of this. Minor point first: it really is true that I wanted to point to what I take to be the shockingly bad attitude of some people who had posted here. That was, in all honesty, my intention. While I didn't say or mean that everyone who is opposed to student reviews is a bad teacher or has a bad attitude, many of those people do. Spiros is, it seems obvious to me, one of them. Many of the other people who posted here, and many people whom I have seen whining about this issue in other contexts, are as well.

Are there interesting epistemic problems to be raised concerning whether I, or anyone else, can have adequate grounds under this, that or the other circumstance for determining whether someone has a bad attitude? Sure, just as there are in all cases of attitude attribution. Would I bet my house on my accuracy? No. How about the fact that I find it obvious that someone who can't even be bothered to _read_ his or her fucking evaluations is grossly arrogant and irresponsible, while others do not find it so? How about the fact that the confirmation I have for this view -- a basic ethical sense, and some personal experience -- cannot easily be presented argumentatively to others? For that matter, what about the fact that it may well be impossible for non-sociopaths to argue sociopaths into the merits of a truly moral life? These are all, I suppose, avenues for wonderful philosophical speculation. And added to the fun is everyone's insistence on going over and over and over and over the initial post I wrote, to cross-examine me to see whether you can trip me up. What fun! And of course, none of you have yet thought of doing this to Spiros, say. Why would it?

But here's the thing: _it doesn't matter_ whether I phrased things this way or that way. _It doesn't matter_ whether I'm an idiot, or humourles, or pompous, or whatever else you want to call me. What matters is that _the attitude of many people who have posted here is ethically very dubious_. That was the original point, which most of you seem to have lost.

Spiros' original post, asking about why we "put faculty through this exercise in intimidation and humiliation", bears on the surface of it no acknowledgement of why those faculty members are hired on: _to serve the interests of our students, and of society through the training of our students_. This is the point hardly anyone in the post seems to have understood. It's a point hardly anyone in these academic whine-fests seems to understand. If you think about it for a little, rather than trying to think up some clever way of showing that I misphrased something or other while ignoring all my other points, you would get it.

Presumably, by the time many of you have got to this point, you will have already thought up your brilliant response, and you will have a contest with one another to see who can come up with the greatest epistemological or scientific principle with which to nit-pick this or that phrase that I use. I hope you have fun with it: I'm through with this, since each time I try to clarify the obvious, the people who probably need to hear this message the most seem to distract themselves further and further from the point. It's really obvious, people: if you're not interested in listening to feedback from those you're paid amply to serve, there's something very wrong with you.

Anonymous said...

Shorter anonymous. Noticing that I what I say is malicious, ignorant or downright idiotic is evidence of a bad attitude.

Anonymous said...

Um... shorter anon... I think you just fell for it. Longer anon shouldn't have come off so belligerently, and he lost me at first because his attitude was too confrontational. But he does make some good points, and I've found myself being more convinced that there is something wrong here with the things we take for granted about teaching.

Parts of both sides of the debate are beneath what normally passes for civilized discussion in my part of the world. But the 'throw away your evaluations' side has sunk to lower depths.

It really is depressing watching intelligent people doing everything they can to avoid THINKING ABOUT -- not using intellectual tricks and bogus charges of fallacy to avoid thinking about, which is what I've seen here -- a simple but uncomfortable point. Thanks, longer anonymous (if you're still reading this) for posting. You've given me something to think about, but I've found it very depressing to see this kind of behaviour from everyone (including, to a smaller extent, your overly aggressive introduction). Everyone else, maybe we should take a break from this and think about it before we come back to a discussion. Maybe then we can figure out what parts of what's been said are good without distracting ourselves by getting at each other's throats all the time.

Anonymous said...

Anonymous, other people claimed that evaluations were epistemically useless. You chimed in with a vitriolic attack on them. It was YOU who introduced the vitriol into the debate. Now we are to accept that you did so in the name of upholding correct standards of respect. You are a piece of work.

Skeptic.

Anonymous said...

Is there any way we could actually start talking about how we can best hold ourselves, and be held, accountable for the quality of our teaching? That seems to me to be the most interesting topic for discussion here. To quote one of my favourite movie lines, "I don't give a shit who said what, who did what, or who did who." I just think that would be interesting to discuss. If nobody here is interested, maybe I'll take my business to another blog (sniff).

-Justin

Anonymous said...

Sorry, been reading this so long I had to comment: the contempt shown for students in the earlier posts _was_ a form of vitriol. Anonymous's sin was just turning that same contempt on professors, and for that he was crucified.

Anonymous said...

Last anonymous: I wonder if you're misunderstanding the nature of the claim. You might think that in saying that student evaluations don't track truth, we're saying 'stoopid students'. But we're not. The results in cognitive and social psychology should come as a blow to all our pretensions: we - all of us - are unexpectedly bad at lots of things, and for many of these things we have to look at the data to find out what they are. Gil Harman writes about the reaction I think you're having, IIRC. Its common, but its a mistake.

Skeptic

Mostly Anonymous said...

Anon @ 2:52,

I suppose we have to start with two questions. First, what general criteria are we going to agree on for describing good teachers (think virtue ethics for teaching)? I have in mind here the worries expressed in two or three posts above about the (non-commensurable?) values of instilling knowledge and fostering love for the subject. Second, for which collections of classes are we to apply these criteria? Should we set separate goals by department, by course title, or by some intermediate way? Once these questions are answered, then we can discuss how to go about measuring teacher success (or failure).

Anon @ 3:43

Maybe I'm just being thick here, but I don't really see much contempt for students in the posts on this thread. There have been a few (four?) posts that cited some humorous comments from evaluations. I took it that these were intended to be more humorous than snide, though I suppose one could read them that way. Is this the vitriol? Or maybe it was Glaucon's "stupils" quip?

There were also a number of posts discussing the merits of evaluation scores. I hadn't really noticed until I looked back at the thread again, but most of the early critical posts actually acknowledge that student *comments* in evaluations are often useful for self-correction (especially when there is widespread agreement). See for instance Cogburn's first two posts. Again, maybe the criticism was itself read as defamatory to students, but I think a rather milder point is being made, namely that most everyone is subject to cognitive biases and illusions that affect our abilities to make correct evaluative judgments. Evaluation of teaching is a domain where the biases are especially prevalent, so the evaluation scores, which are often used by administrators to make hiring/firing decisions, are not very reliable. I take it that the thread moved to this (almost immediately) because it is the most serious problem with student evaluations ... [whispered aside] and this isn't exactly a problem with evaluations per se, but with administrative use of evaluations.

Anyway, if I'm missing something, please point it out to me, but as far as I can see, the thread had a more or less reasonably toned discussion of the merits of evaluations (with good posts on both sides -- notably imho from Cogburn and English Jerk on the con side and JP, Almeida, and 729 on the pro side) up until two Anonymous posts (one on May 10 at 9:22 and the big bomb on May 15 at 1:52) and the subsequent silliness.

Anonymous said...

Mostly anonymous: you raise interesting questions about possibly incommensurable standards and values in teaching (fostering love for the subject vs. training students in a skill vs. imparting knowledge), the difficulty of finding general criteria, the question of what should be attributed to what class, etc. You say, "Once these questions are answered, then we can discuss how to go about measuring teacher success (or failure)."

That sounds interesting, but I'm curious here: if there's really such remarkable unclarity surrounding the concept 'good teacher of philosophy' (and perhaps there is, I don't know), then how can we be so sure that the student evaluations are so bad at tracking teaching goodness? Surely, for those studies (i.e. the ones that suggested that students are _not_ good at tracking teaching goodness, which seem to be just a subset) to have shown anything, they need to have shown that student approval is not correlated with teaching goodness. But this would involve their measuring teaching goodness in some way, wouldn't it? So it seems that either those studies are more dubious than I already suspected (since they didn't operate on a clear definition of teaching goodness), or else that we do have a good general concept of teaching goodness to work from.

As for Gil Harman, Skeptic, that guy makes all kinds of claims that I'm not sure follow from the evidence he presents. He thinks it follows from the fact that people are more likely to help someone in need if they don't feel hurried that there are no good or bad people (I'm exaggerating a bit, but not much). If you want to conclude, in your Skeptic-al way, that we have reason to doubt our ability to know all kinds of things we think we know, then well and good: I'm not going to try to beat the skeptic! But the thing with skepticism is that it usually turns out to be closer than originally thought to a universal acid. In this case, I think it might well lead to the conclusion, if allowed to run free, that there are just no good methods of evaluation for philosophy teachers. And I have to agree with longer anonymous that there is something quite morally dubious in saying that about your own profession and still expecting to be paid for it.

Mostly Anonymous said...

Anon @ 12:19,

I'm not sure whether others have a clearly defined notion of "good teacher." I don't think I do, though that doesn't mean I don't have a *vague* idea of what a good teacher is like. In any event, I'm actually more worried by the possibility that the various members of the community have importantly different ideas of what makes a teacher good or bad, so I'm proposing that we simply come to some agreement about what to measure before devising instruments to make measurements.

If you have a working definition of "good teacher," then by all means share it! My questions were meant seriously not rhetorically. I lean more toward knowledge and skill learning than I do toward inspiration. So, if it were me setting the criteria, then I would do the following.

Each department should construct a large database of general skills questions/problems designed to assess facility with the topics covered in that department (probably, this should be split into two or three tiers, so that students always see questions at or around their level of ability). Each course should build a similar (but smaller) database to cover what is supposed to be learned in that class. (So, you would see questions about Plato and Aristotle in a history of ancient philosophy course but not in a symbolic logic course.) Each term, for each class, students take a pre-class and post-class assessment, where each assessment is constructed by randomly selecting questions from the relevant databases. One might append to this something meant to get at inspiration ... maybe questions like: (1) will you take another course in this department, or (2) did you enjoy this course? These could be made more specific as needed.

I am not sure whether the disciplinary threshold should be set in terms of an absolute scale -- students improve by such and so or more on the relevant tests, or rather in terms of a comparison to other teachers. If the latter, then we have to decide where the cutoff should be. I would recommend disciplining the lower tenth percentile, but maybe that's too small a group. Poor performance in a single semester should receive a verbal warning or maybe rather a conversation with a department head about what might be improved. Two consecutive poor semesters should result in a written warning and some mandatory teaching rehab. Four consecutive poor semesters should result in termination.

Mostly Anonymous said...

The previous post attempts to address your first point, but you make a second point that needs to be addressed as well. You say:

"Surely, for those studies (i.e. the ones that suggested that students are _not_ good at tracking teaching goodness, which seem to be just a subset) to have shown anything, they need to have shown that student approval is not correlated with teaching goodness. But this would involve their measuring teaching goodness in some way, wouldn't it? So it seems that either those studies are more dubious than I already suspected (since they didn't operate on a clear definition of teaching goodness), or else that we do have a good general concept of teaching goodness to work from."

I hope you will agree that one doesn't have to have a perfectly clear idea of what an x is to be convinced that whatever an x is, it doesn't have property y. I don't know what simplicity is or how to define it, but I know that a Rube Goldberg apparatus isn't simple. I find the notion of God very unclear (or rather, vague), but I understand it well enough to know that my cat is not God and that no person I've met so far is God.

In the experimental studies of teaching quality, being physically attractive, fitting a gender-role stereotype, and artificially inflating grades were found to predict evaluation scores (and I suspect that these are causes, not just associated predictors). Supposing the studies are valid (which might not be true, but if not, then more studies are required), student evaluations track these features. My notion of "good teacher" might be vague, but I think I can still say that these qualities have nothing to do with being a good teacher. If you disagree with that, then tell me why. If not, then I don't see what your complaint is about.

Spiros said...

I hadn't been following this thread, and was surprised to see all the comments. I've not had the time to read through everything, but I do want to re-emphasize a few things:

1. I didn't take my original post to have asserted anything about the effectiveness of TE's in gauging quality of instruction. Maybe TE's are highly accurate. Maybe not. I do know of many cases in which consulting TE's has made for worse teaching. And I mean to cover both kinds of case: good TE's entrenching bad pedagogy (e.g., grade inflation); and bad TE's defeating good pedagogy. The question of the value of consulting one's TE's seems to me to be quite distinct from the question of whether they track anything.

2. In advising my colleague to ignore her TE's I was not recommending that she stop caring about her teaching, or hold all students in contempt, etc. (In fact, I told her to ask a few colleagues to observe her class.) In her case, it seems to me that consulting her TE's will most likely make her a much worse teacher. So it's for the sake of good teaching that one might decide to ignore TE's.

3. In asking whether anyone has learned anything of value from TE's, I meant to ask whether it's common for people to find in their TE's things about their teaching that they didn't already know (or couldn't have come to know had they reflected a bit).

4. As for the obviousness of my bad attitude and bad teaching (see Anon 5/19 11:43pm), I suppose I have no real retort. I haven't consulted my TE's in well over a decade. Sometimes I skim them for amusing nonsense and bad grammar. But that's about it.

As I understand it, my job is *not* primarily to teach anyway. At my institution, all real incentives -- promotion, pay raises, fellowships, distinction-- are firmly tied to research. It happens to be my view that good teaching is an Elster-style byproduct of good research, but I can't argue that here.

I suppose that the fact that my courses tend to be over-enrolled, my average grades are well below my dept's average, I get a high number of requests from UG's for Independent Study and Senior Thesis supervision, and have won teaching awards (even though I teach Logic), could be taken as evidence that my attitude and teaching are exceptional. But I don't take any of this as proof of much.

Anonymous said...

Anonymous, I work on the stuff I'm talking about. It's not a universal acid though it is powerful stuff. In any case, I'm not being inconsistent (though I could of course) be wrong in saying that TEs are epistemically worthless. I claim to *know* this (I am not a radical skeptic). I claim we can know all kinds of things, including (roughly) who is a better philosopher than whom (Lewis better than me: I know that). You don't have a right to accuse the posters here of hypocrisy, is all I'm saying: they are taking a position like mine. They could be wrong (again), but a little epistemic humility is called for: so could you. And whether they're wrong or not, they are not evincing a bad attitude.

Anonymous said...

Anonymous skeptic: I think you're confusing me with someone else.

-The other anonymous.

Anonymous said...

Mostly Anonymous: thanks for your 4:21 on May 20 post. Very interesting positive suggestions! I'm going to have to think about them

On another point, though, I have to disagree with you. You attribute the problems in the thread hitherto to May 10 at 9:22 and 'the big bomb on May 15 at 1:52'. I've just read over those posts again and think they both had good things to say -- but in both cases, they were responded to in a sarcastic, 'I refuse to take this seriously' fashion (by Glaucon in the May 10 case and by Neil in the May 15 case).

-Justin

Anonymous said...

Justin, are you of the opinion that people should take 'big bombs' seriously (where 'big bombs' = things that are not reasonable in tone; that's what mostly anon said)? Isn't sarcasm justified in these cases? You will have to work hard to convince me otherwise.

Skeptic

Anonymous said...

I don't think mere sarcasm is adequate in those cases, no. There were serious concerns presented in both cases, even though the tone was one of incredulous disapproval. Those concerns were backed up by reasons. I don't think it's OK to ignore those sorts of things (whether or not you want to call them 'bombs') just because the writers are upset by the way others are talking during the conversation.

Think of how we would react if we were part of a conversation that seemed to us to involve disparaging talk about some less-privileged group: we would be bound to have our dander up also. I don't think that should disqualify our comments or subject us to sarcasm.

-Justin

Mostly Anonymous said...

Justin,

Looking back, I may have been too hard on the May 10th post, and I agree with you on two points: that both posts raised some interesting issues and that some reactions to the posts were too extreme.

However, I still think the real vitriol began with the May 15th posting, which I still find offensive. And I don't think "incredulous disapproval" quite captures its tone (probably so for the May 10th post). The May 15th post was (and is) much more *abusive* than incredulous. Glaucon's follow-up to the May 10 post was a bit much for my taste, especially "question 4" ... but it was also quickly and good-naturedly settled in the two posts immediately following.

Anyway, not all of the responses were over-reactions. And not all of them were cynical or sarcastic. Some were quite cogent replies to content.

Also, I would still like it if you could point out where the contempt for students appears in the posts before May 15th. Both the May 10th and the May 15th posts accuse critics of student evaluations of disparaging students, but I just don't see it. (And hence, I don't think your analogy to under-privileged groups holds up as a defense for the May 15th post.)

Anonymous (not THAT one) said...

It's the thread that wouldn't leave!!

Mostly Anonymous said...

On another note, I was thinking about Almeida's claim that since there are some students who are good at evaluating teaching, a good statistical analysis ought to be able to filter out the noise. And I was also thinking about Neil's reply -- that the good ones are balanced by the bad. (I will here be concerned only with evaluation scores, not with comments.)

Whether Neil's reply works depends on what good and bad evaluators are like. It looks like there are two competing models for "good/bad evaluators."

Model 1: Good evaluators are unbiased for the property at issue. Bad evaluators are strongly biased "opposite" to the truth. So, if a teacher has real ability 5 (on a five point scale), bad evaluators will give scores clustering around 1. (I have no idea what happens when a teacher has real ability 3.)

Model 2: A good evaluator should be thought of as coming from a distribution of evaluations with a low mean squared error. A bad evaluator should be thought of as coming from a distribution with a high mean squared error.

In the second case, the students might not be *biased* away from the truth but might simply have high variance in their answers. If that is the case, then I think Almeida is correct: if some evaluators are clustered tightly around or nearby the truth and all the others are randomly distributed across the whole scale, then we should be able to pick out the true value. If Neil's model is correct, then his reply is correct as well. On Almeida's model, for instance, if you asked a large number of people a multiple-choice question about quantum mechanics, and you had enough few physicists in the group, you should be able to detect the correct answer.

I'm interested in this as a statistical problem. How many subjects do you need and what percentage need to be good evaluators in order to get Almeida's answer to work? Given solutions to that problem, how likely is it that the actual world instantiates the set-up?

(Just to be clear, I don't think being able to detect signals in noise helps answer the biggest challenge to student evaluations -- the existence of nuisance causes of evaluation scores.)

Spiros said...

Jethro, Hank, etc.:

Go away.


Maybe it's time to close the comments on this post.

Anonymous said...

Spiros, thanks for deleting those satirical comments on your post. How cowardly. Shows what I suspected all along about you.

Maybe it's time to delete you and your friends.

Anonymous said...

Come on, Spiros: that was at least an initially interesting analogy that you just deleted. Please post it again, and offer a response. If you don't, I will.

-Justin

Spiros said...

Anon 5:25: Yawn. As if I give a toss about your suspicions.

Justin: Agreed about the potentially interesting analogy. But you know how these things go-- before long the analogy will be lost and taken by some random reader to be an invitation to chime in with something truly offensive. I just don't have the time to police the blog for this kind of thing. Feel free to continue the discussion of the analogy on your own blog.

Anonymous said...

Frankly, this is a pathetic attitude for someone who considers himself an intellectual. After all this, you want to end off by ducking the most insightful argument that has appeared on this thread.

I'm done with reading any more of this. I have conversations with greater rigour and integrity at my local bar, and you should see my local bar.

-Jason N.

Anonymous said...

Uh... I'd like to post and discuss it on my blog, Spiros, but I can't, since you just deleted it.

Anonymous said...

Sorry, I'm lost. What did I miss? What got deleted?

Anonymous said...

Anonymous (who knows which one) wrote: Maybe it's time to delete you and your friends.

Congrats, Spiros. Your first death threat.

Skeptic

Anonymous said...

Get real, Skeptic.

Anonymous said...

By the way, what got deleted was a great series entitled 'Black People's Votes: Ignore Them'. I can't remember the rest, but it worked perfectly and very insightfully. Spiros, Neil and Glaucon were brilliantly satirized. I really think it should be posted again.

Anonymous said...

I didn't see the post, but I do notice a teensy disanalogy between voting - expressing a preference - and tracking an independent truth. I will however conclude that the post was anonymous's first and only moment of brilliance, now unfortunately permanently lost to scrutiny. Congrats, anonymous, you won the thread.

Skeptic

A better analogy? said...

RateMyProfessor/MyProfessorSucks.com: Ignore them!I spent an hour or so this morning talking to a young colleague who had just made the mistake of looking at what various student-feedback websites had to say about her. She was upset, antagonized, demoralized, and stricken with anxiety. The student criticisms ranged from silly and irrelevant (e.g., comments about the instructor's shoes) to simply mean. [Alas, she did not get a hotness tamale.] No doubt she'll carry all of this into her classes next semester, thereby enabling the all too familiar self-fulfilling downward spiral from mere unpopularity to truly ineffective (because fixated on the feedback rather than the subject-matter) teaching. Why do we put faculty through this exercise in intimidation and humiliation?

So when she asked me what she should do, I told her that the answer was simple: stop looking at those websites. Ignore them. Seriously: Has anyone has learned a damn thing of value from these student-feedback websites?

The Anonymouse said...

It's really obvious, people: if you're not interested in listening to feedback from those you're paid amply to serve, there's something very wrong with you.

Anonymous said...

Really obvious? Really? What if the original post is right that in some cases listening to the feedback will cause you to serve students less well?

Anonymous said...

There seems to be good empirical evidence that black people's votes do not, in any vaguely reliable manner, track those characterisistics in candidates that make good political leaders.

Black people's votes tend, the research shows, to track such extraneous factors as personal appearance, charisma, sex, race, religion, and what candidates do with their hands while speaking.

Oh: and if you want black people to vote for you, the research shows that the best you can do is not focus on your foreign policy, environmental policy, fiscal management, etc: the most important thing is to offer them goodies (like a tax refund, even one they might not be able to capitalize on) near election day.

The reality is that counting black people's votes actually seems to give us _worse_, not better, leaders than some other methods of selecting one.

I fail to see why saying this implies that I am a racist. Maybe white people's votes are also bad at tracking good characteristics in candidates. I don't know. Maybe we should have a different method of government altogether. But what we do know is that black people's votes are less than worthless. For those individuals -- including me -- who want to look after the interests of black people more effectively, it's pretty clear now that allowing black people's votes to count is not the way to go.

Chiming In.... said...

This supposed analogy between votes and evals is misplaced through and through and Spiro was right to nuke it the first time round. Evals are not votes. Classrooms are not polities in which students are citizens and profs are governments or representatives. Profs are not agents charged with serving the interests of the students. Colleges are not coercive institutions, make no claim to moral authority, and participation is not compulsory. Equality is not a moral requirement for legitimate pedagogy, but is for legitimate government.

The list goes on. This discussion was great until this red-herring, guaranteed to foul the light-to-heat ratio, was introduced.

Anonymous said...

Chiming in wrote: "Profs are not agents charged with serving the interests of the students."

Could you please offer support for this extraordinary claim?

Chiming In... said...

Anon 1:25:

Easily. Profs are charged with providing college-level instruction in a subject. It might in fact not at all serve a particular student's interest to be instructed at that level. This does not create an obligation for the prof to lower the level of instruction. Alternatively, it may be in a student's interest to get high grades in a particular course without having to master the course material. That fact does not give rise to an obligation on the part of the prof to inflate the grades. Again, it might be in a student's interest to go on vacation on the same day an exam is scheduled. This does not create an obligation for a prof to reschedule the exam. Duh.

Anonymous said...

*Sigh*

You're confusing several different things: 1) serving the interests of our students; 2) doing whatever our students wish us to do; 3) inflating grades for students; etc.

Our obligation as instructors is to serve the interests of our students, and to serve the interests of society, through our teaching. Inflating grades does not serve those interests; it just serves the short-term desires of certain students. That's why we have an obligation not to inflate them.

You are also confusing _reading_ and _considering_ our teaching evaluations with _slavishly following them_, which is not what anyone seems to have suggested.

Perhaps you disagree with all this and think that taking your courses does not serve the interests of your students, or of society more broadly through the training you offer to your students. In that case, what possible justification could you have for thinking that you are justified in making a living from teaching your students?

Easily. Profs are charged with providing college-level instruction in a subject. It might in fact not at all serve a particular student's interest to be instructed at that level. This does not create an obligation for the prof to lower the level of instruction. Alternatively, it may be in a student's interest to get high grades in a particular course without having to master the course material. That fact does not give rise to an obligation on the part of the prof to inflate the grades. Again, it might be in a student's interest to go on vacation on the same day an exam is scheduled. This does not create an obligation for a prof to reschedule the exam. Duh.

Anonymous said...

*Sigh*

You're confusing several different things: 1) serving the interests of our students; 2) doing whatever our students wish us to do; 3) inflating grades for students; etc.

Our obligation as instructors is to serve the interests of our students, and to serve the interests of society, through our teaching. Inflating grades does not serve those interests; it just serves the short-term desires of certain students. That's why we have an obligation not to inflate them.

You are also confusing _reading_ and _considering_ our teaching evaluations with _slavishly following them_, which is not what anyone seems to have suggested.

Perhaps you disagree with all this and think that taking your courses does not serve the interests of your students, or of society more broadly through the training you offer to your students. In that case, what possible justification could you have for thinking that you are justified in making a living from teaching your students?

Chiming In... said...

Anon:

*Sigh* Everything you just said reinforces the point in the post that prompted your comment: the eval/vote analogy is misplaced. If teaching involves serving our student's interest in some sense that does not necessarily track their *expressed* interests, then teaching is different from governing, and, more importantly, the act of refusing to read one's evals is not analogous to refusing to count the votes of certain citizens.

To be sure, the failure of the analogy does not entail that profs may rightly ignore evals. That's a different kind of question, one that I was hoping would reemerge on this blog.

Glaucon said...

Hey Chiming In:

Well, what do you make of A Better Analogy?'s analogy? (Who, truth be told, is me.) If the original post had been about RateMyProfessor.com, I doubt that anyone would have thought the advice inappropriate -- and the response of The Anonymouse (again, me), quoted from that Anonymous (5/19@11:43pm) would be off-target. (I know, I know: I don't know when to quit -- like the "momma's boys" in Tom Waits' "Time".)

Though there clearly are differences between university-administered student surveys and self-selecting sites like RateMyProfessor, insofar as student feedback is supposed to be formative -- i.e., help teachers improve their teaching -- the analogy seems pretty good, at least on the surface.

(I too often forget that distinction between formative and summative. I think perhaps the Heumer essay, cited a couple of weeks ago (holy crap, this has been going on for a while!), made the distinction. The question of the validity of student evals as summative -- i.e., as evaluations -- is separate from the question of whether they're formatively helpful. (I'm probably guilty of occasionally blurring this distinction above, to the detriment of the discussion.) It seems to me that skepticism about their summative validity is empirically quite well-founded. My belief that they do not track teaching quality may turn out to be false, but given the evidence I think it's justified.)

So here's an argument for ignoring student evals as formative:
It's not objectionable to ignore feedback from RateMyProfessor and the like.
Feedback from RateMyProfessor, etc. is relevantly analogous to feedback from university-solicited surveys.
Therefore, it's (probably) not objectionable to ignore feedback from university-administered surveys.

It may be that I find it safe to ignore the website feedback only because I have access to officially solicited feedback. But even if my university ceased to provide formative feedback from students, I don't think that would render the first premise false. For there are other ways to get feedback on whether students are learning (quizzes and short writing assignments in the short-term, exams and papers in the long-term (the term-term, so to speak) and on one's teaching (peer visits, video, etc.).

People are probably sick of this, and of me, as well. (I can be a tool at times -- and no doubt this post has been one of those times, at times.) But if not, what do you make of that argument? I'm not married to it, but it seems persuasive, at first blush -- but it's only as good as the analogy, obviously.

English Jerk said...

Glaucon:

Not only is rmp.com (which is a Viacom product, by the way) analogous to student evaluations in the relevant respects, but they produce identical results:

http://www.insidehighered.com/news/2008/04/25/rmp

The article suggests that this 'validates' rmp.com, but I'd say it proves what several of us have been saying about student evaluations here. So keep fighting the good fight!

Also, it seems to me that to defend student evaluations one would have to show that they provide information that wouldn't be available through either the coursework students produce (which tells us what they're learning) or through peer evaluation (which provides us with an informed evaluation of our practices). I don't think anyone's done so thus far (thought it's a long enough thread I might have forgotten), and I'd be curious to hear what those who favor student evaluations (minus that anonymous) have to say about it.

Anonymous said...

I did start to write a response to the analogy with Black voting. But I deleted it. There really is no point (Glaucon, give it up too!). Anyone stupid enough to take it seriously, let alone write it, is obviously not capable of understanding arguments against it. I'm going away: that's the only sensible thing to do.

Skeptic

Anonymous said...

Wow: I just read through this entire thread and have never been more depressed about the sorry state of my fellow academicians. The original post was arrogant and stupid, and the rest of the discussion was, in general, even more depressing. I hate to say it, but longer anonymous has a point: based on the evidence I'm seeing here, most of you can't see the forest for the trees.

I hope you are an unrepresentative sample.

Spiros said...

Just back, and I'm glad to see that the blog did not spin out of control in my absence. Glad to see also that the proposed analogy between ignoring TE's and not counting votes got dismantled. I think Chiming nailed it. And Glaucon's RMP.com analogy is helpful too.

To the many (?) who have left us on the basis of the disgust they claim this thread has caused: good riddance.

Anonymous said...

If only I could say good riddance to people like you who keep teaching as easily as you say good riddance to me. Unfortunately, you are in the position of privilege, despite your whining.

-Justin

Spiros said...

Justin:

You *can* say good riddance to me very easily: go away. No one's forcing--or even asking--you to be here. You don't like my whining? Too bad. You think I'm an irresponsible teacher? Your views on the issue don't matter. It bothers you that I'm undeservedly privileged in some respect relative to you? Life's tough.

But here's a tip: Find another blog to read.

Anonymous said...

No, I'm saying that I'd like to say good riddance to you by seeing you and all the rest of the wining leeches taken off the public payroll. I'm sick and tired of your ilk screwing over students (I was one once, too) and bringing down the profession because you can't distinguish caring about your students from becoming an easy marker.

I _am_ forced, unfortunately, to share my profession with undeserving rubes like you.

Spiros said...

I'll be sure to alert the relevant persons that I'm an undeserving rube who screws over my students next time I win an award or commendation for my teaching. I'll also mention it to the dozens of students who get closed out of my classes each semester because they fill up so quickly; and I'll bring it to the attention of those writing dissertations and senior theses under my direction. TE's must be every bit as worthless as I've claimed, since I apparently get excellent reviews, but, alas, am an uncaring rube. QED.

Finally, let me remind you that you don't have to share your profession with me-- you can leave. Have you ever considered doing something else with your life? That you apparently think that you're in a suitable position to draw inferences about my teaching and other abilities based on blog posts is only the first of several reasons why one might think you'd be better at something else.

Now how about going away already?

Anonymous said...

"I win an award or commendation for my teaching. I'll also mention it to the dozens of students who get closed out of my classes each semester because they fill up so quickly; and I'll bring it to the attention of those writing dissertations and senior theses under my direction. TE's must be every bit as worthless as I've claimed, since I apparently get excellent reviews, but, alas, am an uncaring rube. QED."

Right. And I just won the nobel prize for physics.

Spiros said...

Wow! Another excellent retort. You showed me. It's true: We really don't belong in the same profession.

Now, please go away. I'm sure you can find a place where your talents will be better appreciated.

Anonymous said...

Oho! Sarcasm!

Anonymous said...

The unfortunate thing about debating with little self-absorbed blowhards online is that, when they pull shit like you pull, Spiros, you don't get to do the natural thing and take them out to the parking lot.

Anonymous said...


boyfriend broke up with me two weeks ago, we were in love, we were happy and everything was perfect not until he started dating my best friend, I cry myself to sleep every night, I tried everything to get him back. It looks like its hopeless, He spends time with my best friend and not me. I miss him so incredibly much and everyday, he was a great guy and I needed him back and i never thought possible that Mr Robinson can restore broken relationship! At first I was scared cos I read a lots of stories of fake spell casters, scams and i never really believed in magic but I played along with a little faith, the love spell worked like a magic, 3 days after the love spell was done Robinson made my lover to come back to me with greater love, i was happy, I was floored that his spells worked, everything felt dreamy and unbelievable thanks to Mr Robinson for helping me to bring my lover back, please if you have the same issues call robinsonbuckler@yahoo.com for help @ his number + 19715126745