Teaching Seminar 1997
Dr. Scharff

Once again -- the disclaimer: this is not an absolute guideline. Much of it is my opinion and a rehash of what I was told when I took a teaching course.

Tests

You hated them as a student... you'll hate them as a teacher....

Instructional Objectives:

1. Knowledge
2. Comprehension
3. Application
4. Analysis (partition of material to components)
5. Synthesis ( combination of components)
6. Evaluation (judging material)

The Purposes of Exams

  • To see if you are getting our point across; you send; you need to know if they receive; usually tests cover the first three objectives (knowledge, comprehension, and application), the other three are often harder to test objectively.
  • Ideally, it forces students to review and synthesize the information.
  • Motivation - tests motivate students. One of their major motivations is to get good grades. This desire is extrinsic motivation rather than intrinsic motivation. (Remember: in intrinsic motivation the activity itself is motivating, whereas in extrinsic motivation something else is the incentive rather than the activity itself.) This extrinsic aspect of grades motivating studying is a problem. Often students don't care to learn the material unless it is on the test.
  • Evaluation - if you don't give tests, you still must evaluate the students in some way (e.g. papers, homework, other projects).

    • Differentiating Students with Tests

    If you are dealing with large numbers of students, your tests should divide them into grading categories. If all the students are making A's and B's, your tests aren't doing their job properly. Even at a highly selective school, differentiating should be possible. To accomplish this goal you should include items of varying difficulty. Analogy questions, for example, should divide the A's from the B's and the B's from the C's.

    Type of Test

    You have several choices: essay, short answer, multiple choice, matching, true/false.

  • Essays are probably the best way to test for analysis, synthesis, and evaluation of information. Essay tests are recall tests. (Hardest type of memory test.) They are hard to score objectively. They take longer to score, so are especially ineffective for large classes.
  • Short answer exams may be a bit less subjective, but have similar drawbacks as essays.
  • Multiple choice exams allow you to cover more material in a short time period. Multiple choice exams are recognition tests, so the scores will tend to be higher than if a pure recall format was used. Multiple-choice exams have the advantage of being objective and quick to score.
  • Matching items involve recognition and are objective and easy to score. The "process of elimination" technique can be avoided by giving more possible answers than question items.
  • True / False questions are the least desirable form of the multiple choice question. The problems associated with them out weigh their benefits. First, knowledge is not usually clear cut, and dichotomous questions do not always capture reality. Second, they are often too easy to answer, and hence may not differentiate well between students. To avoid this problem, some teachers write true-false questions which are ambiguous, but this reduces their validity because they no longer measure only knowledge. Third, true-false questions test mostly rote memorization.

    • Representativeness of Material on Tests

    Test material comes from the lecture and the book. If most of the material in lecture is unrelated to the book, and the test covers mostly book material, then the test is unfair. If, however, you tell students that only book material will be covered on the exams, then you may have an empty classroom. You should talk about the book in lecture and include it on the test with some consideration of the amount of emphasis laced on the readings.

    Another problem is that some material is more easily tested than other material. This is true of multiple choice and essays. We must find ways to test both easy and hard material.

    Finally, footnotes aren't good or fair testing sources.

    Writing Multiple Choice Questions

    Because this will probably be the most often used format (given class sizes and concerns for objectivity), we will spend some time discussing how to create a good multiple choice question. This is not a trivial thing to do.

  • Vary the difficulty of items to differentiate between students.
  • How many choices should be used for each question? Three choices appear to be too easy -- can get 33% right just by guessing. It's hard to come up with five good alternatives, so four choices is usually a good compromise.
  • Types of questions include factual (simple memory test), conceptual and application. Try to include some of each to differentiate students.

    • There are various categories that possible answers fall into:

    1. memory items; simply measures a student's ability to regurgitate
    2. applied knowledge; measures ability to applying learning to a situation not covered in class.
    3. negative items; a single item in a list renders the option incorrect
    4. sequences; answers are organized into a time or flow sequence
    5. theoretical; measures ability to integrate and apply learning
    6. use of neither / both connectives; should be done sparingly
    7. use of nonsense answers or answers that do not apply for that question
    8. comparison answers
    9. choose the best answer; requires more subtle knowledge

    The following are some more specific pointers to keep in mind when writing:

  • Use basic English -- isn't a vocabulary test and you don't want somebody to do bad on the test just because s/he does not understand the question.
  • Avoid technical jargon and abbreviations.
  • Write clearly and concisely (avoid long stems -- it is not a reading test).
  • Avoid negatively worded questions! They are harder to read, remember and answer. One way to get around this issue is to ask students to indicate the one choice that does not belong with the others (exclusion task).
  • Avoid "all of the above", "none of the above", and "d) b+c" style questions. If you do decide to use any of these, limit them to just a couple per test.
  • It's nice to come up with comparable choices (similar in length etc.).
  • Use words that start with the same syllable or rhythm as the correct answer
  • Make up technical terms that do not exist for alternate answers.
  • Put down a correct answer for another question.
  • Save your presents for Christmas -- don't give away the correct answer by wording, obviously incorrect choices or putting information in other questions.
  • Try to avoid parentheses in the stem
  • Inserts (or words left out / fill-in-the-blank) are tricky -- try to write the question to that the missing word(s) is at the end of the sentence.
  • Words like ÒaccurateÓ are tricky to use because they are vague. What do you mean by accurate (most effective, most true, when)?
  • Try not to make your stems to general. When you use general stems and specific answers, you are testing for specific knowledge anyway, so why not make the question more specific (and less ambiguous).
  • Why test students on names? Wouldn't it be better to know what the people did than to simply know the name? You may be able to make a case for testing "famous" names, but in general there are many other things that you can test students on about people besides what their names are.
  • Analogy questions are good. They will probably help separate the A's from the B's.
  • It is good to use an example and have them pick a concept (or can give them a concept and have them pick an example).

    • Other Points

  • Scheduled v. Surprise Tests: The use of surprise tests forces the students to study throughout the entire semester (if you're lucky enough to get any students to register for the class). Students hate surprise tests and their emotional reactions may affect their ability or motivation to study in class.
  • Test Length: This aspect of a test is often hard to judge a head of time. It will obviously take you less time to complete it since you are very familiar with the topic. The problems with giving exams that are too long are:1. They devolve into a test of reading / comprehension skill, rather than being a test of subject knowledge and integration.
    2. Onset of fatigue due to maintenance of concentration (question of mental hardiness)
    3. You have to make up all the questions -- and then grade them all.
  • Non-standard test items: What about having students choose 3 out of 5 questions to answer? Problems with this technique: How do you know if all the questions are equally hard? How do you compare tests? And, there are alot of individual differences. So, although this approach may initially seem appealing, it is often a bad idea because of fairness issues.

    • Take Home Exams

    One way to avoid using up alot of class time with exams is to give take home exams. Take home exams may also allow you to assign questions which would require deeper synthesis and more complete, detailed answers. Also, it gets rid of the time pressure problem (to some extent). However, the big potential difficulty with take home exams is cheating (i.e. collusion). One way to avoid collusion is to forewarn students that if you see alot of similarity between two exams, then those students will be questioned personally.

    How long for a take home exam? This is a hot issue. If you give a long time (e.g. 5-7 days) then students have time to put it off a little if they have another test, etc. They may just put it off. If there is a short time limit (e.g. 48 hours) then students can put off the other work until they finish the test.

    Be sure that the students actually read the test questions when they are handed out. Make them type their responses!

    Grading Essay and Short Answer Exams

    Grade all of one question before going on to the next question. This will help ensure fairness across tests. You'll be better able to remember how many partial credit points you gave for what if you do all of one question at a time.

    What about students with better writing skills? Should they get extra points for this? It shouldn't matter but it does. Making allowances for poor writing skill is like making allowances for poor IQ. One of the goals of college is that students learn to express themselves well. It's really two issues:

    1. you're really testing for knowledge and one particular skill is interfering with the expression of that knowledge, but
    2. that skill is part of what they're supposed to learn.

    A similar problem occurs with oral exams. How much should one allow for individual differences in speaking style? Again, speaking style is something that you should learn.

    So, when grading, set up an explicit criterion for grading. State it clearly on the test, or make it implicit in the question. This way students will know exactly what you want, and tests will be easier to grade.

    Oral Exams

    There are pros and cons of oral exams.

    Bad points: They are time consuming. There is generally high test anxiety in the student. You must isolate the student or give different questions (which may not really be equally difficult). Grading is pretty subjective, and thus may lead to fairness problems.

    Good points: It's more like real life. You can find out how the student "thinks on his/her feet." You can go into more depth. you can lead a student -- help them think through the problem and find the solution.

    In a committee, the student can be unfairly skewered between persons with opposing positions. Always try to avoid or cut off this sort of situation.

    Questions must be open ended. Always ask the student to clarify what he/she means -- he/she may not know what's being asked, so the question should be restated back to the questioner. If the student doesn't know the answer? Move on...

    Papers

    Papers give students the chance to be more active and do something besides simple regurgitation. (This may be difficult for some, because regurgitation is what is usually stressed throughout school, and they may have gotten good at it without developing other skills.)

    Promote enthusiasm and active learning by letting them choose their own topic (within the range of the course). This encourages the student to master an area of interest and exposes him/her to original literature.

    Tell the students precisely what you want. When it is the first paper, the students may need alot of help. Let them know exactly how important each part is: literature review, organization, evaluation, critique. Do they need to present original thoughts? There are different kinds of students, and some will be good at one part while others will be good at other parts.

    Grading papers: This is difficult, and time consuming if you do a good job. Especially if more than one paper is required in a course, give alot of feedback so that students will have an idea about how to improve their future papers. A letter grade alone at the end of a paper is easier for you, but doesn't help the learning process much. One particular problem is the great paper from the poor student -- what happened? There is no easy answer here, and plagiarism is difficult to prove.