Question? Leave a message!




A Short Guide to Writing Effective Test Questions

A Short Guide to Writing Effective Test Questions 3
Is This a Is This a Is This a Is This a Is This a Trick Trick Trick Trick Trick Question? Question? Question? Question? Question? A Short Guide to Writing Effective Test QuestionsIs This a Is This a Is This a Is This a Is This a Trick Trick Trick Trick Trick Question? Question? Question? Question? Question? A Short Guide to Writing Effective Test Questions Designed & Developed by: Ben Clay Kansas Curriculum Center Formatting & Text Processing by: Esperanza Root This publication was developed by the Kansas Curriculum Center with funds provided by the Kansas State Department of Education. First printing: October, 2001Table of Contents Table of Contents Table of Contents Table of Contents Table of Contents Preface ............................................................................................ i-ii Pre-Test ........................................................................................... 1-2 Generally ........................................................................................ 3-5 General Tips About Testing ............................................... 3-4 When to Use Essay or Objective Tests ............................... 4-5 Matching Learning Objectives with Test Items ...................... 5 Planning the Test .......................................................................... 6-12 Cognitive Complexity ........................................................ 6-7 Content Quality .................................................................... 8 Meaningfulness .................................................................... 8 Language Appropriateness .................................................... 9 Transfer and Generalizability ................................................ 9 Fairness .............................................................................. 10 Reliability ........................................................................... 10 How to Defeat Student Guessing ........................................ 11 General Test Taking Tips .................................................... 12 Multiple Choice Test Items ......................................................... 13-19 Section Summary................................................................ 13 Test Your Knowledge.......................................................... 14 Suggestions for Writing Multiple Choice Test Items ........ 15-16 Multiple Choice Test Taking Tips ...................................17-18 Aim for Higher Levels of Learning....................................... 19 True-False Test Items................................................................... 20-26 Section Summary................................................................ 20 Test Your Knowledge.......................................................... 21 Suggestions for Writing True-False Test Items ................. 22-23 Extreme Modifiers and Qualifiers ........................................ 23 True-False Test Taking Tips ................................................. 24 Variations in Writing True-False Test Items ..................... 24-25 Aim for Higher Levels of Learning....................................... 26 Matching Test Items .................................................................... 27-33 Section Summary................................................................ 27 Test Your Knowledge.....................................................28-29 Suggestions for Writing Matching Test Items .................. 30-31 Matching Test Taking Tips .................................................. 32 Variations for Creating Matching Tests ................................ 33Completion or Fill-in-the-Blank Test Items .................................. 34-37 Section Summary................................................................ 34 Test Your Knowledge.......................................................... 35 Suggestions for Writing Completion Test Items .............. 36-37 Completion Test Taking Tips .............................................. 37 Essay Test Items .......................................................................... 38-44 Section Summary................................................................ 38 "I'd Like to Use Essay Tests, But…" ..................................... 39 Read'Em and Weep Essay Test Items................................... 39 Test Your Knowledge.......................................................... 40 Suggestions for Writing Essay Test Items ........................ 41-42 Four-Step Process in Grading Essay Tests ............................ 43 Essay Test Taking Tips......................................................... 44 Additional Types of Test Items .....................................................45-51 Problem Solving ................................................................. 45 Using Authentic Assessments ......................................... 46-47 Grading Authentic Assessments .......................................... 48 Rubric Development ..................................................... 48-51 Etc…Etc…Etc… ........................................................................... 52-60 Purpose of Testing .............................................................. 52 Tips on Test Construction ................................................... 52 Test Layout Tips.................................................................. 52 Returning Tests and Giving Feedback ................................. 53 Alternative Testing Modes .................................................. 54 Creating Fair Tests and Testing Fairly .................................. 55 "I'd Like to Use Essay Tests, But…" ................................ 56-57 Test Administration Assignment .......................................... 58 Cognitive Domain Guide.................................................... 59 Affective Domain Guide ..................................................... 60 Bibliography ............................................................................... 61-63Preface… Preface… Preface… Preface… Preface… A notable concern of many teachers is that they frequently have the task of constructing tests but have relatively little training or Research indicates… information to rely on in this task. Is This a Trick Question? is an Teachers tend to use tests that information sourcebook for writing effective test questions. The they have prepared themselves central focus of the sourcebook’s content is derived from standards much more often than any other developed by the National Center for Research on Evaluation, Stan- type of test. (How Teaching Mat- dards, and Student Testing (CRESST). ters, NCATE, Oct. 2000) CRESST’s criteria for establishing the technical quality of a test While assessment options are di- encompasses seven areas: cognitive complexity, content quality, verse, most classroom educators meaningfulness, language appropriateness, transfer and rely on text and curriculum-em- generalizability, fairness, and reliability. Each aspect is discussed in bedded questions and tests that the sourcebook in a straight-forward, jargon-free style. are overwhelmingly classified as paper-and-pencil (National Com- Part One contains information concerning general test construction mission on Teaching and and introduces the six levels of intellectual understanding: knowl- America’s Future, 1996). edge, comprehension, application, analysis, synthesis, and evalua- tion. These levels of understanding assist in categorizing test Formal training in paper-and-pen- questions, with knowledge as the lowest level. Since teachers tend cil test construction may occur at to construct questions in the knowledge category 80% to 90% of the the preservice level (52% of the time, throughout the sourcebook are examples of or suggestions for time) or as inservice preparation developing higher order thinking skills. This supports Kansas’ (21%). A significant number of current Quality Performance Accreditation initiative which has professional educators (48%) re- established content and performance standards that cannot be port no formal training in devel- measured by low-level tests. oping, administering, scoring, and interpreting tests (Education Part Two of the information sourcebook is devoted to actual test Week, “National Survey of Public question construction. Because of the diversity of assessment School Teachers, 2000”). options, the sourcebook focuses primarily on paper-and-pencil tests, the most common type of teacher-prepared assessment. Five Students report a higher level of test item types are discussed: multiple choice, true-false, matching, test anxiety over teacher-made completion, and essay. Information covers the appropriate use of tests (64%) than over standard- each item type, advantages and disadvantages of each item type, ized tests (30%). The top three and characteristics of well written items. Suggestions for addressing reasons why: poor test construc- higher order thinking skills for each item type are also presented. tion, irrelevant or obscure mate- rial coverage, and unclear direc- This sourcebook was developed to accomplish three outcomes: tions. (NCATE, “Summary Data ! Teachers will know and follow appropriate principles for devel- on Teacher Effectiveness, Teacher oping and using assessment methods in their teaching, avoiding Quality, and Teacher Qualifica- common pitfalls in student assessment. tions”, 2001.) (Continued on next page…) i! Teachers will be able to identify and accommodate the limitations of different informal and formal assessment methods. ! Teachers will gain an awareness that certain assessment ap- In Kansas… proaches can be incompatible with certain instructional goals. The Kansas Commission on Teaching and America’s Future These three outcomes directly support the standards developed by (KCTAF), chaired by Dr. Andy a joint commission established by the National Education Associa- Tompkins, Kansas Commissioner tion, the American Federation of Teachers, and the National Coun- of Education, proposes to “de- cil on Measurement in Education. The initial standards were velop higher-quality alternative identified in 1990 and revised in 1999. In May 2001, a new listing pathways to teaching” as well as was issued under the title “Standards for Teacher Competence in to “reinvent teacher preparation Educational Assessment of Students”. The first two standards and professional development.” directly reflect the outcomes of this sourcebook: As secondary and postsecondary ! Teachers should be skilled in choosing assessment methods institutions are exploring (out of appropriate for instructional discussion necessity mostly) alternatives to ! Teachers should be skilled in developing assessment methods traditional teacher recruitment, appropriate for instructional decisions. the need for training in assess- ment procedures and paper-and- While no one document can thoroughly address the needs and pencil test construction in par- concerns expressed in all of this information, this sourcebook can ticular, become more and more be a valuable resource for any teacher who is interested in measur- evident. ing outcomes of significance, tapping into higher-level thinking and problem solving skills, and constructing tests that effectively and fairly capture what a student knows. Ben Clay, Coordinator Kansas Curriculum Center iiPre-Test Pre-Test Pre-Test Pre-Test Pre-Test Two general categories of Test Item Quiz test items Circle the correct answer 1. Objective items which require students to select the T=True F=False ?=Unsure correct response from several alternatives or to supply a 1. Essay exams are easier to construct word or short phrase to answer than are objective exams. T F ? a question or complete a statement 2. Essay exams require more thorough student preparation and study time 2. Subjective or essay items than objective exams. T F ? which permit the student to organize and present an original answer. 3. Essay exams require writing skills where objective exams do not. T F ? Objective items include: ! multiple choice 4. Essay exams teach a person how ! true-false to write. T F ? ! matching ! completion 5. Essay exams are more subjective in nature than are objective exams. T F ? Subjective items include: ! short-answer essay 6. Objective exams encourage guess- ! extended-response essay ! problem solving ing more so than essay exams. T F ? ! performance test items 7. Essay exams limit the extent of content covered. T F ? Test your knowledge of 8. Essay and objective exams can be used to measure the same these two item content or ability. T F ? types by 9. Essay and objective exams are answering the both good ways to evaluate a student’s level of knowledge. T F ? following Answers on next page… questions 1Quiz Answers Quiz Answers Quiz Answers Quiz Answers Quiz Answers 1. Essay exams are easier to construct than are objective exams. TRUE Essay items are generally easier and less time consuming to construct than are most objective test items. Technically correct and content appropriate mul- tiple choice and true-false test items require an extensive amount of time to write and revise. 2. Essay exams require more thorough student preparation and study time than objective exams. ? (QUESTION MARK) According to research findings it is still undetermined whether or not essay tests require or facilitate more thorough (or even different) student study preparation. 3. Essay exams require writing skills where objective exams do not. TRUE Writing skills do affect a student’s ability to communicate the correct “fac- tual” information through an essay response. Consequently, students with good writing skills have an advantage over students who do not. 4. Essay exams teach a person how to write. FALSE Essays do not teach a student how to write but they can emphasize the importance of being able to communicate through writing. Constant use of essay tests may encourage the knowledgeable but poor writing student to improve his/ her writing ability in order to improve performance. 5. Essay exams are more subjective in nature than are objective exams. TRUE Essays are more subjective in nature due to their susceptibility to scoring influences. Different readers can rate identical responses differently, the same reader can rate the same paper differently over time, the handwriting, neatness or punctuation can unintentionally affect a paper’s grade. 6. Objective exams encourage guessing more so than essay exams. ? (QUESTION MARK) Both item types encourage some guessing. Multiple choice, true-false and matching items can be correctly answered through blind guessing, yet essay items can be responded to satisfactorily through well written bluffing. 7. Essay exams limit the extent of content covered. TRUE Due to the extent of time required to respond to an essay question, only a few essay questions can be included on a exam. A larger number of objective items can be tested in the same amount of time, covering more content. 8. Essay and objective exams can be used to measure the same content or ability. TRUE Both item types can measure similar content or learning objectives. Re- search has shown that students respond almost identically to essay and objective test items covering the same content. 9. Essay and objective exams are both good ways to evaluate a student’s level of knowledge. TRUE Both objective and essay test items are good devices for measuring student achievement. However, as seen in the previous quiz answers, there are particular measurement situations where one item type is more appropriate than the other. 2Generally… Generally… Generally… Generally… Generally… General Tips About Testing Creating a test is Length of Test one of the most In theory, the more items a test has, the more reliable it is. On a short test a few wrong answers can have a great effect on the over- challenging tasks all results. On a long test, a few wrong answers will not influence confronting an the results as much. A long test does have drawbacks. If a test is too long, and particularly if students are doing the same kind of instructor. item over and over, they may get tired and not respond accurately or seriously. If a test needs to be lengthy, divide it into sections Unfortunately, with different kinds of tasks, to maintain the student's interest. many of us have Clear, Concise Instructions had little, if any, It is necessary to give clear, concise instructions. It is useful to provide an example of a worked problem, which helps the stu- preparation in dents understand exactly what is necessary. What seems to be writing tests. clear to the writer may be unclear to someone else. Mix It Up! Well constructed tests motivate It is often advantageous to mix types of items (multiple choice, students and reinforce learning. true-false, essay) on a written exam or to mix types of exams (a Well constructed tests enable performance component with a written component). Weaknesses teachers to assess the students connected with one kind of item or component or in students’ test mastery of course objectives. taking skills will be minimized. Tests also provide feedback on Test Early teaching, often showing what was It is helpful for instructors to test early in the term and consider or was not communicated clearly. discounting the first test if results are poor. Students often need a practice test to understand the format each instructor uses and an- ticipate the best way to prepare for and take particular tests. While always Test Frequently demanding, test Frequent testing helps students to avoid getting behind, provides instructors with multiple sources of information to use in comput- writing may be ing the final course grade (thus minimizing the effect of “bad days”), made easier by and gives students regular feedback. It is important to test various topics in proportion to the emphasis given in class. Students will considering the expect this practice and will study with this expectation. following Check For Accuracy suggestions for Instructors should be cautious about using tests written by others. Often, items developed by a previous instructor, a textbook pub- general test lisher, etc., can save a lot of time, but they should be checked for accuracy and appropriateness in the given course. construction. (Continued on next page…) 3General Tips About Testing (Continued from previous page) Proofread Exams On written exams, it is important to proofread exams carefully and, when possible, have another person proofread them. Tiny mis- takes, such as misnumbering the responses, can cause big prob- lems later. Collation should also be checked carefully, since miss- What makes a test ing pages can cause a great deal of trouble. good or bad? The One Wrong Answer most basic and obvious Generally, on either a written or performance test, it is wise to avoid having separate items or tasks depend upon answers or skills answer to that required in previous items or tasks. A student’s initial mistake will question is that good be perpetuated over the course of succeeding items or tasks, pe- tests measure what nalizing the student repeatedly for one error. you want to measure, Special Considerations and bad tests do not. It is important to anticipate special considerations that learning dis- abled students or non-native speakers may need. The instructor needs to anticipate special needs in advance and decide whether or not students will be allowed the use of dictionaries, extra time, separate testing sites, or other special conditions. A Little Humor Instructors have found that using a little humor or placing less dif- ficult items or tasks at the beginning of an exam can help students with test anxiety to reduce their preliminary tension and thus pro- vide a more accurate demonstration of their progress. When to Use Essay or Objective Tests When to Use Essay or Objective Tests When to Use Essay or Objective Tests When to Use Essay or Objective Tests When to Use Essay or Objective Tests Essay tests are appropriate when: It is always tempting ! the group to be tested is small and the test is not to be reused. to emphasize the ! you wish to encourage and reward the development of student parts of the course skill in writing. ! you are more interested in exploring the student’s attitudes than in that are easiest to measuring his/her achievement. test, rather than the parts that are Objective tests are appropriate when: ! the group to be tested is large and the test may be reused. important to test. ! highly reliable scores must be obtained as efficiently as possible. ! impartiality of evaluation, fairness, and freedom from possible test scoring influences are essential. (Continued on next page…) 4Conventional wisdom When to Use Essay or Objective Tests When to Use Essay or Objective Tests When to Use Essay or Objective Tests When to Use Essay or Objective Tests When to Use Essay or Objective Tests accurately portrays short- (Continued from previous page) answer and essay Either essay or objective tests can be used to: examinations as the easiest ! measure almost any important educational achievement to write and the most a written test can measure. difficult to grade, ! test understanding and ability to apply principles. particularly if they are ! test ability to think critically. graded well. ! test ability to solve problems. The matching of Matching Learning Objectives learning objective with Test Items expectations with Instructions: Below are four test item categories labeled certain item types A, B, C, and D. Following these test item categories are sample learning objectives. On the line to the left of each provides a high learning objective, place the letter of the most appropriate degree of test test item category. validity: testing A = Objective Test Item (multiple choice, true-false, matching) what is supposed B = Performance Test Item to be tested C = Essay Test Item (extended response) D= Essay Test Item (short answer) ____1. Name the parts of the human skeleton Certain item types are better ____2. Appraise a composition on the basis of suited than others for measuring its organization particular learning objectives. ____3. Demonstrate safe laboratory skills For example, learning objectives requiring the student to demon- ____4. Cite four examples of satire that Twain strate or to show, may be better uses in Huckleberry Finn measured by performance test ____5. Design a logo for a web page items, whereas objectives requir- ing the student to explain or to ____6. Describe the impact of a bull market describe may be better measured ____7. Diagnose a physical ailment by essay test items. ____8. List important mental attributes necessary To further illustrate this principle, for an athlete several sample learning objec- ____9. Categorize great American fiction writers tives and appropriate test items are provided on the right. Match ____10. Analyze the major causes of learning the most suitable test item with disabilities each of the learning objectives. 5 Answers: 1-A, 2-C, 3-B, 4-D, 5-B, 6-C, 7-B, 8-D, 9-A, 10-CPlanning the Test… Planning the Test… Planning the Test… Planning the Test… Planning the Test… Criteria for Establishing By definition no Technical Quality of a Test* test can be truly 1. Cognitive Complexity objective: Standard: The test questions will focus on appropriate existing as an intellectual activity ranging from simple recall of facts to problem solving, critical thinking, and reasoning. object of fact, Cognitive complexity refers to the various levels of learning independent of that can be tested. A good test reflects the goals of the instruction. If the instructor is mainly concerned with students the mind. memorizing facts, the test should ask for simple recall of material. If the instructor is trying to develop analytic skills, a test that asks for recall is inappropriate and will cause students In general, to conclude that memorization is the instructor's true goal. test items should… Refreshing the old bloom… ! Assess achievement of instruc- During the 1948 convention of the American Psychological tional objectives Association, a group of educational psychologists decided it ! Measure important aspects of would be useful to classify different levels of understanding the subject (concepts and con- that students can achieve in a course. ceptual relations) In 1956, after extensive research on educational goals, the ! Accurately reflect the empha- group published its findings in a book edited by Dr. Ben- sis placed on important aspects jamin S. Bloom, a Harvard professor. Bloom’s Taxonomy of of instruction Educational Objectives lists six levels of intellectual ! Measure an appropriate level understanding: of student knowledge ! Knowledge ! Analysis ! Vary in levels of difficulty ! Comprehension ! Synthesis ! Application ! Evaluation These levels of understanding assist in categorizing test ques- Implying that one type of tions. Teachers tend to ask questions in the knowledge cat- question is automatically egory 80% to 90% of the time. These questions are not bad, objective and the other but using them all the time is. Try to utilize higher order level of questions. These questions require much more brain necessarily subjective is a power. (See the next page for a definition and sample ques- faulty assumption, since tion frames for each level of learning.) bias can occur with either *Adapted from material developed by the National Center for Research on Evaluation, type of test. Standards, and Student Testing (CRESST). 6See pages 59 & 60 for Cognitive and Affective Domain Guides. 1. Cognitive Complexity (continued) Knowledge Recognizing and recalling information, Sample Question Frames including dates, events, persons, places; Who invented the…? terms, definitions; facts, principles, What is meant by…? theories; methods and procedures Where is the…? Comprehension Understanding the meaning of informa- Sample Question Frames tion, including restating (in own words); Restate in your own words…? translating from one form to another; or Convert fractions into…? interpreting, explaining, and List three reasons for…? summarizing. Application Applying general rules, methods, or Sample Question Frames principles to a new situation, including How is...an example of... ? classifying something as a specific How is...related to... ? example of a general principle or using Why is...significant? a formula to solve a problem. Analysis Identifying the organization and patterns Sample Question Frames within a system by identifying its compo- What are the parts of... ? nent parts and the relationships among Classify ...according to... the components. Outline/diagram... Synthesis Discovering/creating new connections, Sample Question Frames generalizations, patterns, or perspectives; What would you infer from... ? combining ideas to form a new whole. What ideas can you add to... ? How would you create a... ? Evaluation Using evidence and reasoned argument Sample Question Frames to judge how well a proposal would Do you agree…? accomplish a particular purpose; How would you decide about... ? resolving controversies or differences What priority would you give... ? of opinion. 7Criteria for Establishing Technical Quality of a Test (continued) To Achieve Content Quality… 2. Content Quality The first activity in planning a test Standard: The test questions will permit students to demon- is to outline the actual course con- strate their knowledge of challenging and important subject tent that the test will cover. A con- matter. venient way of accomplishing this is to take a few minutes following Some important questions need to be answered concerning each class to list on an index card the content quality of the test. What are the test specifica- the important concepts covered in tions? What skills do they indicate will be tested. How many class and in assigned reading for questions and how many areas will be covered? How many that day. These cards can then be sections will there be? What formats will be used to test? used later as a source of test items. If an instructor has focused on the War of 1812 in the major- An even more conscientious ap- ity of the class sessions and activities, this emphasis should proach would be to construct the be reflected in the test. A test that covers a much broader test items themselves after each period will be regarded as unfair by the students, even if the class. The advantage of either of instructor has told them that they are responsible for material these approaches is that the result- that has not been discussed in class. Students go by instruc- ing test is likely to be a better rep- tors' implicit values more than their stated ones. resentation of course activity. 3. Meaningfulness Standard: The test questions will be worth students’ time and students will recognize and understand their value. To Achieve Meaningfulness… "In my opinion, students should not be forced to guess what It is very easy to will be on a test, or psych-out the teacher to decide what to study. Research shows that the less able students are heavily write items which penalized by a failure to realize what is required for a test. require only rote The more able students seem to sense what the teacher wants, but the students most in need of help are likely to flounder recall but are even more painfully if they must guess what to study. nonetheless difficult because they are "The obvious solution to this problem is to give students spe- cific study questions, then draw the test from the study ques- taken from obscure tions. Sometimes this is criticized as teaching the test, as if passages (footnotes, having study questions in itself encourages a superficial ap- for instance). proach. That may be true if there are very few study ques- tions. However, if a teacher offers questions for all of the most important ideas in an assignment, then teaching the test is teaching the course." Russell A. Dewey, PhD Georgia Southern University, Statesboro, GA 8Criteria for Establishing Technical Quality of a Test (continued) Preliminary findings by the National 4. Language Appropriateness Center for Research on Evaluation, Standards, and Student Testing (CRESST): Standard: The language demands will be clear and appro- priate to the assessment tasks and to students. Results of Applying Language Evaluation Criteria to Test questions should reflect the language that is used in the Standardized Content Test Items classroom. Test items should be stated in simple, clear lan- guage, free of nonfunctional material and extraneous clues. Math and science subsections: Test items should also be free of race, ethnic, and sex bias. 67% percent of items had gen- Beyond these two qualifications, students' language back- eral vocabulary evaluated as grounds impact their performance on tests. The vocabulary uncommon or used in an (uncommon usage; nonliteral usage) and the syntax of the atypical manner; 33% of items test (atypical parts of speech; complex structures) may create had syntactic structures eval- language barriers. uated as complex or atypical in their construction. Modifications of the test for students that are limited English proficient include: assessment in the native language; text Reading comprehension: Same changes in vocabulary; modification of linguistic complex- as above for vocabulary and ity; addition of visual supports; use of glossaries in native lan- syntax; 50% of items also had guage; use of glossaries in English; linguistic modification of discourse level demands. test directions; and additional example items/tasks. To reduce frustration for good stu- dents, avoid all of these and none 5. Transfer and Generalizability of these and both a & b answers. These items are acceptable from Standard: Successful performance on the test will allow valid a theoretical standpoint, but most generalizations about achievement to be made. prepared test-takers dislike them! As an example, the more subject Presentations, scenarios, projects and portfolios add dimen- matter a student knows, the easier sions to assessment that traditional testing cannot. Teachers it is to make arguments in favor can make valid generalizations about achievement more eas- of answers that the teacher might ily using authentic and performance assessments. These gen- regard as wrong. eralizations may involve instructional placement decisions, formative evaluation decisions and diagnostic decisions. Well True-false questions are the worst constructed tests—whether they are objective or performance of all in this regard. Often the oriented—allow teachers to understand what needs to be truth value of an isolated state- taught next. Teachers are also able to monitor a student’s ment is quite debatable! It all learning, while instruction is underway, and can change the depends on how it is interpreted, instruction program as needed. the definition of a key term, or the context. 9Criteria for Establishing Technical Quality of a Test (continued) Five hundred secondary and 6. Fairness postsecondary students were surveyed for suggestions Standard: Student performance will be measured in a way on how an instructor could that does not give advantage to factors irrelevant to school grade fairly and accurately. learning; scoring schemes will be similarly equitable. Here are the top 10 responses. Here are a few basic rules of fairness: test questions should reflect the objectives of the unit; expectations should be clearly ! Consider grading based only on known by the students; each test item should present a clearly mastery of material and not on formulated task; one item should not aide in answering personalities or perceived effort. another; ample time for test completion should be allowed; and assignment of points should be determined before the ! Do not over emphasize grades. test is administered. Emphasize learning over grades. Grading constructively requires the instructor to provide ! Keep students informed of their feedback (written and/or oral) that helps the students to progress throughout the term. appreciate what they achieved and did not achieve by taking ! Clearly state grading policies and the test. This feedback could include the following: procedures in the syllabus and encouraging comments on a test or paper that convey respect review them with the class for what the student attempted to accomplish; praise for what during orientation. the student did accomplish and suggestions for improving performance. ! Avoid modifying policies during the term. 7. Reliability ! Provide plenty of opportunities for assessment. This will avoid Standard: Answers to test questions will be consistently unnecessary pressure and allow trusted to represent what students know. for some mistakes. The whole point of testing is to encourage learning. A good ! Provide some choice in format test is designed with items that are not easily guessed without or topic when assigning work. proper studying. It is possible to construct all types of test questions which are not readily guessed and therefore require ! Keep accurate records of grades. a student to comprehend basic factual material. Record numerical grades, rather than letter grades, when Multiple choice questions are widely scorned as multiple possible. guess questions. The solution to this problem is to design ! Consider allowing rewrites multiple choice items so that students who know the subject on papers. or material adequately are more likely to choose the correct alternative and students with less adequate knowledge are ! If many do poorly on an exam, more likely to choose a wrong alternative. (On the next page schedule an exam for the follow- are suggestions on how to defeat the TEST-WISE strategies of ing week to retest the class. students who do not study.) 10How to Defeat the Common Rules of Thumb How to Defeat the Common Rules of Thumb How to Defeat the Common Rules of Thumb How to Defeat the Common Rules of Thumb How to Defeat the Common Rules of Thumb Which Students Use to Guess Correct Answers Which Students Use to Guess Correct Answers Which Students Use to Guess Correct Answers Which Students Use to Guess Correct Answers Which Students Use to Guess Correct Answers Rule of thumb: Pick the longest answer. Way to defeat this strategy: Make sure the longest answer is right about a fifth of the time (if there are five alternatives for each question). Rule of thumb: Pick the ‘b’ alternative. Way to defeat this strategy: Make sure each answer is used the same number of times, in random order. Rule of thumb: Never pick an answer which uses the word ‘always’ or ‘never’ in it. Way to defeat this strategy: Make sure such answers are correct about a fifth of the time. Rule of thumb: If there are two answers which express opposites, pick one or the other and ignore other alternatives. Way to defeat this strategy: Sometimes offer opposites when neither is correct. Rule of thumb: If in doubt, guess. Way to minimize the impact of this strategy: Use five alternatives instead of three or four. Rule of thumb: Pick the scientific-sounding answer. Way to defeat this strategy: Use scientific sounding jargon in wrong answers Rule of thumb: Do not pick an answer which is too simple or obvious. Way to defeat this strategy: Sometimes make the simple, obvious answer the correct one. Rule of thumb: Pick a word which you remember was related to the topic. Way to defeat this strategy: When drawing up distractors (wrong answers) use terminology from the same area of the text as the right answer, but in distracters use those words incorrectly so the wrong answers are definitely wrong. 11Criteria for Establishing Technical Quality of a Test (continued) 7. Reliability (continued) "Remind, remind, remind students to Studies have shown that the grade given to an essay test depend in part upon the neatness of the handwriting. That stop and ask for seems like a poor way to assign a grade. However, if stu- directions or dents are asked to do the test on a word processor, it is hard clarification if there is to ensure that the work is original. Studies have also shown that grades for essay tests are influenced by length. If a stu- something they don’t dent rambles on, there is greater likelihood of hitting a few understand. points that the teacher is looking for. But do we want to reward verbosity? Directions are the roadmap to their final Despite all this, essay and short answer tests have many vir- destination." tues. Students need practice formulating arguments, express- ing things clearly, and integrating ideas. Nobody would ar- gue that all testing should be multiple choice. However, for teachers in many situations, a good objective test is both Suggestion… fairer and more efficient than an essay or short answer test. Encourage students to design One way to ensure reliability is to share with your students… their own test. This will help them anticipate some of the questions or information to be General Test Taking Tips included on the instructor’s exam. 1. Tell students to survey the entire test before they begin. This will help them identify which section will be quick and/or easy and which will require more time and thought. Various kinds of objective 2. Encourage students to underline important words in the directions such as list, discuss, define, etc. and essay test items are presented in the following 3. Instruct students that when they take a test, they should do the easy questions first. sections of this document. 4. Help students schedule their time by estimating the total Each kind of test item is time available compared to the number of questions on the test. They need to recognize that some types of ques- briefly described in terms tions will take longer than others. of advantages and 5. Suggest that students put a checkmark next to any ques- limitations for use. tions which they left blank and will need to come back to for completion later. General suggestions are 6. Prompt students to hold onto their test until they have also presented for the looked it over thoroughly. They should make sure they construction of each test have completed each task and have reread the entire test item variation. to verify that they have given the answers they intended. 12Multiple Choice Test Items Multiple Choice Test Items Multiple Choice Test Items Multiple Choice Test Items Multiple Choice Test Items The multiple choice item consists of the stem, which identifies the question or problem and the response alternatives or choices. "…almost any well Usually, students are asked to select the one alternative that best completes a statement or answers a question. For example, defined cognitive Item Stem: Which of the following is a chemical change? objective can be tested Response Alternatives: a. Evaporation of alcohol fairly in a multiple b. Freezing of water choice format." c. Burning of oil " d. Melting of wax Multiple choice items are considered to be among the most versa- tile of all item types. They can be used to test factual recall as well Section Summary as levels of understanding and ability to apply learning. As an example, the multiple choice item below is testing not only infor- Good for: mation recall but also the ability to use judgment in analyzing and ! Application, synthesis, evaluating. analysis, and evaluation levels Multiple choice tests can be used to test the ability to: 1. recall memorized information Types: 2. apply theory to routine cases ! Question/Right answer 3. apply theory to novel situations ! Incomplete statement 4. use judgment in analyzing and evaluating ! Best answer A. 1 only Advantages: B. 1 and 2 only ! Very effective C. 1, 2 and 3 only ! Versatile at all levels D. 1, 2, 3 and 4 " ! Minimum of writing for student Multiple choice items can also provide an excellent basis for post- ! Guessing reduced test discussion, especially if the discussion addresses why the in- ! Can cover broad range correct responses were wrong as well as why the correct responses of content were right. Unfortunately, multiple choice items are difficult and time consuming to construct well. They may also appear too dis- Disadvantages: criminating (picky) to students, especially when the alternatives ! Difficult to construct good are well constructed and are open to misinterpretation by students test items who read more into questions than is there. ! Difficult to come up with plausible distractors/alterna- Test your knowledge of tive responses multiple choice tests by taking the multiple choice test on the next page… 13Circle the Most Correct Answer Circle the Most Correct Answer Circle the Most Correct Answer Circle the Most Correct Answer Circle the Most Correct Answer 1. Multiple choice items provide highly 5. The right answers in multiple choice reliable test scores because: questions tend to be: A. they do not place a high degree of A. longer and more descriptive dependence on the students reading B. the same length as the wrong answers ability C. at least a paragraph long B. they place a high degree of depen- D. short dence on a teacher's writing ability C. they are a subjective measurement of student achievement 6. When guessing on a multiple choice D. they allow a wide sampling of question with numbers in the answer: content and a reduced guessing factor A. always pick the most extreme B. pick the lowest number C. pick answers in the middle range 2. You should: D. always pick C A. always decide on an answer before reading the alternatives B. always review your marked exams 7. What is the process of elimination in a C. never change an answer multiple choice question? D. always do the multiple choice items A. skipping the entire question on an exam first B. eliminating all answers with extreme modifiers C. just guessing 3. The above multiple choice item is D. eliminating the wrong answers structurally undesirable because: A. a direct question is more desirable than a incomplete statement 7. What should you not do when taking B. there is no explicit problem or a multiple choice test: information in the stem A. pay attention to patterns C. the alternatives are not all plausible B. listen to last minute instructions D. all of the above C. read each question carefully E. A & B only D. read all choices F. B & C only G. A & C only H. none of the above 8. It is unlikely that a student who is un- skilled in untangling negative statements will: 4. The above multiple choice item is A. quickly understand multiple choice undesirable because: items not written in this way A. it relies on an answer required in a B. not quickly understand multiple choice previous item items written in this way B. the stem does not supply enough C. quickly understand multiple choice information items written in this way C. eight alternatives are too many and D. not quickly understand multiple choice too confusing to the student items not written in this way D. more alternatives just encourage guessing 14 Answers: 1-D, 2-B, 3-D, 4-C, 5-A, 6-C, 7-D, 8-CSuggestions For Writing Multiple Choice Test Items Suggestions For Writing Multiple Choice Test Items Suggestions For Writing Multiple Choice Test Items Suggestions For Writing Multiple Choice Test Items Suggestions For Writing Multiple Choice Test Items Remember… 1. When possible, state the stem as a direct question rather than as an incomplete statement. ! Use at least four alternatives for each item to lower the probabil- Undesirable: Desirable: Alloys are ordinarily How are alloys ordinarily ity of getting the item correct by guessing. produced by… produced? ! Use capital letters (A, B, C, D) as 2. Present a definite, explicit and singular question response signs rather than lower or problem in the stem. case letters (“a” gets confused with “d” and “c” with “a” if the Undesirable: Desirable: type or duplication is poor). Psychology… The science of mind and be- havior is called… ! Randomly distribute the correct response among the alternative 3. Eliminate excessive verbiage or irrelevant positions throughout the test, hav- information from the stem. ing approximately the same pro- portion of alternatives A, B, C, Undesirable: Desirable: and D as the correct response. While ironing her formal, Jane Which of the following ways burned her hand accidently of heat transfer explains why on the hot iron. This was due Jane’s hand was burned after ! Avoid irrelevant clues such as to a transfer of heat between... she touched a hot iron? grammatical structure, well known verbal associations or sim- plistic connections between stem and answer. 4. Include in the stem any word(s) that might other- wise be repeated in each alternative. Undesirable: Desirable: ! When possible, present alterna- In national elections in the In national elections in the tives in some logical order (e.g., United States the President is United States the President is chronological, most to least, officially: officially chosen by alphabetical). A. chosen by the people. A. the people. B. chosen by members of B. members of Congress. Congress. C. the House of Reps. ! Use the alternatives none of the C. chosen by the House of D. the Electoral college. " above and all of the above spar- Representatives. ingly. When used, such alterna- D. chosen by the Electoral tives should occasionally be used College." as the correct response. 155. Use negatively stated stems sparingly. When used, Remember… underline and/or capitalize the negative word. ! In testing for definitions, use the Undesirable: Desirable: term in the stem rather than as Which of the following is not Which of the following is NOT an option. cited as an accomplishment of cited as an accomplishment of the Kennedy administration? the Kennedy administration? ! List alternatives on separate lines (rather than including the options as part of the stem) so 6. Make all alternatives plausible and attractive to the that all options can be clearly less knowledgeable or skillful student. distinguished. Undesirable: Desirable: What process is most What process is most ! Keep all alternatives in a simi- nearly the opposite of nearly the opposite of lar format (i.e., all phrases, all photosynthesis? photosynthesis? sentences, etc.). A. Digestion A. Digestion B. Assimilation B. Relaxation C. Respiration " C. Respiration " ! Try to make alternatives for an D. Catabolism D. Exertion item approximately the same length. (Making the correct re- sponse consistently longer is a 7. Make the alternatives mutually exclusive. common error.) Undesirable: Desirable: The daily minimum required What is the daily minimum amount of milk that a 10 year required amount of milk a 10 ! Use misconceptions students old child should drink is year old child should drink? have indicated in class or errors A. 1-2 glasses. A. 1 glass. commonly made by students in B. 2-3 glasses. " B. 2 glasses. the class as the basis for incor- C. 3-4 glasses. " C. 3 glasses. " rect alternatives. D. at least 4 glasses. D. 4 glasses. ! Way to judge a good stem: stu- dents who know the content 8. Make alternatives approximately equal in length. should be able to answer before reading the alternatives. Undesirable: Desirable: The most general cause of low What is the most general cause individual incomes in the of low individual incomes in Multiple choice exams pro- United States is: the United States? vide easier conditions for A. lack of valuable productive A. A lack of valuable produc- cheating than essay tests since services to sell. " tive services to sell. " single letters or numbers are B. unwillingness to work. B. The population’s overall easier to see than extensive C. automation. unwillingness to work. text. Cheating can be mini- D. inflation. C. The nation’s increased mized by using alternative test reliance on automation. forms and controlling seating. D.An increasing national level of inflation. 16Attention Students: Multiple Choice Test Taking Tips Attention Students: Multiple Choice Test Taking Tips Attention Students: Multiple Choice Test Taking Tips Attention Students: Multiple Choice Test Taking Tips Attention Students: Multiple Choice Test Taking Tips 1. Read the directions carefully The directions usually indicate that some alternatives may be partly correct or correct statements in themselves, but not when joined to the stem. The directions may say: “choose the most correct answer” or “mark the one best answer.” Sometimes you may be asked to “mark all correct answers.” Pay attention to the words… 2. Do the multiple choice items first ! Note qualifying words: usually, If your exam has types of questions other than multiple choice, often, generally, may, and sel- just reading the stems and alternatives acts is a warm-up to the dom are qualifiers that could material. (The stem is the question and the alternatives are the indicate a true statement. choices). Also, the ideas embedded in these multiple choice questions will fuel your thinking for doing the other parts of the ! Words such as every, all, none, exam. Use the process of elimination procedure. Eliminate the always, and only are superla- obviously incorrect alternatives. tives that indicate the correct answer is an undisputed fact. In general, absolutes are rare. 3. Read all of the stem and every alternative ! If a negative word such as none, Read the stem with each alternative to take advantage of the not, never, or neither is in the correct sound or flow that the correct answer often produces. stem, assume that the correct Also, you can eliminate any alternatives that do not agree gram- alternative must be a fact or ab- matically with the stem. solute and that the other alter- natives could be true statements, Some students find it effective to read the stem and anticipate but not the correct answer. the correct alternative before actually looking at the alternatives. If you generally do better on essay exams, this strategy may help you a great deal. 4. Consider "all of the above" and "none of the above" Examine the “above” alternatives to see if all of them or none of them apply totally. If even one does not apply totally, do not consider “all of the above” or “none of the above” as the correct answer. Make sure that a statement applies to the question since it can be true, but not be relevant to the question at hand! (Continued on next page…) 175. Plan your time Often you are required to answer up to 70 multiple choice ques- tions in an hour or less. This means you may have less than a minute, on average, to spend on each question. Some ques- tions, of course, will take you only a few seconds, while others will require more time for thought. Plan to progress through the exam 6. Changing answers in three ways: Research has shown that changing answers on a multiple choice ! Read every question carefully or true-false exam is neither good nor bad: if you have a good reason for changing your answer, change it. The origin of the but quickly, answering only those of which you are 100% myth that people always change from “right” to “wrong” is that certain. Put a “?” on those that those (i.e. the wrong ones) are the only ones you will see when need more thought. you review your exam—you will not notice the ones you changed from “wrong” to “right.” ! Then, examine/study the ques- tions not yet answered. Answer those you are reasonably sure This will pay dividends on future exams… of without pondering too long on each. Erase the “?”. Study your marked and returned exam in order ! Finally, study the remaining un- to learn from your successes and mistakes. answered questions. If you can- not come to a decision by rea- soning or if you run out of time, guess. Erase the “?”. Note that After Your Exam some examinations penalize Has Been Returned “guessing” by subtracting points for incorrect answers. 1. Examine each question you did get correct. Remember how you knew that the information was important when If there is no penalty, you studied. How did you study? then a guess is better 2. Examine each question you did not get correct in order to than a blank. understand the distinction between the correct alternative and the incorrect alternatives. Ask yourself why the correct answer is correct and why the other alternatives are incorrect. 3. Determine the level of thought your instructor expects of you by reading through all of the questions. Are you expected to recognize, analyze, synthesize and/or apply the material that has been presented to you? Study accordingly for the next exam. 18Multiple Choice Test Items: Conclusion Multiple Choice Test Items: Conclusion Multiple Choice Test Items: Conclusion Multiple Choice Test Items: Conclusion Multiple Choice Test Items: Conclusion Ask yourself: Aim for Higher Levels of Learning Why are these Most teachers find it easier to construct multiple choice items to multiple choice test recall and comprehension and to use essay items to test higher- level learning objectives. But other possibilities exist. Multiple questions crummy? choice items that require students to do such things as classify state- ments as fact or opinion go beyond simple recall of facts. 1. How frequently do you take a sick day from work? A. never Here are two examples of multiple choice test B. once or twice a year items designed for higher order thinking skills. C. 3 to 5 times a year D. 6 to 12 times a year A common goal of the Salt March in India, the E. at least once a month Boxer Rebellion in China, and the Zulu resistance in southern Africa was to: 2. Identify the issue that you be- A. overthrow totalitarian leaders lieve is most critical to this B. force upper classes to carry out land reform country's future. programs A. the economy C. remove foreign powers B. education D. establish Communist parties to lead the C. integrity in government governments D. national defense E. some other issue In western Europe, which development caused the other three? A. decline of trade B. fall of Rome "Understand that there is C. breakdown of central government always one clearly best D. rise in the power of the Roman Catholic Church answer. My goal is not to trick students or require them to make difficult One way to write multiple choice questions that require more than judgments about two options recall is to develop questions that resemble miniature "cases" or that are nearly equally situations. Provide a small collection of data, such as a description correct. My goal is to design of a situation, a series of graphs, quotes, a paragraph, or any cluster questions that students who of the kinds of raw information that might be appropriate material understand will answer for the activities of your discipline. correctly and students who do not understand will Then develop a series of questions based on that material. These answer incorrectly." questions might require students to apply learned concepts to the case, to combine data, to make a prediction on the outcome of a John A. Johnson process, to analyze a relationship between pieces of the informa- Dept. of Psychology, Penn State University tion, or to synthesize pieces of information into a new concept. 19True-False Test Items True-False Test Items True-False Test Items True-False Test Items True-False Test Items There are many situations In the most basic format, true-false questions are those in which a which call for either-or statement is presented and the student indicates in some manner decisions, such as deciding whether the statement is true or false. In other words, there are whether a specific solution only two possible responses for each item, and the student chooses is right or wrong, whether between them. True-false questions are well suited for testing stu- dent recall or comprehension. Students can generally respond to to continue or to stop, many questions, covering a lot of content, in a fairly short amount whether to use a singular of time. or plural construction, and so on. For such situations, From the teacher’s perspective, true-false questions can be written the true-false item is an quickly. They are easy to score. Because they can be objectively scored, the scores are more reliable than for items that are at least ideal measuring device. partially dependent on the teacher’s judgment. Select or Supply? Section Summary True-false questions require the students to select a response (true Good for: or false) that shows recognition of correct or incorrect information ! Knowledge level content that is presented to them. These are included among the items ! Evaluating student under- that are called selection, in contrast to supply items in which the standing of popular miscon- student must supply the correct information. ceptions ! Concepts with two logical Forced Choice responses Another term applied to true-false items is forced choice because the student must choose between two possible answers. Educa- Advantages: tional objectives that specify the student will identify, select, and ! Can test large amounts of recognize material are appropriately targeted to either forced choice content questions or more complex matching or multiple choice questions. ! Students can answer 3-4 questions per minute Much Maligned and Abused… Many educators feel that true-false test items serve little or no mea- Disadvantages: surement purposes because true-false items are subject to guess- ! They are easy ing. (But the likelihood of obtaining a substantially higher than ! It is difficult to discriminate chance score by guessing alone is very small). In general, indi- between students that know vidual true-false items are less discriminating than individual mul- the material and students tiple choice items. There is a tendency to write trivial true-false who do not items, which lead students to verbatim memorization. At the same ! Students have a 50-50 time, no diagnostic information is available from incorrect responses chance of getting the right to true-false items. Finally, true-false items are not amenable to answer by guessing concepts that cannot be formulated as propositions. ! Need a large number of items for high reliability 20Summarizing the Argument for the Value of True-False Test Items Since true-false questions tend to be either extremely easy or ! The essence of educational achievement is the command of extremely difficult, they do not useful verbal knowledge. discriminate between students of ! All verbal knowledge can be expressed in propositions. varying ability as well as other ! A proposition is any sentence that can be said to be true or false. types of questions do. ! The extent of students’ command of a particular area of knowl- edge is indicated by their success in judging the truth or falsity of propositions related to it. Ebel and Frisbie (1991) Making the Case for Check Your Knowledge of True-False Items True-False Test Items Directions: For each question below, circle A or B. ! Versatility—True-false items are adaptable to the measure- 1. Is it recommended to take statements directly from the ment of a wide variety of text to make good true-false questions? learning outcomes. A. Yes B. No ! Scoring accuracy and 2. Two ideas can be included in a true-false statement if economy—Scoring keys can the purpose is to show cause and effect. be economically applied by A. Yes machine or clerical assistants. B. No 3. When a true-false statement is an opinion, it should be ! Reliability—True-false tests attributed to someone in the statement. that are highly reliable can be A. Yes constructed. B. No 4. Underlining or circling answers is preferable to having ! Amenable to item analysis— the student write them. True-false items are amenable A. Yes to item analysis, by means of B. No which they can be improved. Circle "Good" if it describes a good practice in true-false questions, circle “Poor” if it characterizes a poor practice. ! Efficiency—More test re- sponses can be obtained from 5. Complex statements are used to measure higher order a given amount of written thinking. Good Poor material and in a given 6. If negatives, such as “not,” are used, they should be highlighted in some way. Good Poor amount of time than from 7. True and false statements should be approximately the other forms. same length. Good Poor 8. There should be a recognizable pattern in the answers, ! True-false items are useful in such as TFTFTFTF. Good Poor testing misconceptions. 9. The following are examples of words that should be avoided: “all,” “none,” “never,” “sometimes,” “gener- ! True-false items can be ex- ally,” and “often.” Good Poor pressed in few words, making them less dependent on reading ability. 21 Answers: 1-B, 2-A, 3-A, 4-A, 5-Poor, 6-Good, 7-Good, 8-Poor, 9-GoodSuggestions For Writing True-False Test Items Suggestions For Writing True-False Test Items Suggestions For Writing True-False Test Items Suggestions For Writing True-False Test Items Suggestions For Writing True-False Test Items Remember… 1. Base true-false items upon statements that are absolutely true or false, without qualifications or ! Keep language as simple and exceptions. clear as possible. Undesirable: Desirable: ! Use a relatively large number of Nearsightedness is Geneticists and eye special- items (75 or more when the en- hereditary in origin. ists believe that the predispo- tire test is T/F). sition to nearsightedness is hereditary. ! Be aware that extremely long or complicated statements will test 2. Express the item statement as simply and as reading skill rather than content clearly as possible. knowledge. Undesirable: Desirable: ! Require students to circle or un- When you see a highway with The construction and mainte- derline a typed “T” or “F” rather a marker that reads, “Interstate nance of interstate highways than to fill in a “T” or “F” next to 80” you know that the con- are provided by both state and the statement, thus avoiding hav- struction and upkeep of that federal governments. ing to interpret confusing hand- road is maintained by the state writing. and federal government. ! If a proposition expresses a rela- tionship, such as cause and effect 3. Express a single idea in each test item. or premise and conclusion, present the correct part of the Undesirable: Desirable: statement first and vary the truth Water will boil at a higher Water will boil at a higher tem- or falsity of the second part. temperature if the atmo- perature if the atmospheric spheric pressure on its surface pressure on its surface is is increased and more heat is increased. ! Make true and false items of ap- proximately equal average length applied to the container. throughout the test. 4. Include enough background information and quali- ! Randomize the sequence of true fications so that the ability to respond correctly to the and false statements. item does not depend on some special, uncommon ! Make use of popular misconcep- knowledge. tions/beliefs as false statements. Undesirable: Desirable: The second principle of edu- According to John Dewey, the ! Write items so that the incorrect cation is that the individual second principle of education response is more plausible or at- gathers knowledge. is that the individual gathers tractive to those without the spe- knowledge. cialized knowledge being tested. 22Extreme Modifiers: 5. Avoid the use of extreme modifiers or qualifiers. all none Undesirable: Desirable: always never —All sessions of Congress are —The sum of the angles of a only nobody o called by the President. (F) triangle is always 180 . (T) invariably no one —The Supreme Court —The galvanometer is the in- best absolutely frequently rules on the con- strument usually used for the worst absolutely not stitutionality of law. (T) metering of electrical energy everybody certainly —An objective test is used in a home. (F) everyone certainly not generally easier to score than an essay test. (T) Qualifiers: usually frequently often sometimes 6. Avoid lifting statements from the text, lecture or some seldom other materials so that memory alone will not permit a many much correct answer. probably a majority apt to most Undesirable: Desirable: might a few For every action there is an If you were to stand in a may unlikely opposite and equal reaction. canoe and throw a life jacket forward to another canoe, chances are your canoe would jerk backward. Remember… ! Determine that the questions are 7. Avoid using negatively stated item statements. appropriately answered by “True” or “False” rather than by Undesirable: Desirable: some other type of response, The Supreme Court is not The Supreme is composed of such as “Yes” or “No.” composed of nine justices. nine justices. ! Arrange the statements so that there is no discernible pattern of 8. Avoid the use of unfamiliar vocabulary. answers (such as T , F, T, F, T, F and T, T, F, F, T, T, F, F) for True Undesirable: Desirable: and False statements. According to some politicians, According to some politicians, the raison d’etre for capital justification for capital punish- ! Avoid the tendency to add de- punishment is retribution. ment is retribution. tails in true statements to make them more precise. The answers should not be obvious to stu- Writing Hint… dents who do not know the ma- One method for developing true-false items is to write a set of terial. true statements that cover the content, then convert approximately half of them to false statements. Remember: When changing ! Be sure to include directions that items to false (as well as in writing the true statements initially), tell students how and where to state the items positively, avoiding negatives or double negatives. mark their responses. 23Attention Students: Attention Students: Attention Students: Attention Students: Attention Students: True-False Test Taking Tips True-False Test Taking Tips True-False Test Taking Tips True-False Test Taking Tips True-False Test Taking Tips ! When you do not know or cannot remember information to determine the truth of a statement, assume that it is true. ! There are generally more true questions on true-false exams than false questions because instructors tend to emphasize true questions. ! If there is specific detail in the statement, it may also tend to be true. For example, the statement "There are 980 endangered species worldwide" has specific detail and is likely to be true. ! Look for extreme modifiers that tend to make the question false. Extreme modifiers, such as always, all, never, or only make it more likely that the question is false. ! Identify qualifiers that tend to make the question true. Qualifiers (seldom, often, many) make the question more likely true. ! Questions that state a reason tend to be false. ! Words in the statement that cause justification or reason (since, because, when, if) tend to make the statement false because they bring in a reason that is incorrect or incomplete. Variations in Writing Variations in Writing Variations in Writing Variations in Writing Variations in Writing True-False Test Items True-False Test Items True-False Test Items True-False Test Items True-False Test Items The True-False-Correction Question… In this variation, true-false statements are presented with a key word or brief phrase that is underlined. It is not enough that a student correctly identify a statement as being false. To receive credit for a statement labeled false, the student must also supply the correct word or phrase which, when used to re- place the underlined part of the statement, makes the statement a true one. This type of item is more thorough in determining whether students actually know the information that is presented in the false statements. While a student might correctly guess that a statement is false, no credit would be given unless the student could change the statement to a true one by writing word/words to replace underlined word(s). (Continued on next page…) 24Variations in Writing Variations in Writing Variations in Writing Variations in Writing Variations in Writing True-False Test Items True-False Test Items True-False Test Items True-False Test Items True-False Test Items (continued) The teacher decides what word/phrase can be changed in the sentence; if stu- dents were instructed only to make the statement a true statement, they would have the liberty of completely rewriting the statement so that the teacher might not be able to determine whether or not the student understood what was wrong with the original statement. If, however, the underlined word/phrase is one that can be changed to its oppo- site it loses the advantage over the simpler true-false question because all the student has to know is that the statement is false and change is to is not. The Yes-No Variation… In the yes-no variation, the student responds to each item by writing, circling or indicating yes-no rather than true-false. An example follows: What reasons are given by students for taking evening classes? In the list below, circle Yes if that is one of the reasons given by students for enrolling in evening classes; circle No if that is not a reason given by students. Yes No They are employed during the day. Yes No They are working toward a degree. Yes No They like going to school. Yes No There are no good television shows to watch. Yes No Parking is more plentiful at night. The A-B Variation… The example below shows a question for which the same two answers apply. The answers are categories of content rather than true-false or yes-no. This is another form of forced choice question because for each item the student must choose between A and B. Indicate whether each type of question below is a selection type or supply type by circling A if it is selection, B if it is supply. Select Supply AB Multiple choice AB True-False AB Essay AB Matching AB Short Answer 25True-False Test Items: Conclusion True-False Test Items: Conclusion True-False Test Items: Conclusion True-False Test Items: Conclusion True-False Test Items: Conclusion Ask yourself: Aim for Higher Levels of Learning Why are these While true-false and other forced choice questions are generally true-false questions used to measure knowledge and understanding, they could also be used at higher levels. Students could be provided with a set of crummy? information new to them, perhaps a portfolio, set of data, or a writ- ten work of some type, then asked various forced choice questions 1. There is no advantage for not related to the content or the presence/absence of certain character- using specific determiners in istics in the work. true-false items. T F 2. Test validity is a function of Anticipate Scoring Ranges test reliability, which can be improved by using fewer Scores on true-false items tend to be high because of the ease of items. T F guessing correct answers when the answer is not known. With only two choices (true or false) the student could expect to guess 3. A nickel is larger than a correctly on half of the items for which correct answers are not dime. T F known. 4. An eagle's range of sight is If a student knows the correct answers to 10 questions out of 20 precisely 1,000 ft. T F and guesses on the other 10, the student could expect a score of 15. The teacher can anticipate scores ranging from approximately 5. The telephone was invented 50% for a student who did nothing but guess on all items to 100% a long time ago. T F for a student who knew the material. "A major distinction between In the final analysis… the true-false test item and items The true-false test is probably the best known of the various types in a multiple choice format, is of objective test items. It is the easiest to construct and at the same that the true-false statement time the most abused. The students learn the weaknesses that are contains no criterion for inherent in many such items and are able to obtain high scores by answering the question. Each noting the grammatical construction, the choice of words or other examinee must ask the question: clues. True or false with respect to what? Each true-false item must The true-false test can be used effectively as an instructional test to be unequivocally true or promote interest and introduce points for discussion. This perhaps unequivocally false. It is is the most important use for the plain true-false item. It is a valu- imperative that proper wording able type of test to use in giving short, daily quizzes that may be and the elimination of used to motivate the students for a new assignment, to review a extraneous clues are more previous lesson, to locate points to be retaught or to introduce con- crucial with the true-false item troversial points for class discussion. than with any other test format." Writing Test Items, na, Michigan State University Dept. of Education, Dec. 1999 26Matching Test Items Matching Test Items Matching Test Items Matching Test Items Matching Test Items Matching questions A simple matching item consists of two columns: one column of provide a most efficient stems or problems to be answered, and another column of re- way to test knowledge in sponses from which the answers are to be chosen. Traditionally, the column of stems is placed on the left and the column of re- courses in which events, sponses is placed on the right. An example is given below. dates, names, and places are important. Matching Directions: On the line next to each children’s book in questions are also Column A print the letter of the animal or insect in Column appropriate for the B that is a main character in that book. Each animal or insect sciences in which in Column B can be used only once. numerous experiments, Column A Column B experimenters, results, ____1. Charlotte’s Web A. Bear and special terms and ____2. Winnie the Pooh B. Chimpanzee definitions have to be ____3. Black Beauty C. Cricket remembered. ____4. Tarzan D. Deer ____5. Pinocchio E. Horse ____6. Bambi F. Pig Section Summary Good for: ! Knowledge level The student reads a stem (Column A) and finds the correct response ! Some comprehension level, from among those in Column B. The student then prints the letter if appropriately constructed of the correct response in the blank beside the stem in Column A. An alternative is to have the student draw a line from the correct Types: response to the stem, but this is more time consuming to score. ! Terms with definitions ! Phrases with other phrases In the above example notice that the stems in Column A are as- ! Causes with effects signed numbers (1, 2, 3, etc.). The items in Column B are desig- ! Parts with larger units nated by capital letters. Capital letters are used rather than lower ! Problems with solutions case letters in case some students have reading problems. Also there are apt to be fewer problems in scoring the student’s hand- Advantages: written responses if capital letters are used. ! Maximum coverage at know- ledge level in a minimum Also in the above example, the student only has to know five of amount of space/preptime the six answers to get them all correct. Since each animal in Col- ! Valuable in content areas that umn B can be used only once, the one remaining after the five have a lot of facts known answers have been recorded is the answer for the sixth premise. One way to reduce the possibility of guessing correct Disadvantages: answers is to list a larger number of responses than premises. ! Time consuming for students ! May not be appropriate for higher levels of learning 27Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items 1.Problem: Faulty directions. Directions: "Place the letter of the term in the right hand column on the line to the left of the definition column." Circle the letter(s) that describe the best way to revise these directions: A. Add: “Match the following” B. Add: “Each term may not be used more than once” C. Change the order of the directions provided D.No changes needed 2.Problem: Unrelated topics. ____1.Year in which WWII began A. Joseph Stalin ____2.British Prime Minister in WWII B. Franklin D. Roosevelt ____3.U.S. President during WWII C. 1939 ____4.German dictator in WWII D. Winston Churchill E. Adolf Hitler Circle the letter(s) that describe the best way to revise this matching test. A. Change one of the descriptions to read: “Russian dictator in WWI” B. Add an item to the left hand column C. Add a description that reads: “Year in which WWI began” D.Remove option C. from the right hand column E. Remove all stimuli and responses that do not concern leaders in WWII 3.Problem: Mixing matching with completion. Directions: On the line to the left of each statement write the letter of the atomic particles from the right hand column that the statement describes. Use each particle only once. ____1.An ____orbits the nucleus. A. Electron ____2.Positively charged particles are called _____. B. Neutron ____3.A _____ has no charge. C. Protons ____4.The _____ is located in the center of an atom. D. Nucleus E. Ion Circle the letter(s) that describe the best way to revise this matching test. A. Edit all the stimuli on the left to be complete statements. B. Remove all the blanks from the stimuli on the left. C. Change the order of the responses on the right. D.Edit the stimuli to be grammatically unbiased (i.e. singular/plural) 28 Answers: 1-C, 2-E, 3-A, B & DTest Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items Test Your Knowledge of Matching Test Items 4.Directions: The four statements presented below refer to the structure of the matching test, specifically what elements should be in Column A and what elements should be in Column B. At the left of each statement are the letters A and B. Circle A if Column A is the best choice; circle B if Column B is the best choice. A B I. When presenting words and their definitions, which column should contain the definitions, which are longer than the words? A B 2. Items arranged in chronological order would be found in which column? A B 3. Premise is the term applied to the items in which column? A B 4. Items are designated by numbers in which column? 5.Directions: For the four learning objectives listed below, decide whether a matching exercise would be an appropriate method of assessment (Assume that you can construct a list of 6-8 items for the matching question.) Circle YES if appropriate; circle NO if not appropriate. YES NO A. The student will be able to recognize the cities in/near which the major battles in the American Revolution took place. YES NO B. The student will be able to differentiate between words that are spelled correctly and those spelled incorrectly. YES NO C. The student will be able to identify the elements with their symbols from the periodic table. YES NO D. The student will be able to identify the English words for various fruits that are represented by their Spanish language counterparts. 6.Directions: On the lines following this matching question supply four recommendations to improve this question. Hitler A. Year in which WWII began Mulroney B. A Canadian Prime Minister Tank C. A German dictator during the WWII 1939 D. An armored vehicle used originally to break the trench war stalement in WWI Recommendations: 1._____________________________________________________________________ 2._____________________________________________________________________ 3._____________________________________________________________________ 4._____________________________________________________________________ 29 Answers: 4-A, B, A, A; 5-Yes, No, Yes, Yes; 6-Examples: Need directions, reverse Column A and Column B, make items similar, increase the number of responsesSuggestions For Writing Matching Test Items Suggestions For Writing Matching Test Items Suggestions For Writing Matching Test Items Suggestions For Writing Matching Test Items Suggestions For Writing Matching Test Items 1. Include directions which clearly state the basis for matching the stimuli with the responses. Remember… Explain whether or not a response can be used more than ! Review your teaching objectives once and indicate where to write the answer. to make sure that a matching Undesirable: component is appropriate. Directions: Match the following. Desirable: ! Keep matching items brief, limit- Directions: On the line to the left of each identifying location ing the list of stimuli to 10 - 15. and characteristics in Column I, write the letter of the country in Column II that is best defined. Each country in Column II may be used more than once. ! When possible, reduce the amount of reading time by in- 2. Use only items that share the same foundation of cluding only short phrases or information. single words in the response list. Unrelated topics included in the same matching item may ! Use the more involved expres- allow for obvious matches and mismatches. sions in the stem and keep the Undesirable: responses short and simple. Directions: Match the following. 1. Water A. NaCl 2. Discovered Radium B. Fermi ! Arrange the list of responses in 3. Salt C. NH3 some systematic order if possible 4. Ammonia D. 1942 (chronological, alphabetical). 5. Year of the first E. H 0 2 Nuclear Fission F. Curie G. 1957 ! Make sure that there are never multiple correct responses for Desirable: one stem (although a response Directions: On the line to the left of each compound in Col- may be used as the correct an- umn I, write the letter of the compound’s formula presented in swer for more than one stem). Column II. Use each formula only once. Column I Column II ! Avoid breaking a set of items ____1. Water A. H S0 2 4 (stems and responses) over two ____2. Salt B. HCl pages. (Students go nuts flipping ____3. Ammonia C. NaCl pages.) ____4. Sulfuric Acid D. H 0 2 E. H HCl 2 303. Avoid grammatical or other clues to the correct response. Undesirable: Directions: Match the following in order to complete the sentences on the left. ____1.Plato insisted that government was A. The Prince. ____2.Machiavelli wrote about achieving political B. desirable and inevitable unity in C. a science requiring experts. ____3.Hobbes argued that human nature made D. organized along industrial lines. absolute monarchy E. Communism. ____4.Marx was a German philosopher and economist who founded Desirable: Directions: On the line to the left of each statement write the letter of the philosopher from the right hand column that the statement describes. Use each philosopher once. ____1.Thought government was a science requiring experts. A. Hobbes ____2.Described methods of achieving political unity. B. Marx ____3.Founded Communism. C. Machiavelli ____4.Believed that human nature made absolute D. Durkheim monarchy desirable and inevitable E. Plato 4. The column of stimuli on the left should set the question clearly. Undesirable: Directions: Match the following. ____1. City dwellers A. Wild animals ____2. Hunter-gatherers B. Farm ____3. Pastoral nomads C. Apartment buildings D. Graze animals Desirable: Directions: On the line to the left of each definition, write the letter of the term in the right hand column that is defined. Use each term only once. ____1. Live in areas of high population density. A. Pastoral nomads ____2. Move from one place to another in search of wild animals. B. Ranchers ____3. Move from one place to another with grazing animals. C. Hunter-gatherers ____4. Till land for cash crops. D. City dwellers E. Farmers 31Attention Students: Attention Students: Attention Students: Attention Students: Attention Students: Matching Test Taking Tips Matching Test Taking Tips Matching Test Taking Tips Matching Test Taking Tips Matching Test Taking Tips ! Read the directions. There are usually two lists that need to be matched. Take a look at both lists to get a feel for the relationships and build your confidence. ! Use one list as a starting point and go through the second list to find a match. This process organizes your thinking. It will also speed your answers because you become familiar with the second list and will be able to go straight to a match that you saw when looking through the lists a previous time. ! Move through the entire list before selecting a match. If you make a match with the first likely answer, you may make an error, because an answer later in the list may be more correct. ! Cross off items on the second list when your are certain that you have a match. This seems simplistic, but it helps you feel confident and stay organized. ! Do not guess until all absolute matches have been made. If you guess early in the process, you will likely eliminate an answer that could be used correctly for a later choice. How to Study For a Matching Test How to Study For a Matching Test How to Study For a Matching Test How to Study For a Matching Test How to Study For a Matching Test If your instructor usually includes a matching section in a typical exam, here is one way to prepare for it. As you read the textbook, be alert for facts and ideas that are associated with people’s names. On a separate sheet, list the names and facts opposite each other, resulting in two distinct vertical columns, as in the following example. NAMES FACTS OR IDEAS SUBJECT Susan B. Anthony Women’s movement Sociology Jack London Call of the Wild Literature George Washington Carver Agricultural chemist Science Lewis and Clark American explorers History George A. Miller Magic number seven Psychology William James Pragmatism Philosophy Mozart Marriage of Figaro Music To master your list, cover the fact column with a sheet of paper. Look at each item in the name column, and recite and write the corresponding fact or idea. Then, to make sure that you learn the material both ways, block out the name column and use the facts as your clues. The example given above includes items from various subject areas. 32Variations for Creating Matching Tests Variations for Creating Matching Tests Variations for Creating Matching Tests Variations for Creating Matching Tests Variations for Creating Matching Tests Aiming for Higher Keylists or Masterlists Example Keylists or Masterlists Example Keylists or Masterlists Example Keylists or Masterlists Example Keylists or Masterlists Example Order Thinking Skills 11 17 1 A (Na ) B (CI )C (H ) Usually matching items measure recognition of factual knowledge 22.9898 35.453 1.00797 rather than higher order thinking l + 1, 3, 5, 7 l 2 5 skills such as analysis and synthe- (Ne)35 (Ne) 3s 3p ls’ sis. This does not mean, however, 892 -34.7 -252.7 that variations cannot be con- 97.5 -102 -259.2 structed to aim for higher levels of 0.97 1.56 0.071 thinking. Refer to the chemical symbols above to answer the following: ___1. Which of the above elements has the largest atomic weight? One variation, presented below, ___2. Which of the above elements has the largest atomic number? combines elements of a multiple ___3. Which of the above elements has the lowest boiling point? choice test item with a matching ___4. Which of the above elements has the lowest melting point? component. ___5. Which of the above elements has the highest density? ___6. Which of the above elements has the least number of electrons? Item Components ___7. Which of the above elements has the least number of protons ? ___8. Which of the above elements represents chlorine? A. Correct answer(s) ___9. Which of the above elements represents sodium? B. foil(s) C. option(s) D. stem(s) Ranking Example Ranking Example Ranking Example Ranking Example Ranking Example ___1. The components of a TOPIC: Social Studies, Western Civilization multiple choice item are a Directions: Number (1-8) the following events in the history of ___2. a. stem and several foils. ancient Egypt in the order in which they occurred, using 1 for the ___3. b. correct answer and earliest event. several foils. ___4. c. stem, a correct answer, _____Egypt divided; ruled by Libyan kings, Nubian pharaohs, and some foils. Assyrians, and Persians ___5. d. stem and a correct answer _____Seizure of power by Hyksos kings ___6. _____Upper and Lower Egypt are united by Menes _____Alexander the Great conquers Egypt In the above example it is neces- _____Reunification of Egypt under pharaoh Mentuhotop II sary to answer the multiple choice _____Rise of feudal lords leads to anarchy item in order to answer the match- _____Thutmose III expands empire to the Euphrates ing item. Note also that the re- _____Many kings with short reigns; social and political chaos sponses (item components) in the list at the top have an (s) added to Note in the above example the implied column of responses is each response to eliminate singu- 1, 2, 3, 4, 5, 6, 7, 8. lar-plural clues. 33 Answers: 1-D, 2-B, 3-B, 4-A, 5-B, 6-CCompletion or Fill-in-the-Blank Test Items Completion or Fill-in-the-Blank Test Items Completion or Fill-in-the-Blank Test Items Completion or Fill-in-the-Blank Test Items Completion or Fill-in-the-Blank Test Items No-Hint Test Construction Completion items are especially useful in assessing mastery of fac- tual information when a specific word or phrase is important to Section Summary know. They preclude the kind of guessing that is possible on lim- Good for: ited-choice items since they require a definite response rather than ! Knowledge levels simple recognition of the correct answer. Because only a short ! Recall and memorization answer is required, their use on a test can enable a wide sampling of facts of content. A completion item requires the student to answer a question or to Advantages: finish an incomplete statement by filling in a blank with the correct ! Good for who, what, word or phrase. For example, where, when content According to Freud, personality is made up of three major ! Minimizes guessing systems, the________, the _________ and the __________. ! Encourages more intensive study. Student must know the answer vs. recognizing the answer. What About Synthesis and Evaluation? ! Can usually provide an Completion items tend to test only rote, repetitive responses and objective measure of may encourage a fragmented study style since memorization of student achievement or bits and pieces will result in higher scores. They are more difficult ability to score than forced-choice items and scoring often must be done by the test writer since more than one answer may have to be considered correct. Disadvantages: ! Difficult to assess higher levels of learning because the answers to completion Is Short Answer the Same Thing? items are usually limited to A distinction should be made between completion—often referred a few words to as fill-in-the-blank—and short answer questions. With comple- ! Difficult to construct so that tion questions the response is usually one or two words that fit on the desired response is a line provided by the tester. Short answer questions may require clearly indicated one sentence or even a paragraph to fully answer the question. ! May overemphasize memo- rization of facts Short answer questions are appropriate in measuring a student's ! Questions may have more understanding of principles or the ability to solve problems or ap- than one correct answer ply principles. Short answer questions go beyond simple recall or ! Scoring is time consuming recognition. They require students to consider various factors and to arrive at solutions, whether they deal with mathematical or other situations. (Continued on next page…) 34Short Answer (continued from previous page…) Strategies for developing short answer questions are similar to those On the whole, completion concerning completion but have an added dimension requiring test items have little strategies appropriate for essay questions. As an example, scoring advantage over other item completion questions can be more objective than scoring short types unless the need for answer questions which require a subjective interpretation on the specific recall is essential. teachers part. The information contained in this section primarily focuses on completion or fill-in-the-blank questions. Test Your Knowledge of Completion Items Directions: Fill in the blanks. 1. A fill-in-the-blank question asks students to supply rather than _____________ the answer. 2. The main problem in constructing completion items is to limit the number of possible ____________. 3. Put blanks at the ____________ of the statement rather than the ____________. 4. Completion items are faster to answer than ____________ items because there are no alternatives to consider. 5. Make the ____________ of equal length. 6. A direct ____________ is often more desirable than an incomplete ____________. 7. When doing fill-in-the-blank test items, read the ____________ with the intent to give an answer that is ____________ correct. 8. Always concentrate on the ____________ of blanks to fill in. 9. When you do not know the exact ____________, provide a descriptive answer. 10. Scoring completion items is less ____________ than multiple choice or true-false because the student supplies the response. 35 Answers: 1-select; 2-answer; 3-end, beginning; 4-multiple choice; 5-blanks; 6- question, sentence; 7-question, grammatically; 8-number; 9-response; 10-objectiveSuggestions For Writing Completion Test Items Suggestions For Writing Completion Test Items Suggestions For Writing Completion Test Items Suggestions For Writing Completion Test Items Suggestions For Writing Completion Test Items Remember… 1. Omit only significant words from the statement. ! When possible, provide explicit Undesirable: directions as to what amount of Every atom has a central ____________ called a nucleus. variation will be accepted in the answers. Desirable: Every atom has a central core called a(n) ____________. ! Give much more credit for completions than for true-false or matching items. 2. Do not omit so many words from the statement that the intended meaning is lost. ! Avoid using a long quote with multiple blanks to complete. Undesirable: The ____________ were to Egypt as the ____________ were to ! When working with definitions, Persia and as ____________ were to the early tribes of Israel. supply the term, not the defini- tion, for a better judge of student Desirable: knowledge. The Pharaohs were to Egypt as the ____________ were to Per- sia and as ____________ were to the early tribes of Israel. ! For numbers, indicate the degree of precision/units expected. ! Facilitate scoring by having the 3. Avoid obvious clues to the correct response. students write their responses on lines arranged in a column to the Undesirable: left of the items. Most of the United States’ libraries are organized according to the ____________ decimal system. Desirable: Suggestion… Which organizational system is used by most of the United It is difficult to write completion States’ libraries? ____________. items so that there is only one correct answer. When prepar- ing a scores key, list the correct answer and any other accept- 4. Be sure there is only one correct response. able alternatives. Be consistent in using the key; it would not be Undesirable: fair to accept an answer as right Trees which shed their leaves annually are ____________. on one paper and not accept it Desirable: on others. Trees which shed their leaves annually are called ____________. 36Ask yourself: 5. Avoid grammatical clues to the correct response. Why are these If the indefinite article is required before a blank, use a(n) so that the student does not know if the correct answer begins completion items with a vowel or a consonant. crummy? Undesirable: A subatomic particle with a negative electric charge is called ! The_______ of _______ took an ____________. place in the year _______. Desirable: A subatomic particle with a negative electric charge is called ! _______ was a crucial event a(n) ____________. to German history. 6. If possible, put the blank at the end of a statement rather than at the beginning. Beware of Asking for a response before the student understands the intent Clever Students of the statement can be confusing and may require more read- ing time. Nudity, infancy, and bliss are some of the answers for the fol- Undesirable: lowing completion item: ____________ is the measure of central tendency that is most affected by extremely high or low scores. ! George Washington was born in the state of _______. Desirable: The measure of central tendency that is most affected by ex- tremely high or low scores is the ____________. Attention Students: Attention Students: Attention Students: Attention Students: Attention Students: Completion Test Taking Tips Completion Test Taking Tips Completion Test Taking Tips Completion Test Taking Tips Completion Test Taking Tips ! Read the question with the intent to give an answer and make the sentence grammatically correct. In this process it is important to focus on how the sentence is written. For example, if the blank is preceded by the article “an,” you know the word that goes in the blank must start with a vowel. ! Concentrate on the number of blanks in the sentence and the length of the space. The test maker is giving you clues to the answer by adding spaces and making them longer. ! Provide a descriptive answer when you cannot think of the exact word or words. The instructor will often give you credit or partial credit when you demonstrate that you have studied the material and can give a credible answer, even when you have not given the exact words. 37Essay Test Items Essay Test Items Essay Test Items Essay Test Items Essay Test Items A typical essay test usually consists of a small number of questions Essay tests present a to which the student is expected to recall and organize knowledge realistic task to the in logical, integrated answers. An essay test item can be an ex- tended response item or a short answer item. An example of each student. In real life, a type follows. person is required to Extended Response organize and commun- Compare the writings of Bret Harte and Mark Twain in terms icate thoughts rather of settings, depth of characterization, and dialogue styles than respond to multiple of their main characters. (10 pts. 20 minutes) choice questions. Short Answer Identify research methods used to study the S-R (Stimulus- Response) and S-O-R ( Stimulus-Organism-Response) theo- Section Summary ries of personality. (5 pts. 10 minutes) Good for: ! Application, synthesis and The Benefits of Essay Tests evaluation levels The main advantages of essay and short answer items are that they permit students to demonstrate achievement of such higher level Types: objectives as analyzing and critical thinking. Written items offer ! Extended response: synthe- students the opportunity to use their own judgment, writing styles, sis and evaluation levels; a and vocabularies. They are less time consuming to prepare than lot of freedom in answers any other item type. ! Restricted response: more consistent scoring, outlines Research indicates that students study more efficiently for essay parameters of responses type examinations than for selection (multiple choice) tests. Stu- dents preparing for essay tests focus on broad issues, general con- Advantages: cepts, and interrelationships rather than on specific details. This ! Students less likely to guess studying results in somewhat better student performance regard- ! Easy to construct less of the type of exam they are given. Essay tests also give the ! Stimulates more study instructor an opportunity to comment on students' progress, the ! Allows students to demon- quality of their thinking, the depth of their understanding, and the strate ability to organize difficulties they may be having. knowledge, express opin- ions, show originality. Unfortunately… Essay tests consisting only of written items permit only a limited Disadvantages: sampling of content learning due to the time required for students ! Can limit amount of mate- to respond. Essay items are not efficient for assessing knowledge rial tested, therefore has of basic facts and provide students more opportunity for bluffing, decreased validity. rambling, and snowing than limited choice items. They favor stu- ! Subjective, potentially dents who possess good writing skills and neatness. They are pit- unreliable scoring. falls for students who tend to go off on tangents or misunderstand ! Time consuming to score. the main point of the question. 38I’d like to use essay tests, but... Marilla Svinicki, University of Texas at Austin The Professional & Organizational Development Network in Higher Education Essay question: "Discuss the importance of the Do you cringe when you read the kind of tortured prose and frac- nature/nurture controversy in the tured thinking that is represented by the example on the left? Or shaping of current developmen- plow through paragraph after paragraph of detail in a student’s tal theory." answer in hopes of finding an original thought. It has been our habit in the past to blame the students, the school system and the English department for not teaching our students Student answer: how to write a solidly argued, concisely worded essay answer. But we must face the fact that we are as much to blame for their "The nature/nuture (sic) contro- imprecise prose as those other entities. versy was very impotent (sic) in Part of the problem may lie in the way instructors help (or fail to shaping current developmental help) students prepare for writing essay tests. Learning specialists theory becuse (sic) it was needed have known for a long time that the kind of preparation needed for to help people who were doing responding to essay questions is different from that needed for work in that area to come up objective tests. Unfortunately, many of our students prepare for all with their current theories." exams with the same learning strategies, and then are ill-equipped to tackle the kind of thinking needed during essay tests. (See page 56 for a continuation of this essay.) Read 'Em and Weep Essay Test Items* Read 'Em and Weep Essay Test Items* Read 'Em and Weep Essay Test Items* Read 'Em and Weep Essay Test Items* Read 'Em and Weep Essay Test Items* History: Describe the history of the papacy from its origins to the present day, concentrat- ing especially, but not exclusively, on its social, political, economic, religious, and philo- sophical impact on Europe, Asia, America, and Africa. Be brief, concise, and specific. Biology: Estimate the differences in subsequent human culture if our form of life had devel- oped 500 million years earlier, with special attention to its probable effect on the English parliamentary system. Prove your thesis. Psychology: Based on your knowledge of their works, evaluate the emotional stability, degree of adjustment, and repressed frustrations of each of the following: Alexander the Great, Rameses II, Gregory of Nicia, Hammurabi. Support your evaluation with quotations from each man’s work, making appropriate references. It is not necessary to translate. Sociology: Estimate the sociological problems which might accompany the end of the world. Construct an experiment to test your theory. Economics: Develop a realistic plan for refinancing the national debt. Trace the possible effects of your plan in the following areas: Cubism, the Donatist controversy, the wave theory of light. Physics: Explain the nature of matter. Include in your answer an evaluation of the impact of the development of mathematics on science. Philosophy: Describe everything in detail. Be objective and specific. *Just kidding 39Test Your Ability to Construct Essay Test Items Test Your Ability to Construct Essay Test Items Test Your Ability to Construct Essay Test Items Test Your Ability to Construct Essay Test Items Test Your Ability to Construct Essay Test Items Requiring Higher Order Thinking Skills Requiring Higher Order Thinking Skills Requiring Higher Order Thinking Skills Requiring Higher Order Thinking Skills Requiring Higher Order Thinking Skills Directions: In the blank to the left of the essay item statement, place the letter of the learning level that is best indicated by the words contained in this statement. A=Knowledge B=Comprehension C=Application D=Analysis E=Synthesis F=Evaluation _____1. Essay items may begin with modify, prepare, or solve. _____2. Essay items may begin with define, label, outline, or state. _____3. Essay items may begin with convert, predict, or estimate. _____4. Essay items may begin with appraise, interpret, or criticize. _____5. Essay items may begin with categorize, compile, or re-arrange. _____6. Essay items may begin with diagram, illustrate, or separate. Directions: Use the above letters to identify the learning level of the essay test items listed below. Place your answer in the blank to the left of each item. _____1. In the president's State of the Union Address, which statements are based on facts and which are based on assumptions? _____2. How would you restructure the school day to reflect children's developmental needs? _____3. Why is Bach's Mass in B Minor acknowledged as a classic? _____4. Calculate the deflection of a beam under uniform loading _____5. Summarize the basic tenets of deconstructionism. _____6. List the steps involved in titration. Directions: Write a test item for each learning level. My topic:______________________________________ 1. Knowledge____________________________________________________________ _____________________________________________________________________ Example Topic: Asbestos: What is asbestos? 2. Application____________________________________________________________ _____________________________________________________________________ Example Topic: Asbestos: Consider the crystal structures of chrysotile and crocidolite. Why should the most common mineral be the less hazardous? 3. Synthesis______________________________________________________________ _____________________________________________________________________ Example Topic: Asbestos: Design a study to reasonably demonstrate the dangers posed by asbestos to the general populace. 4. Evaluation_____________________________________________________________ _____________________________________________________________________ Example Topic: Asbestos: The “asbestos hazard” is either (1) nothing more than a costly bureaucratic creation or (2) a hazard that accounts for tens of thousands of deaths annually. Which of the two controversial arguments has the best scientific support? 40Suggestions For Writing Essay Test Items Suggestions For Writing Essay Test Items Suggestions For Writing Essay Test Items Suggestions For Writing Essay Test Items Suggestions For Writing Essay Test Items Standard Phrases for 1. Formulate the question so that the task is clearly Writing Essay Test Items defined for the student. Agreement or Disagreement: The Use words that aim the student to the approach you want them student is being asked to assert and to take. Words like discuss and explain can be ambiguous. If support a thesis with evidence. you use discuss, then give specific instructions as to what points should be discussed. Analyze: Analyzing is a picking apart of the whole. Undesirable: Discuss Karl Marx’s philosophy. Classification and Division: Grouping items into a category ac- Desirable: cording to a consistent principle. Compare Marx and Nietzsche in their analysis of the underly- ing problems of their day in 19th century European society. Compare/Contrast: Comparing shows similarities, while contrast- ing points out differences. 2. Pay attention to the number of items. Cause and Effect: Establishes a link between two things and also to describe the outcome. In order to obtain a broader sampling of course content, use a relatively large number of questions requiring shorter answers Define: Consists of three parts: (one-half page) rather than just a few questions involving long term, class, and differentiating answers (2-3 pages). characteristics. Define and give an example of: 3. Avoid the use of optional questions on an essay test. Asks students to not only define the term, but to supply an example. When students answer different questions, they are actually tak- Describe: Requires students to ex- ing different tests. If there are five essay questions and students plain something in detail. are told to answer any three of them, then there are ten differ- Discuss: Too vague and may elicit ent tests possible. It makes it difficult to discriminate between vague, overgeneralized, unsup- the student who could respond correctly to all five, and the ported responses. student who could answer only three. Illustrate: Give examples and/or Use of optional questions also affects the reliability of the scor- analogies to demonstrate a particu- ing. If we are going to compare students for scoring purposes, lar process/idea or steps in a series. then all students should perform the same tasks. Another prob- lem is that students may not study all the course material if they Summarize: The overall view of know they will have a choice among the questions. some process, speech, play, con- cept, etc. (Continued on next page…) 415. Write essay items at different levels of learning. Remember… The goal is to write essay items that measure higher cognitive processes. The item should represent a situation that tests the ! Make essay questions compre- student’s ability to use knowledge in order to analyze, justify, hensive rather than focused on explain, contrast, evaluate, and so on. small units of content. Try to use verbs that elicit the kind of thinking you want the ! Provide clear directions as to the students to demonstrate. Instructors often have to use their best expectations. judgment about what cognitive skill each question is measur- ing. Ask a colleague to read the questions and classify them ! Allow students an appropriate according to Bloom’s taxonomy. amount of time. (It is helpful to give students some guidelines on how much time to use on each 5. Choose a scoring model. question, as well as the desired length and format of the re- The major task in scoring essay tests is to maintain consistency, sponse, such as full sentences, to make sure that answers of equal quality are given the same phrases only, outline, etc.) number of points. There are two approaches to scoring essay items: (1) analytic or point method and (2) holistic or rating ! Inform students, in advance of method. answering the questions, of the proportional value of each item 1. Analytic: Before scoring, prepare an ideal answer in which in comparison to the total grade. the major components are defined and assigned point val- ues. Read and compare the student’s answer with the model ! Require students to demonstrate answer. If all the necessary elements are present, the stu- command of background infor- dent receives the maximum number of points. Partial credit mation by asking them to provide is given based on the elements included in the answer. In supporting evidence for claims order to arrive at the overall exam score, the instructor adds and assertions. the points earned on the separate questions. 2. Holistic: This method involves considering the student’s an- ! Students should be informed swer as a whole and judging the total quality of the answer about how you treat such things relative to other student responses or the total quality of the as misspelled words, neatness, answer based on certain criteria that you develop. handwriting, grammar, etc. ! Decide how to treat irrelevant or 5. Prepare students to take essay exams. inaccurate information contained in students’ answers. Essay tests are valid measures of student achievement only if students know how to take them. Many college freshmen do ! Write comments on the students’ not know how to take an essay exam, because they have not answers. Teacher comments been required to learn this skill in high school. make essay tests a good learning experience for students. Com- Take some class time to tell students how to prepare for and ments serve to refresh your how to take an essay exam. Use old exam questions and let memory should the student ques- students see what an "A" answer looks like and how it differs tion the grade. from a "C" answer. 42A Four-Step Process in Grading Essay Test Items A Four-Step Process in Grading Essay Test Items A Four-Step Process in Grading Essay Test Items A Four-Step Process in Grading Essay Test Items A Four-Step Process in Grading Essay Test Items Step One Research Shows That a When the assignment is given… Number of Factors ! Figure out what the purpose of the assignment is, and gener- Can Bias the Grading ate grading criteria based upon that purpose. of Essay Tests ! Share the criteria you decide upon with your students: hand it out in class, and post it on your door. ! Provide models of your grading criteria to your students. ! Different scores may be assigned by different readers or by the Step Two same reader at different times. When the assignments are turned in… ! A context effect may operate; an ! Quickly overview a percentage of the papers to get an over- essay preceded by a top quality all sense of how the group did on the assignment. essay receives lower marks than ! Skim some papers that you feel are representative of the range when preceded by a poor qual- of quality in the student work. ity essay. ! Use these papers to start four piles: High, Medium High, Medium Low, and Low. ! The higher the essay is in the stack of papers, the higher the Step Three score assigned. Start the grading… ! Papers with strong answers to ! Always use a pencil on your first run through: as you de- items appearing early in the test velop your sense of how the students did, you will probably and weaker answers later will go back and fine-tune the papers you graded first! fare better than papers with the ! Having separated the papers into piles (high, medium, low: weaker answers appearing first. not letter grades yet), do an initial read through and assign a preliminary, holistic grade based upon a general impression ! Scores are influenced by the ex- of the work. Do not get bogged down in details yet, short of pectations that the reader has for marking a plus (+) or minus (-) in the margins next to issues the students performance. If the that strike you. reader has high expectations, a ! Now re-read each paper for how it addresses the criteria iden- higher score is assigned than if tified for the assignment. Two papers may address the same the reader has low expectations. criteria differently. Focus first on what the paper does, be- If we have a good impression of fore getting to what it does not. After a sympathetic read, the student, we tend to give him/ give it a critical read. her the benefit of the doubt. Step Four ! Scores are influenced by qual- Mark up the papers… ity of handwriting, neatness, spelling, grammar, etc. ! Interactive grading poses questions and presents problems the student needs to resolve. For example: “Is this (x) what you mean? How does this connect to your main point?” 43Attention Students: Attention Students: Attention Students: Attention Students: Attention Students: Essay Test Taking Tips Essay Test Taking Tips Essay Test Taking Tips Essay Test Taking Tips Essay Test Taking Tips 1. Organize your thoughts before you begin to write A short outline on a separate piece of paper will improve your thinking. There is usually a main idea or issue, several supporting issues and examples to illustrate the issues. 2. Paraphrase the original question to form your introductory statement This benefits you in two ways. First, it helps you get the question straight in your mind. Second, it may protect you from the teacher. If you have re- phrased the question, the teacher can see how you understood the question. Perhaps you understood it to mean something other than the teacher in- tended. If so, the teacher may give you credit for seeing another perspective. 3. Remember: Neatness counts! Write your answer clearly, so the reader will be able to decode your writing and understand your ideas. Without clearly written words your chances of a good grade are severely diminished. Write or print clearly, using a dark- colored erasable ball point pen. Avoid crossing out words or sentences, and do not smudge your paper. 4. Verb alert Read each essay question with the intent to identify the verbs or words that give you direction. These are the verbs that describe the task you are ex- pected to complete. Circle the direction verbs in the question to make sure that you are focusing on the desired task. 5. Use the principles of good English composition Form a clear thesis statement (statement of purpose) and place it as near to the beginning as possible. Provide supporting issues to back up the main concept you present. Underline or highlight the main and supporting issues. Examples will improve your answers and set them apart from other students’ answers. Remember to save some space for a brief but adequate summary. 44Additional Types of Test Items Additional Types of Test Items Additional Types of Test Items Additional Types of Test Items Additional Types of Test Items Advantages in Using Problem Solving Problem Solving Items An essay is not the only form of a subjective test item. Another ! Minimize guessing by requiring form is the problem solving or computational exam question. Such the students to provide an origi- items present the student with a problem situation or task and re- nal response rather than to select quire a demonstration of work procedures. Problem solving is from several alternatives. classified as subjective due to the procedures used to score item responses. Instructors can assign full or partial credit to either cor- ! Easier to construct than are mul- rect or incorrect solutions depending on the quality and kind of tiple choice or matching items. work procedures presented. An example of a problem solving test item follows: ! Can most appropriately measure objectives which focus on the It was calculated that 75 men could complete a strip on a ability to apply skills or knowl- new highway in 70 days. When work was scheduled to edge in the solution of problems. commence, it was found necessary to send 25 men on an- other road project. How many days longer will it take to !Can measure an extensive complete the strip? Show your work for full or partial credit. amount of content or objectives. Suggestions for Writing Problem Solving Test Items Suggestions for Writing Problem Solving Test Items Suggestions for Writing Problem Solving Test Items Suggestions for Writing Problem Solving Test Items Suggestions for Writing Problem Solving Test Items Limitations in Using 1. Provide directions which clearly inform the student Problem Solving Items of the type of response called for. ! Require an extensive amount of Undesirable: instructor time to read/grade. An American tourist in Paris finds that he weighs 70 kilograms. ! Subject to scorer bias when par- When he left the United States he weighed 144 pounds. What tial credit is given. was his net change in weight? Desirable: Remember… An American tourist in Paris finds that he weighs 70 kilograms. When he left the United States he weighed 144 pounds. What ! State in the directions whether or was his net weight change in pounds? not the student must show the work procedures for full or par- 2. Separate item parts and indicate their point values. tial credit. A man leaves his home and drives to a convention at an aver- ! Ask questions on which experts age rate of 50 miles per hour. Upon arrival, he finds a telegram could agree that one solution and advising him to return at once. He catches a plane that takes one or more work procedures are him back at an average rate of 300 miles per hour. better than others. Desirable: If the total traveling time was 1¾ hours: ! Work through each problem be- (1) How long did it take him to fly back? (1 pt.) fore classroom administration to (2) How far from his home was the convention? (1 pt.) double check accuracy. Show your work for full or partial credit. 45Authentic Assessments Authentic Assessments Authentic Assessments Authentic Assessments Authentic Assessments How well do multiple choice tests really evaluate student under- Authentic assessment standing and achievement? Many educators believe that there is a values the learning process more effective assessment alternative. These teachers use testing as much as the finished strategies that do not focus entirely on recalling facts. Instead, they product. ask students to demonstrate skills and concepts they have learned. This strategy is called authentic assessment. Examples… Authentic assessment aims to evaluate students’ abilities in ‘real- world’ contexts. In other words, students learn how to apply their In authentic assessment, students: skills to authentic tasks and projects. Authentic assessment goes ! do science experiments beyond rote learning and passive test-taking. Instead, it focuses on ! conduct social-science research students’ analytical skills; ability to integrate what they learn; cre- ! write stories and reports ativity; ability to work collaboratively; and written and oral expres- ! read and interpret literature sion skills. ! solve math problems that have real-world applications Use Authentic Assessment For… Use Authentic Assessment For… Use Authentic Assessment For… Use Authentic Assessment For… Use Authentic Assessment For… Advantages in Using Authentic Assessment 1. Performance Tests ! Can measure objectives related to the ability of the students to Performance tests assess students’ ability to use skills in a vari- apply skills or knowledge in real ety of authentic contexts. They frequently require students to life situations. work collaboratively and to apply skills and concepts to solve complex problems. Short- and long-term tasks may include: ! Provide a degree of test validity ! writing, revising, and presenting a report to the class not possible with standard paper ! conducting a week-long science experiment and analyzing and pencil test items. the results ! working with a team to prepare a classroom debate ! Useful for measuring objectives in the psychomotor domain. In theory, a performance test could be constructed for any skill and real life situation. In practice, most performance tests have Limitations in Using been developed for the assessment of vocational, managerial, Authentic Assessment administrative, leadership, communication, interpersonal and physical education skills in various simulated situations. ! Difficult and time consuming to construct. Suggestions for Writing Performance Test Items ! Primarily used for testing stu- ! Prepare items that elicit measurable behavior dents individually and not for ! Clearly identify and explain the simulated situation testing groups. Consequently, they are relatively costly, time ! Make the simulated situation as life-like as possible consuming, and inconvenient ! Directions should indicate the type of response called for forms of testing. ! Clearly state time and activity limitations in the directions 46Use Authentic Assessment For… Use Authentic Assessment For… Use Authentic Assessment For… Use Authentic Assessment For… Use Authentic Assessment For… (continued) 2. Short Investigations Many teachers use short investigations to assess how well students have mastered basic concepts and skills. Most short investigations begin with a stimulus: a math problem, cartoon, map, or excerpt from a primary source. The teacher may ask students to interpret, describe, calculate, explain, or predict. These investigations may use enhanced multiple choice questions. Or they may use concept mapping, a technique that assesses how well stu- dents understand relationships among concepts. 3. Open Response Questions Open response questions, like short investigations, present students with a stimulus and ask them to respond. Responses include: ! a brief written or oral answer ! a mathematical solution ! a drawing ! a diagram, chart, or graph 4. Portfolios A portfolio documents learning over time. This type of authentic assess- ment is a purposeful collection of work that shows the achievement or growth of a student. A portfolio is not a specific test but rather a cumulative collec- tion of a student’s work. Students decide what examples to include that characterize their growth and accomplishment over the term. While most common in composition classes, portfolios are beginning to be used in other disciplines to provide a fuller picture of a students' achievement. This long-term perspective accounts for student improvement and teaches students the value of self-assessment, editing, and revision. A student port- folio may include: ! journal entries and reflective writing For additional information… ! peer reviews Using Portfolios for ! artwork, diagrams, charts, and graphs Authentic Assessment ! group reports For a free copy contact… ! student notes and outlines Kansas Curriculum Center ! rough drafts and polished writing (785) 231-1010 x1534 ! electronic, video, and/or digital items esperanza.root@washburn.edu 47Grading Authentic Assessments Grading Authentic Assessments Grading Authentic Assessments Grading Authentic Assessments Grading Authentic Assessments Many educators find that authentic assessment is most successful A rubric enhances the when students know just what is expected. For this reason, teach- quality of authentic ers should clearly define standards and expectations. Educators assessment… use rubrics, or established sets of criteria, to assess a student's work. Advantages in Using A rubric is a scoring guide that seeks to evaluate a student’s perfor- Rubrics mance based on the sum of a full range of criteria rather than a single numerical score. Rubrics can be created in a variety of forms ! Many experts believe that rubrics and levels of complexity, however, they all contain common fea- improve students’ end products tures which: and therefore increase learning. ! focus on measuring a stated objective (performance, behavior, ! When teachers evaluate papers or quality) or projects, they know implicitly ! use a range to rate performance what makes a good final prod- ! contain specific performance characteristics arranged in levels uct and why. indicating the degree to which a standard has been met ! When students receive rubrics Rubrics can be created for any content area including math, sci- beforehand, they understand ence, history, writing, foreign languages, drama, art, music, and how they will be evaluated and even cooking! Once developed, they can be modified easily for can prepare accordingly. various grade levels. The following information presents general guidelines for developing a rubric. To Illustrate the various steps a ! Developing a grid and making it sample rubric is used. This rubric was created by a group of post- available as a tool for students’ graduate education students at the University of San Francisco, but use will provide a frame of refer- could be developed easily by a group of elementary students. ence for the self-assessment of the quality of their work. Steps in Rubric Development Steps in Rubric Development Steps in Rubric Development Steps in Rubric Development Steps in Rubric Development 1. Determine Learning Outcomes Once a Rubric is ! Determine which concepts, skills, or performance standards Created… you are assessing An established rubric can be used ! List the concepts and rewrite them into statements which re- or modified and applied to many flect both cognitive and performance components activities. For example, the stan- ! Identify the most important words within the concepts or skills dards for excellence in a writing being assessed in the task rubric remain constant throughout Chocolate Chip Cookie Rubric the school year; what does change The cookie elements the students chose to judge were: is students’ competence and your ! Number of chocolate chips teaching strategy. Because the es- ! Texture sentials remain constant, it is not ! Color necessary to create a completely ! Taste new rubric for every activity. ! Richness (flavor) 482. Determine Measurable Criteria Terms to Use in Measuring Range/Scoring Levels ! On the basis of the purpose of the task, determine the num- ber of points to be used for the rubric (example: 4-point scale ! Needs Improvement or 6-point scale) Satisfactory ! Starting with the desired performance, determine the descrip- Good tion for each score remembering to use the importance of Exemplary each element of the task or performance to determine the ! Beginning score or level of the rubric Developing Accomplished Chocolate Chip Cookie Rubric Exemplary The students developed a 4-point scale with the following descriptions: ! Needs work ! Delicious (4) Good ! Good (3) Excellent ! Needs improvement (2) ! Novice ! Poor (1) Apprentice Proficient The measurable criteria for each point of the scale follows: Distinguished 4—Delicious: ! Chocolate chip in every bite ! Numeric scale ranging from ! Chewy 1 to 5, for example ! Golden brown ! Home-baked taste ! Rich, creamy (high-fat flavor) Concept Words That 3—Good: Convey Various Degrees ! Chocolate chips in about 75 percent of the bites taken of Performance ! Chewy in the middle, but crispy on the edges ! Either brown from overcooking, or light from being 25 ! Presence to absence percent raw ! Complete to incomplete ! Quality store-bought taste (medium-fat content) ! Many to some to none ! Major to minor 2—Needs Improvement: ! Consistent to inconsistent ! Chocolate chips in 50 percent of the bites taken ! Frequency: always to generally to ! Texture is either crispy from overcooking or does not hold sometimes to rarely together because it is at least 50 percent uncooked ! Color is either dark brown from overcooking or light from undercooking As students become familiar with ! Tasteless (low-fat content) rubrics, they can assist in the 1—Poor: rubric design process. This ! Too few or too many chocolate chips involvement empowers the ! Texture resembles a dog biscuit students and as a result, their ! Burned learning becomes more focused ! Store-bought flavor with a preservative aftertaste—stale, and self-directed. Authentic hard, chalky nonfat contents assessment, therefore, blurs the lines between teaching, learning, and assessment. 49Steps in Rubric Development Steps in Rubric Development Steps in Rubric Development Steps in Rubric Development Steps in Rubric Development (continued) 3. Develop a Grid 2-Needs Criteria 4-Delicious 1-Poor 3-Good Improvement Too few or Number Chocolate chip Chips in about Chocolate in too many of chips in every bite 75% of bites 50% of bites chips Texture either Chewy in Texture crispy/crunchy Texture Chewy middle, crisp resembles a or 50% on edges dog biscuit uncooked Either dark Either brown brown from from overcook- Golden brown overcooking or Burned Color ing or light light from from being undercooking 25% raw Store-bought flavor, preservative Home-baked Quality store- Taste Tasteless aftertaste— taste bought taste stale, hard, chalky Rich, creamy, Medium fat Low-fat Nonfat high-fat Richness contents contents contents content 4. Compare Student Work to the Rubric ! Assign a rating to the various criteria you have identified as important. ! Revise the rubric descriptions based on performance elements reflected by the student work that you did not capture in your original rubric ! Rethink your scale: Does the number of points differentiate enough be- tween types of student work to satisfy you? ! Adjust the scale if necessary. Reassess student work and score it against the developing rubric. To assist in the initial development of a rubric, sample criteria (on a 5-0 point score range) are presented on the next page. 50Criteria for Scoring ! This is the highest rating ! The student is extremely knowledgeable about the topic ! The student demonstrates in-depth understanding of important ideas ! The student shows a depth of understanding of important relationships 5 ! The answer is fully developed and includes specific facts or examples ! The answer is organized around big ideas, major concepts/principles ! The response is exemplary, detailed and clear ! The student is knowledgeable about the topic ! The student has a good understanding of the topic ! The student includes some of the important ideas related to the topic ! The student shows a good understanding of the important relationships 4 ! The answer demonstrates includes adequate supporting facts or examples ! The answer demonstrates some organization around big ideas, major concepts/principles ! The response is good, has some detail, and is clear ! This is the middle score of the scale ! The student demonstrates some know ledge and understanding of the topic. The overall answer is OK but may show apparent gaps in his/her understanding and knowledge ! The student includes some of the important ideas related to the topic ! The student shows some but limited understanding of the relationships 3 ! The answer demonstrates satisfactory development of ideas and includes some supporting facts or examples ! The response is satisfactory, containing some detail, but the answer may be vague or not well developed and may include misconceptions or some inaccurate information ! The student has little knowledge or understanding of the topic ! The student does not develop the ideas or deal with the relationships among the ideas ! The response contains misconceptions or inaccurate information 2 ! The student may rely heavily on the group activity ! The response is poor and lacks clarity ! The student shows no knowledge or understanding of the topic. ! The student either: (1) writes about the topic using irrelevant or inaccurate information (2) recalls the steps of the Group Activity in Part II of the performance 1 assessment, adding no new or relevant information and showing no understanding of how the activity relates to the general topic. ! The student either: (1) left the answer blank (2) wrote about a different topic 0 (3) wrote “I don’t know” 51Etc…Etc…Etc… Etc…Etc…Etc… Etc…Etc…Etc… Etc…Etc…Etc… Etc…Etc…Etc… Tips on Test Construction Tips on Test Construction Tips on Test Construction Tips on Test Construction Tips on Test Construction Remember… 1. Assess information indicative of the material stressed in class, not trivial information Purpose of Testing 2. Have students submit 1 or 2 test questions and give extra credit for appropriate questions. Have them write a question with a ! To provide a record for assign- correct answer and source ing grades 3. To determine how much time the student will need to take the test use the following: ! To provide a learning experience ! 30 seconds per true-false item for students ! 60 seconds per multiple choice item ! 120 seconds per short answer item ! To motivate students to learn ! 10-15 minutes per essay question ! 5 to 10 minutes to review the work ! To communicate to students ! Or, allow triple the amount of time it takes you to their level of understanding of complete the exam the course objectives and serve 4. Select items that at least 50 to 70% of the students can cor- as a guide for further study rectly answer, or are of average difficulty 5. In terms of test reliability, longer tests are considered more re- ! When utilizing pretests, feed- liable than shorter tests back is provided regarding the 6. Be aware that many of the test banks and/or reviews in text- knowledge students bring to the books rarely assess higher levels of learning content ! To assess how well students are Test Layout Tips Test Layout Tips Test Layout Tips Test Layout Tips Test Layout Tips achieving the stated goals and course objectives 1. Include simple, succinct directions to include the following: ! How to record answers if they are not to write on the ! To provide the instructor with an exam opportunity to reinforce the ! Whether or not to show work on problems stated objectives and highlight ! The point value for different items what is important for students to ! Directions on how to use an answer sheet if provided remember 2. Avoid splitting a test item between two different pages 3. Leave the appropriate amount of space for each item 3. Leave wide enough margins for your comments, points, etc. 4. Group similar items together 5. If it is a large exam, it might be worthwhile to group items according to content as well 6. Leave space for the students name if they write on the exam. 7. Start with your easiest items in each section 52Returning Tests and Giving Students Returning Tests and Giving Students Returning Tests and Giving Students Returning Tests and Giving Students Returning Tests and Giving Students Feedback Regarding Tests Feedback Regarding Tests Feedback Regarding Tests Feedback Regarding Tests Feedback Regarding Tests 1. Return exams promptly. If this is not possible, post a corrected Have Students Complete an copy immediately after the exam Exam Evaluation 2. When you return a test make sure the score is not showing (turn the test over; put the score on the last page) Include some or all of the following: 3. Give feedback to the class as a whole regarding the following: ! Items most missed ! How well did the exam questions ! Mistakes most frequently made reflect the content and what was ! What was done particularly well emphasized? 4. When going over the test with the class, ask the students to ! What questions challenged you refer to their class notes (For example, “Look back at your notes to think? on 12-5-01. What do you have regarding this topic?”) ! Which questions seemed like 5. Do not respond to specific questions regarding the details of trick questions and why? an individual students answer. ! How difficult did you find the 6. Consider having students prepare their case in writing if they exam? want you to give them credit for a question ! How much time did you spend 7. Consider full or partial credit for valid arguments. studying for the exam? 8. Ask students to come up with specific questions versus “Why ! Were you clear as to what the is my test score so low?” questions were asking? List the numbers of those questions you 9. When giving one-on-one feedback, do not overwhelm a stu- were unclear about. dent whose performance was overall poor with so much infor- mation that they do not know where to begin. ! Are you satisfied with your an- swers to the questions? 10. Let students know when they have improved, even if it did not result in extremely high marks. ! What grade would you assign to this test? 11. Instead of explaining to a student why they missed a question, ask them to “think out loud”. In other words, have them an- swer the question and tell you out loud their thinking process. 12. To account for missed tests/quizzes you might want to drop the lowest quiz score or double the highest quiz score. 53Alternative Testing Modes Alternative Testing Modes Alternative Testing Modes Alternative Testing Modes Alternative Testing Modes 1. Take-Home Tests Take-home tests allow students to work at their own pace with access to books and materials. Take-home tests also permit longer and more involved questions, without sacrificing valuable class time for exams. ! Problem sets, short answers, and essays are the most appropriate kinds of take-home exams. ! The instructor should avoid designing a take-home exam that is too diffi- cult or an exam that does not include limits on the number of words or time spent. ! The take-home test should have explicit instructions on what the students can and cannot do: for example, are they allowed to talk to other students about their answers? ! A variation of a take-home test is to give the topics in advance but ask the students to write their answers in class. Some instructors hand out ten or twelve questions the week before an exam and announce that three of those questions will appear on the exam. 2. Open-Book Tests Open-book tests simulate the situations professionals face every day, when they use resources to solve problems, prepare reports, or write memos. ! Open-book tests tend to be inappropriate in introductory courses in which facts must be learned or skills thoroughly mastered if the student is to progress to more complicated concepts and techniques in advanced courses. ! On an open-book test, students who are lacking basic knowledge may waste much of their time consulting their references rather than writing. ! Open-book tests appear to reduce stress, but research shows that students do not necessarily perform significantly better on open-book tests. ! Open-book tests seem to reduce students’ motivation to study. A compro- mise between open- and closed-book testing is to let students bring an index card or one page of notes to the exam or to distribute appropriate reference material such as equations or formulas as part of the test. 54Creating Fair Tests Creating Fair Tests Creating Fair Tests Creating Fair Tests Creating Fair Tests A number of decisions and Testing Fairly* and Testing Fairly* and Testing Fairly* and Testing Fairly* and Testing Fairly* must be made about testing Many students with and without identified disabilities need sup- accommodations… port when taking tests. The type and extent of adaptations for fair test administration will vary from student to student and, possibly, ! How much and what kind of help from subject to subject for the same student. In addition, as the will be given? student gains skills, fewer accommodations may be needed. ! Who will give the help (e.g., gen- A number of possible test administration adaptations are listed be- eral or special education teacher, low. Educators should choose the best combinations of strategies para-educator, or volunteer)? for student success based on individual needs. ! Where will the student be tested ! Provide oral and/or written time checks during the test and pro- (e.g., in the regular classroom, a vide breaks during long tests resource or conference room, the ! Give oral interpretation of directions library, or the cafeteria)? ! Confirm correct responses with a nod, thumbs up, or correct mark on the page ! When will the test be given ( e.g., ! Explain the meaning of key vocabulary words time of day, in one sitting or bro- ! Provide additional examples of the expected answer ken into short time periods, dur- ! Trigger associations: “Remember when we...” ing the regularly scheduled class, ! Use a student-generated reference sheet (i.e., a legitimate after school, during recess, with “cheat sheet”) or without additional time, etc.)? ! Review just prior to the test ! Display reference charts in the classroom ! What adaptations should be ! Excuse a student from answering specified test questions or made depending on the student’s sections (i.e., omit the essay or the short answer) disability, the subject, the type of ! Require fewer answers (evens or odds only when appropriate) test, and the student’s increasing ! Remove the pressure to rush through a test by agreeing to base skill in reading, processing, and the grade on the number of correct answers out of the total num- writing independently? ber of questions answered ! Provide a word bank/outline ! Adaptations must be individual- ! Read the test orally ized and kept private between ! Allow use of calculators, computers, dictionaries, electronic spell- teachers and students. Adapta- checkers, and/or tape recorders tions should parallel the accom- ! Allow enough time for completion of tests in one sitting or break modations made during instruc- the testing into two days tion. For instance, if a student ! Give a re-test commonly uses taped books, ! Avoid adding additional pressure during testing by stating nega- then tests should be presented tive consequences of a poor score orally. If a student uses a calcu- ! Allow students to tape record answers to essay questions or to lator for completing daily assign- outline the answer ments, then the calculator should be allowed during tests. *Information from: Including All Students: A General Educator's Guide to Teaching a Diverse Student Population For a free copy contact: Kansas Curriculum Center, (785) 231-1010 x1534, zzroot@washburn.edu 55I’d like to use essay tests, but... I’d like to use essay tests, but... I’d like to use essay tests, but... I’d like to use essay tests, but... I’d like to use essay tests, but... Marilla Svinicki, University of Texas at Austin The Professional & Organizational Development Network in Higher Education (continued from page 39) If we want the students to be able to deal with the complex nature of essay tests and other forms of spontaneous writing, there are some things we can do in our instruction that will prepare them more adequately. 1. Help them think differently about the material Students are conditioned from an early age to think in terms of discrete facts and “correct” answers rather than looking for the relationships which are characteristic of essay answers. One of the first steps toward improved essay answers is to adopt a different perspective on the nature of what is to be learned from the material presented and read. To help students think about the material differently, the instructor can: ! Encourage students to integrate material from class to class and unit to unit. For example, have the students answer some of the questions listed below each time they begin a new topic: —How does this topic compare with/relate to what has gone before? —How is it different? How is it similar? —Why is it included in the course? Why at this point? —What are its main points, its strengths, its weaknesses? —How does it apply to the overall goal of the course? ! Have them write their own sample essay questions for each lecture or reading assignment and then in class. Discuss those that most closely par- allel what you would ask. ! Explain the levels of cognitive complexity (such as Bloom’s taxonomy) which might be expected of them in the course and differentiate between knowledge of facts and ability to analyze and critique material. ! Emphasize process during classtime itself, so that the students begin to understand how conclusions are reached rather than focusing on the con- clusions alone. 2. Help them study the material differently Studying for essay exams is much different from studying for objective ex- ams. Instructors should encourage students to: ! Create outlines of readings and lecture notes which emphasize the rela- tionships among the ideas. Paraphrase or create an executive summary for each reading or lecture. ! Draw concept maps, which are visual diagrams of how terms, principles, and ideas interconnect. (Continued on next page…) 563. Help them write structurally sound answers To help students compile the information they have learned into answers which are written more effectively and efficiently, an instructor can: ! Provide a list of key words used in essay questions and what they imply in terms of answer content and structure. (See page 41 of this document.) ! Give students opportunities to practice writing essay answers in class and discussing the structure of the answers. ! Assign brief out-of-class essay questions with which to practice and pro- vide individual feedback on the writing. You may wish to develop a feed- back phrase sheet, which lists your most commonly used comments and an extended description of what that comment means. ! Give the students an opportunity to grade an essay answer using the sys- tem (rubric) you normally use so that they will understand how they are being evaluated. ! Provide examples of good and poor answers to essay questions with an explanation of why they are evaluated that way. 4. Help them learn time management techniques Here are some examples of efficient time management techniques that the student could possibly benefit from in completing an essay exam: ! Scanning all the items and parceling out an appropriate amount of time to spend on each according to weight or importance ! Spending a few minutes outlining an answer before writing (the teacher could possibly give some credit for content which appears on an outline, but was not included in the answer due to time constraints) ! Having a checklist for quickly evaluating answers before completing the exam (such as “did you answer the question?” “are the transitions clear?” “is evidence provided for each assertion?” and so on). 5. Why should we bother? There is actually an additional selfish motive for improving students’ essay writing skills: it makes the grading process much easier. If students learn how to read and interpret the structure of an essay question, they can create an answer that is comprehensive and well-organized. The task of grading those essay answers becomes less one of interpretation and more one of evaluation. 57Test Administration Assignment Test Administration Assignment Test Administration Assignment Test Administration Assignment Test Administration Assignment Read Through the Following Description of a Teacher Giving a Test 1. Students enter the classroom. Before the students have a chance to put away their things, the teacher announces that they will be having a test. No notice has been given of the test. In response to student complaints about the test, the teacher responds “it will show who is really paying attention in class and keep- ing up with the reading.” 2. Before the students have time to remove their books, notebooks, etc. from their desks, the teacher starts handing out the quiz. (Some books, etc. remain on the desks.) Once the first students receive the test, they start busily taking it, while the teacher is handing out the rest. The teacher announces that the stu- dents will have twenty-five minutes to take the fifty-item test. 3. One minute into the test, one of the students raises her hand and asks whether to mark the correct response to the multiple choice item by circling or placing a check beside the correct response. Later, someone asks how to respond to the true-false items. 4. The desks in the room are close together. During the test, Billy glances over to Juan’s paper and sees what his answers are. Billy sees about five answers. 5. After about twenty minutes, two students have completed the test and start rustling papers and whispering. Some students complain about the noise. 6. At the end of the time, the teacher announces that time is up. Some students complain that they did not know how much time was left and that they are in the middle of answering an item. The teacher collects the papers anyway. 7. While scoring the test, the teacher notices that some students did not have the last page. For those students, the teacher decides to score the test using only 40 items instead of 50. List problems associated with the way Indicate ways the teacher could have the teacher administered the test. avoided the problems. _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ _________________________________ 58Cognitive Domain Guide Cognitive Domain Guide Cognitive Domain Guide Cognitive Domain Guide Cognitive Domain Guide Use this chart when the major topic or task primarily involves the acquisition and processing of knowledge. Then use these key words in objectives, Level If the student must… assignments and evaluations …recall or recognize this define list recite knowledge; giving it back in describe match recognize Knowing nearly the same form as it identify name select was received. label recall state …demonstrate an under- standing of this knowledge, associate distinguish paraphrase seeing relationships and tell- compare explain restate illustrate Comprehending ing in their own words what discriminate match translate it means. apply generalize predict compute illustrate produce convert infer relate demonstrate interpret …use this knowledge in a rewrite Applying dramatize modify concrete situation other than show how employ operate in which it was learned. solve estimate perform transform extend practice use analyze diagram …analyze or break down this break down differentiate separate knowledge into its essential categorize distinguish separate out parts, and differentiate be- Analyzing compare find show how tween facts, opinions, as- contrast isolate subdivide sumptions, hypotheses and detect outline trace the logic conclusions. determine point out arrange a new devise …produce something uni- assemble formulate plan que and original from this combine generalize problem solve knowledge by synthesizing Synthesizing construct harmonize reconcile or combining the elements create integrate relate from an analysis into a new design organize summarize structure or organization. develop from parallel access evaluate select based on appraise judge …form judgments about the support conclude Evaluating justify value or worth of this test critique rank by knowledge. validate defend refute 59Affective Domain Guide Affective Domain Guide Affective Domain Guide Affective Domain Guide Affective Domain Guide Use this chart when the major topic or task is primarily concerned with acquiring new attitudes, values or beliefs. Then use these key words in objectives, Level If the student must… assignments and evaluations be alert to perceive existence be aware of …receive information about receive information on be sensitive to or give attention to this new Receiving take notes on experience attitude, value or belief. take notice of listen to willingly attend look at discuss openly allow other to …participate in, or react to answer questions on enjoy doing Responding this new attitude, value or contribute to participate in belief in a positive manner. cooperate with reply to dialog on respect those who accept as right decide based on …show some definite in- accept as true indicate agreement volvement in or commitment affirm belief/trust in Valuing influence others to this new attitude, value or associate himself with justify based on belief. assume as true seek out more detail consider valuable …integrate this new attitude, advocate value or belief, with the ex- prioritize based on integrate into life isting organization of atti- Organizing persuade others judge based on tudes, values and beliefs, so systematize place in value system that it has a position of prior- ity and advocacy. act based on know by others as …fully internalize this new consistently carry out characterized by attitude, value or belief so Characterizing consistently practice sacrifice for that it consistently character- fully internalize view life based on izes thought and action. 60Bibliography Bibliography Bibliography Bibliography Bibliography "The Advantages of Rubrics," The Learning Network http://www.teachervision.com/lesson-plans/lesson-4522.html "Assessing Student Performance," Ohio State Office of Faculty and TA Development, Ohio State University, Columbus, OH http://www.osu.edu/education/ftad/Publications/TeachingHandbook/chap-6.html "Authentic Assessment Overview," Pearson Education Development Group http://www.teachervision.com/lesson-plans/lesson-4911.html Border, Barbara. "The Status of Alternative Assessments Through the 1990s: Performance and Authentic Assessments in Relation to Vocational-Technical Education, Technical Skills, Workplace Skills, and Related Academic Skills" (1998) V-TECS "Common Characteristics of Authentic Assessment," Aurbach & Associates http://www.aurbach.com/alt_assess.html "Creating Better Student Assessments," The U.S. Department of Education, Improving America's Schools http://www.ed.gov/pubs/IASA/newsletters/assess/pt1.html Davis, Barbara G. "Quizzes, Tests, and Exams," Tools for Teachers, Jossey-Bass (1999) http://uga.berkeley.edu/sled/bdg/quizzes.html "Designing Test Questions," Walker Teaching Resource Center, The University of Ten- nessee at Chattanooga http://www.utc.edu/Teaching-Resource-Center/test-questions.html Dewey, Russell A. "Writing Multiple Choice Items Which Require Comprehension," Georgia Southern University, Statesboro, GA http://www.psywww.com/selfquiz/aboutq.htm "Essay Tests," Office of Teaching Effectiveness, University of Colorado at Denver http://thunderl.cudenver.edu/OTE/nn/ "General Tips For Writing Essay Prompts," Online Writing Lab, Rogue Community College, Grant's Pass, OR http://www2.rogue.cc.or.us/OWL/teaching/prompt.htm "Guidelines for Rubric Development," EdWeb, San Diego State University http://edweb.sdsu.edu/triton/july/rubrics/Rubric_Guidelines.html Hitchcock, David. "How Can I Use Multiple Choice Tests for More Than Just Rote Recall of Facts?" http://www.mcmaster.ca/learning/teaching_tips/tip14.htm "How to Write Tests: Matching Questions," University of Lethbridge, Alberta, Canada http://www.edu.uleth.ca/courses/ed3604/HTWT.html#mat (Continued on next page…) 61"Improving Your Test Questions," Division of Measurement and Evaluation, Office of Instructional Resources, University of Illinois at Urbana-Champaign http://www.oir.uiuc.edu/dme/Exams/ITQ.html#sec23 "Including All Students" Kansas Curriculum Center, Washburn University Jacobs, Lucy C. "How to Write Better Tests," Indiana University, Bureau of Education Studies & Testing http://www.indiana.edu/~best/write_better_tests.shtml Johnson, John A. "Levels of Understanding Assessed by Multiple Choice Questions" http://www.personal.psu.edu/faculty/j/5/j5j/psy002/MCQ.html Kehoe, Jerard. "How to Write Multiple Choice Questions," QuizPlease http://www.robmccormack.com/qphelp/qphowto5.htm "Lack of Training on Standards and Tests". Education Week on the Web http://www.edweek.org/sreports/qc01/articles/qc01chart.cfm?slug=17tools-c1.h20 "The Multiple Choice Exam," University of Victoria, British Columbia Learning Skills Program http://www.coun/uvic.ca/learn/program/hndouts/multicho.html "Performance Assessment," Education Research Consumer Guide, Office of Educational Research and Improvement (OERI), US Department of Education http://www.ed.gov/offices/OERI/ "Performance Assessment Scoring Rubric," National Center for Research on Education, Standards, and Student Teaching (CRESST) http://www/cse.ucla/edu/CRESST/pages/Rubrics.htm "Principles of Good Practice for Assessing Student Learning," University of Montana at Bozeman, Office of Institutional Research http://www.montana.edu/aircj/assess/GoodPractices.html Rowe, Allan D. "Rubric Basics," http://www.rubrics.com: 19815/4DAction/w_start/1 Rudman, Herbert C. "Integrating Testing with Teaching," ERIC Clearinghouse on Assessment and Evaluation http://ericae.net/pare/getvn.asp?v=1&n=6 "Standards for Teacher Competence in Educational Assessment of Students," Buros Institute, University of Nebraska, Lincoln http://www.unl.edu/buros/article3.html "Student Evaluation of Test Item Quality," University of Illinois at Urbana - Champaign http://www.oir.uiuc.edu/dme/ices_cat/NewICESfaq.html "Summary Data on Teacher Effectiveness, Teacher Quality, and Teacher Qualifications," National Council for Accreditation of Teacher Education (NCATE), 2000 http://www.ncate.org/resources/factsheettq.htm Svinicki, Marilla. "I'd Like to Use Essay Tests, But…" The Professional & Organizational Development Network in Higher Education http://www.ucet.ufl.edu/ProgramService/topic6-2.htm (Continued on next page…) 62"Test Construction," Teaching@Indiana, Indiana University http://www.indiana.edu/~handbook/evalsp-t.htm "Test Taking Strategies," Lawrence University, Appleton, WI http://www.lawrence.edu/dept/student_acad/college_place/before_the_exam.html "Test Taking Strategies," Marquette University, Milwaukee, WI http://www.eop.mu.edu/study/ "Test Techniques," The Center for Teaching Excellence, Lansing Community College, Lansing, MI http://www.lansing.cc.mi.us/~cte/workshops/testingtechniques/tt9_references.html "True-False Item Construction," College of Education, Texas A&M University http://empowermentzone.tamu.edu/true_false_item/page2.html "Writing Matching Test Items," Alabama Professional Development Modules http://web.utk.edu/~mccay/apdm/mod3b/mod3b_a.html "Writing Test Items," Michigan State University http://www.msu.edu/dept/soweb/writitem.html "Writing True-False Questions and Evaluating Responses," Alabama Professional Development Modules http://web.utk.edu/~mccay/apdm/mod3at-f_b.html "Writing True-False Items," Office of Measurement Services, University of Minnesota http://www.ucs.umn.edu/oms/truefalse.htmlx 63
sharer
Summaries
Free
Document Information
Category:
Summaries
User Name:
Dr.PeterCena
User Type:
Researcher
Country:
Swaziland
Uploaded Date:
02-07-2017