The most difficult part of creating a Moodle quiz isn’t using Moodle. It is writing the questions. First, write the good questions. Then create the Moodle quiz.*
Test questions – regardless of delivery method – must meet two important criteria:
- They must have validity. This means they test what you say they’re testing. If you’re qualifying someone to do CPR, you must provide valid testing that truly measures a person’s ability to perform CPR in an emergency. You don’t want to be measuring a person’s ability to memorize an acronym or to perform a tracheotomy.
- They must be reliable. This means that if someone takes the test today, he’ll answer the same as he did yesterday – or will tomorrow. It also means that the person, his background, the time of day, the method of delivery, etc., will not affect the answers. The only thing that should alter the answers is the actual understanding of the subject.
Think of each of your questions as a thermometer. To be valid, it must measure temperature, not weight or decibels. To be reliable, it must read 32°C when it is in fact, 32°C, regardless of the relative humidity or other ambient conditions.
How do you create test questions that are both valid and reliable?
- Test the Topic
- Keep a Poker Face
- Mix It Up
Test the Topic, Not the Ability to Do Word Puzzles
Bad Question #1: Don’t confound things by asking two questions in one. Don’t get tricky. Don’t ask irrelevant questions. Don’t write questions like this one:
Which of the following statements is not true?
A. The grass is green and the sky is blue
B. Neither grass nor the sky is blue
C. The grass is green or the sky is blue, but not both
D. All of the above
E. None of the above
Seriously? I don’t even know the right answer to this question. I’m not even sure there is a right answer!
Better Question #1. If you’re teaching the science of color, you want to ask questions about why grass appears green or the sky appears blue. You don’t want to test the ability to reason through a maze of “not”, “or”, “and”, “all”, “none”…
What explains the appearance of a blue sky of varying shades during the day?
A. Scattering of sunlight by air molecules
B. Oxygen molecules are blue
C. The sun lightens the darkness of space, making it appear blue
D. The blue color is not real; it is an optical illusion
Better Question #2. If, on the other hand, you’re teaching logic, you’ll want to make each choice require the student to reason through it.
Which of the following statements is true? Check all that apply.
A. Grass is green and the sky is blue.
B. Grass is green or the sky is blue.
C. Grass is green and/or the sky is blue.
D. Neither grass nor the sky is blue.
E. Neither grass nor the sky is green.
Keep a Poker Face. Don’t Give Away the Right Answers.
Novice test writers often give away the correct answer by putting too much into it. It’s easy to get excited about a concept or just want to really get the point across. It’s hard to put that much effort into the wrong answers, so they are often silly afterthoughts. This reduces the validity of the test by making it obvious what the right answer is, even to someone who knows nothing about the subject.
Bad Question #2: Too long to be wrong…(and confounding by including so many terms in one choice)
Which one of the following is true?
A. Good leadership is a matter of opinion.
B. Poor leadership (no demonstration of values, inconsistent, poor decision making, unavailable or unapproachable) in the workplace can result in poor performance for the entire organization, at all levels.
C. It is difficult to define leadership.
D. Leadership is not necessary if the team is solid.
Bad Question #3: Silly wrong answers…
Choose three of the five core emotions:
E. Ice Cream
Better Question. The wrong answers should not only be plausible, but they should represent some of the most common misunderstandings surrounding the topic. This will provide you with true information about how well the students understood your lesson and gives you an opportunity to reinforce the concept with your feedback.
From the following list, select the five core emotions:
The letter “B” might be your favorite choice, but make sure it is not always the correct choice. Vary the position of the correct answers in your questions. Also, be grammatically consistent; don’t give the correct answer away by making it the only one that flows as a sentence with the question.
For more examples of how to improve the wording of a question, check out the Guide to Objective Tests at the Computer Aided Assessment Centre.
Mix It Up to Keep Them Thinking
Limit True/False questions. They can become hypnotic and don’t afford you with the ability to learn specifics of what the student doesn’t understand.
Open ended questions (fill in the blank, short answer, and essay) can be hard to grade. They should never be used unless just one person grades all submissions and that person is an expert in the field. The last thing you want is a non-expert searching for “key words” the way a computer would. That would be testing the student’s ability to win Buzz Word Bingo, not whether there was an understanding of the topic. After all, it takes true understanding to paraphrase.
Use multiple choice questions with varying formats. Use “check all that apply”, “check the one that does not apply”, “choose only one”, and even an introductory scenario with a few follow-on multiple choices. Use common misconceptions as wrong answers.
Be creative (not tricky!) with matching questions. Have more possible answers than questions (uneven left and right columns) so it doesn’t become a process of elimination. Include common misconceptions as possible matches.
Test Designer has a great reference page for what type of question to use, depending upon your intent.
Stay tuned for an upcoming post on the merits and pitfalls of shuffling questions and choices.
*For more on using Moodle quizzes and testing in a business environment:
- Demystifying Moodle Quiz Settings Part 1: Appearance
- Demystifying Moodle Quiz Settings Part 2: Strictness
- Demystifying Moodle Quiz Settings Part 3: Feedback
- Assessing the Effectiveness of Business Training