Thanks to the comments of Leah from my last post, I searched out (followed Leah's link) to a TedTalk by Martin Bush about different methods of evaluating multiple choice tests. I've requested the original journal article behind the premise of his talk, however the direct request can take a few days it seems, if even approved.
The TedTalk did a fantastic overview however. Essentially, you can use the same multiple choice test, but assign different values and schemes to the scoring. The concept here being to eliminate the guess-work, and improve the credibility of the test as an effective evaluation of knowledge. https://youtu.be/ACB2B2EdiXs Martin describes a few different methods I'll try to simplify here: Negative Marking: 3 marks for a correct response, -1 for incorrect a) wrong b) wrong X. Student would receive -1 for this question. c) correct d) wrong The theory here is that for guessing, there is now a 1/4 chance of scoring 3 marks, and 3/4 chanc of scoring -1, so these should equalize. This can however ultimately lower test averages, and nobody likes being graded in a negative environment. That said, my profession is very much like this, do well and save a life get 3 points, do poorly and not save the life -1. Subset Selection: You may select more than one answer. a) wrong b) wrong X -1 c) correct X +3 Student would receive 2 marks d) wrong a) wrong b) wrong X -1 Student would receive 1 mark c) correct X +3 d) wrong X -1 This marking scheme begins to demonstrate knowledge, particularly in the common case where the student is able to eliminate two options and is left with the classic 50:50, well now they may select both answers and achieve 2 marks. A concern I have here, how much time will students spend debating over what they're willing to risk on each question? Should I go for the 3 marks, or play it safe with 2? Will this increase test anxiety? Subset Selection Without Negative Marking: a) wrong b) wrong X c) correct X Student would receive 1/2 mark d) wrong a) wrong b) wrong X c) correct X Student would receive 1/3 mark d) wrong X a) wrong b) wrong c) correct X Student would receive 1 mark d) wrong a) wrong b) wrong c) correct Student would receive 1/4 mark d) wrong I find this to be a highly positive test, where one could answer nothing and still achieve 25%! This final method is still being studied to determine its effective quality, however the Subset Selection was presented to statistically match the profile of what a good test is meant to show, that increasing knowledge would yield increasing test scores. My initial thoughts are this would be easy to implement with current tests, however the grading would be significantly more time consuming. Last year my school switched from the "scan-tron" to "grade-cam", both utilize the bubble-sheets, however now we can mark instantly with a smartphone app or classroom camera. This technology would not accommodate this type of grading and the selection of multiple different answer patterns, which would mean marking 70 multiple choice questions for 30 students, where each answer must be interpreted, not just mindlessly checked to a defined answer key, and we don't have TA's! I have an opportunity to implement this in a few weeks with a smaller class size, if the other group teachers agree - a multiple choice test written in our smaller lab sections. This would also give me a great insight compared to traditional scores, and doing so in an easily manageable group of students. I need to spend some time playing with the scoring variability to determine if I prefer the Subset Selection or Subset without Negative Marking. I can appreciate how these methods were shown to lower test averages already just playing with some numbers, where students will likely achieve many half marks, but I do like this concept of working to eliminate some of the guesswork and better demonstrate their thought process of ruling out options. I wonder however, does not answering a question with the most correct incorrect option also display this? They didn't pick the right option, but they picked the next best one. I'd love to hear any thoughts or other variations people have tried with this, and if there are opinions both from a student or teacher perspective. If a multiple choice test is being utilized, is this a better option than the traditional scoring?
0 Comments
Leave a Reply. |
AuthorI teach Paramedicine at Cambrian College, in Sudbury, Ontario. I also continue to work as a paramedic, and ride bikes. This is my third semester in the PME program, and I look forward to learning with everyone! Archives
March 2017
Categories |