Skip to content

Case study: Multiple-choice questions and peer feedback with the Ultra Assignment Peer Feedback tool

Department of Health Sciences
David Graham

David Graham describes how he used the Ultra Assignment tool's peer review option to create a multiple-choice question (MCQ) development activity for students taking a first year Bioscience module.

Students were tasked with creating MCQs related to the contents of the module and then providing feedback to peers on the effectiveness of these questions. Once reviewed, David made the questions available to all students as a revision resource (with built in feedback) for the summative assessment (an in-person MCQ-based examination).

David provides a walkthrough of the process of setting up and managing the activity, and outlines the benefits of the activity as a means of enhancing learning through peer support and facilitating better preparation for the exam. He also presents the feedback from students which was generally very positive. Finally, he lays out his plans for developing the activity in future iterations of the module, including providing more guidance to support the process and enable a broader range of questions to be produced.

Watch their presentation:

Multiple-choice questions and peer feedback with the Ultra Assignment Peer Feedback tool (Panopto viewer) (7 mins 43 secs, UoY log-in required)

Transcript

Right then. So. Yeah. David Graham, I'm the lecturer in nursing in the Department of Health Sciences, and I used peer review tool within Ultra for the Bioscience module that occurs in year one of the program.

So the idea came from a course I did on Coursera, a MOOC, many years ago, where you did peer feedback. And what I saw from that was that the people who were attending the course were very diligent in the way they wrote their answers, and there was a lot of collegiality between the attendees of the course. I also saw that it was very efficient in terms of feedback for the people running the course, and it was something I've been wanting to do ever since, and it's only fairly recently that the peer feedback tool in Ultra has become available.

So my objectives were peer support evidence was that, you know, the students supporting each other shows signs of learning and collegiality, etc. My other ideas were, and we've done this in another module where students wrote their own multiple choice questions for each other. And, and that worked really well because to write good questions, you have to have a good knowledge of the subject. So students then have to read and that builds their revision. And then structuring the MCQs is also quite hard. So it helps them read the exam paper when they get it and understand what we're trying to ask when we ask an MCQ in that exam. There's also a certain level of efficiency which we need more and more, don't we, as staff are reduced. In terms of getting more feedback to students.

So in Bio1, what we did with Blackboard Ultra was this development of three MCQs. So students had to write three multiple choice questions and submit them into Blackboard Ultra. One week later, they then had to review three other students' work, and they had to comment on the structure of the question, the clarity of the question, how difficult they thought the question was, and any issues with the answers, you know, is there one answer that's obviously ridiculous or that kind of thing. And we did a little bit of prep with that in terms of how you write multiple choice, good multiple choice question papers. I should say the summative assessment for this was a multiple choice question paper - face to face.

This year, I also reviewed all 140 submissions just to see how students did. And it was about a day of my time to do that. I'm not intending doing that every time, but I just wanted to see how it was working.

So how to set it up? You create a new assessment, a new assignment in your, uh, blackboard page in the ultra page. And then in that assignment, you click, uh, the mark category under the mark category section of the assignment, you click the peer review toggle button. Underneath the peer review toggle button there is a select uh peer settings. And in there you set, uh, how many reviews the student has to do of other students' work. And you can do one or, uh, many more than one. Uh, I thought three was not unreasonable. If they'd done three questions, it's not a lot of work. Um, not loads of work. But then reviewing three other students' work didn't seem unreasonable. And that's what happened in the Coursera course that I did. I guess it was my experience.

This is what the page looks like to the student for this new assignment, it gives the instructions and what they actually have to do and then submit the work. Once they've submitted the work, you get the standard page, an assessment page so you can click into each of the students work and see what they've done. So this next page, I've blanked out the real student data in there so that we can protect the innocent. And I'll just whiz through what you can see on the page where you interact with this tool within, uh, blackboard. So there's the submission instructions at the top, and then, uh, they've got a mark in the top right hand corner. You have to put a mark in to get it released. It's that standard VLE feature. The way I used it this year was I marked it out of three, you got a point for doing one part of it. You got another point for doing another part of it, and then you got an extra point if the, the feedback that you gave was, was quite good. I kept it really simple. It didn't add to a summative mark at all. It was just a way of indicating what students had done.

The lecturer feedback is visible, and I went through each one and left a little bit of feedback for each student. And as I say, it took about a day to do that. The student submission is here on the main page, and their peer feedback is available to see in the right hand column there. And then that student's feedback to others is a is visible in the next section down. So you can look at what the student did in terms of their questions. But you can also look at how valuable their feedback was to other students so then my feedback would reflect that. So I would look at their questions and go, yeah, your questions look okay. And your feedback to other people also looks really constructive as well. That was quite useful.

So we then did a very quick survey of the students to find out what they thought. And this was a quick Likert question. Um, peer review exercise, how useful they found it. And as you can see, it was predominantly positive. The yellow section being the sat on the fence, neither agree or disagree. And then there was one student that put strongly disagree. And I think their comment was something along the lines of I just want previous exam papers so I know what I want to do. And I thought, oh yeah, that kind of defeats the object slightly. This is what student said the next two slides are that.

  • Helped me consolidate my knowledge.
  • Really enjoyed peer reviewing others work.
  • Found writing multiple choice questions difficult to begin with.
  • I thought this exercise was very helpful in not only understanding how questions may be laid out, but also with remembering the answers to certain topics.

And this is the next page:

  • I would like a mix of difficulty questions.
  • It was useful, but I wasn't always sure what to put for feedback and how to help.
  • It would be great to have a feature where we can thank the student who gave us feedback.
  • Would like to do this again.

So just picking up on some of those things for next year, what I would like to do is I would probably reflect on the way that I've given the instructions, and write slightly better instructions. Try and develop the range of questions that students provide. They all tended to write the same sort of questions, and we need to get a broader range of questions. Um, and then also do some peer support for colleagues who want to do this too.

The last thing to say is when we got the questions done, we then got the students to submit the questions via a Google form, the ones that they thought were the better questions of their questions. And that went straight into an Excel sheet or into Google Sheet. I was able to apply those to the VLE for students to actually do them as a quiz, and that worked really well as well.