Case study: Developing an AI-powered tutor for a Masters level conversation analysis module.

Department of Language and Linguistic Science
Kobin Kendrick

Summary

Kobin Kendrick describes how he developed an AI-powered support tutor for a conversation analysis module. He created the tutor as a Google Gem and provided it with access to the module content and instructions on how to engage with students to help them to learn. Instructions focused on ensuring that the tutor would seek and use knowledge of where a student is at in the module to build responses that would be specific to the content of each week of study. They also set ground rules for the types of responses it should give including, for example, that it should not invent hypothetical examples.

Kobin described how he introduced and integrated the tutor within a module seminar to encourage critical approaches. Although he was not in a position to gather evaluation data from students, he felt that the tutor worked more effectively than a generic AI chat bot, grounding responses in the specifics of the module. His instructions are available in the session resources folder (Google drive).

Watch their presentation:

Developing an AI-powered tutor (Panopto viewer) (7 mins 09 secs, UoY log-in required)

Transcript

So this, uh, is an experiment that I conducted last semester with a module on conversation analysis, which is a module that looks at how people use language in everyday social interaction, but has a very technical vocabulary and technical approach to, uh, its subject matter. I should say that, um, for this occasion, I used AI to generate the slides that you'll be seeing. I thought it would be an appropriate use of AI, and also time is always short, but, um, so I decided I would develop a tutor for students in this module, and my thinking is that contact hours are short. Um, students often have questions that they maybe don't want to ask in class or that they're too shy to ask. Um, and I think that I will eventually, once we get the right model down, provide a very important resource for students to augment the human interaction that's involved in learning and teaching. Um, and so I wanted to try to address this by developing an AI tutor. When I started working on this, I realised very quickly that if you just ask a generic AI chat bot questions about a particular area of research or the subject matter of a module, it can give you really good answers, but sometimes it gives you really bad answers. And I realised that I could, by testing the different models, come to some understanding of where it had limitations and where it did things really well.

So it sometimes would confuse boundaries between my own field conversation analysis and neighbouring fields that aren't really part of what I teach. I wanted it to make that boundary really clear. Um, it also, if you talk to a generic chat bot, it doesn't know where the students are in the module. So if there's a progression of topics across weeks, it will start mixing up terminology from different weeks from straight and straight away. And that can be terribly confusing to students because they won't understand the material from week ten in week two. Also, some students ask very sophisticated questions and an AI tutor can engage them at a very high level, whereas others are at a much more basic level. And an AI can actually calibrate its responses very easily to, you know, the PhD level or the high school level. It can do both if it knows that that's what the user wants. Um, and generic chat bots often are not aware of their limitations. They're not self-aware of their limitations, but you can make them self-aware of their limitations. So I wanted to make a tutor, but I knew that there were some issues such as these.

So what I decided to do was to use Google Gemini, because that's what's integrated into University of York's, uh, infrastructure. And it also has this really nice feature called a gem. And a gem allows you to set up a custom prompt, a custom set of instructions, and load files into the chat bot before a user engages with it. So when a student starts to chat with this, that it already has a set of instructions that I've developed and it has access to the modules content. So I upload all of my slides for the lectures into Gemini so that when a student asks a question, it's grounded in the answers from the modules content itself. So it mixes in its general knowledge, but it prefers actually when you give it files to ground it's responses in those files. So they're constantly getting answers that are grounded in, you know, in week two, we did this in week three, you did this. And I made it temporally aware. So, um, one of the issues was, you know, it doesn't know what week the students are in, in the module. So the first question that the chat bot asks or the AI tutor asks is, what week are you in right now? And so the student could say A week two. And then it knows that it should only use concepts from weeks one and two. And it won't talk about weeks three or 4 or 5 and so on. And that works really well. It really kind of constrains itself, and I made it aware of its own limitations. So in conversation analysis we analysed little bits of conversation. And it's a method that uses naturalistic observation. So it has to be real interaction real data. And the AI likes to make up hypothetical data a lot. And it likes to, uh, speculate about, uh, things in the interaction in a way that isn't what we would normally teach them. So I made it not invent hypothetical examples. And if it does, use examples and analyse examples from the, uh, lecture materials, I have made it explicitly warn the students that its analyses could contain mistakes, and they need to examine it carefully and reflect on it. And so, you know, it's become very self aware that it's not very good at analysing data. It's great at like book learning and things like that. But its analytic processes are not really great. Um, but I think it's good for it to have those warnings for students.

Um, so what worked about this was, um, you know, I introduced it in a seminar exercise, so I tried to integrate it into into the module. I used it as, uh, with the seminar activity where I had students critically assess an AI generated analysis of a piece of data and compare that to the published analysis and a journal article of that same piece of data. So they could compare what, what an AI looks like versus what, uh, a human written analysis looks like. And I shared the prompt with students so that they could learn what this prompting technique looks like, and they could drop it into ChatGPT if they want, and drag the lecture files into another AI if they preferred. Um, there were some difficulties. Uh, it turns out I didn't. I didn't plan this as like a pilot or a test or anything, and in hindsight, I should have. I didn't poll them or survey them about their usage of it afterwards, so I wasn't sure how actively they were using it or not. I know they used it sometimes for the when I prompted them to do it. I'm not sure if they got really into it or if it was just another thing like the reading list that they largely ignore. Um, some students really didn't want to talk about AI. Even when I gave them an opportunity, they seemed to think it was. Anything AI related is about academic misconduct, and they get super scared about talking about it and just closed their mouth. Uh, one student didn't want to engage with it for, uh, ethical reasons. He was concerned about the environmental impact of AI and didn't want to talk about it. Um, so I think this is a really interesting approach to try. You can really tailor an AI to the requirements of a module, and you get a lot more out of it. It's a much more connected to what you're doing in the module.

I think that in the next 5 or 10 years, there's going to be huge, uh, opportunities for this kind of approach. It will probably be much better than this. It'll be integrated into our values and just automatically on. Um, and this was just one small step in that direction