I am teaching some math courses, and I see how LLMs disrupted all standard approaches, and I don't know what to do.
I rely on written exams that are open book, but forbid any use of computers and smartphones in class.
The university insists that homework can't be optional, but it lost its meaning. I've tried to explain to my students that it's in their interest to think about home assignments on their own, but the majority will obviously use gen AI, and it's such a waste of time to check and grade LLM output.
I just had an experience where all students were given "sample problems" to try at home and prepare for the written test. Many of them just dumped the document into an LLM, asked it to produce some "exam guide", and showed up with that thing printed out, asking me during the test to explain what LLM output meant.
It seems like the university also has many people pushing for AI use for everything, but I teach basic stuff where the goal is to make students think on their own and digest some fundamental ideas, LLMs can produce perfect solutions, but relying on them is pointless.
I'm taking a year-long sabbatical to study a language at university right now and they're changing their assignment processes to curtail AI use. We have two versions of written assignments - the first to be written by hand in class with just a Swedish-Swedish word book available, and the second version to be typed at home implementing the professor's corrections from v1.
At least in this course students seem to be taking their strict 'no AI' policy quite seriously. Then again this is also the kind of course you do when you want to actually learn the material. It provides a qualification for further higher education in Swedish, so if we don't legitimately learn the stuff there's just no point as we won't manage in future courses.
There is indeed no point in grading home assignments anymore, but why not randomly ask a student at the start of each class to present their solution on the board?
And do a final exam on oen and paper and nothing else, and an exam that relies more on thinking than rote memorization of formulas (or print the relevant parts of the course on the exam document itself).
It's funny how people talk about "teaching" getting more difficult with LLMs.
Clearly teaching got a lot better, since we now have a new teaching tool at our use, the LLMs. It's evaluating that got harder, because students can use the same tool to cheat in traditional forms of evaluation.
If "teaching the students" means the same as "evaluating the students" to you, maybe you shouldn't teach in the first place.
It’s not just evaluating. Students used to learn skills while doing an assignment. LLMing through assignments teaches copy-paste at best.
It’s like copy-paste Wikipedia presentations 2 decades ago. People used to learn things doing research for a project. Then suddenly it became copying off Wikipedia and people not even pre-reading what they copied into PPT.
I have a fairly long experience of both teaching and grading, even though I stopped a couple of years ago. After a couple of years of trial-and-error, I think I found the "right" way:
* Alternate between theory and practice in the course of a single 2-hours slot. Theory is necessary, but the attention span of people is very limited.
* Grade either a homework assignment in the form of a real project with specifications. You give the assignment half-semester, so that students who start early can ask questions. Otherwise, grade a in-session exam, with every resource available, including the course and the whole Internet. However, you don't assess the knowledge, but if the student is able to apply their knowledge to the different tasks at hand.
These approaches are now completely moot in the age of AI.
I was surprised the first time a colleague asked me to do an oral exam. I thought it was a burden on people with poor social interaction skills. It went surprisingly well: you could see in a couple of minutes if the student had integrated the concepts of the course, even if they were shy. Now I wonder if it's the way to go. There's one caveat, though: it doesn't scale.
I was thinking that one of the great ironies of artificial intelligence is that it will make good teaching even more labour intensive. Those kids that can afford the one to one tuition required to push past our natural inclination to be lazy will learn things, while everyone else will succumb to outsourcing all their thinking.
There’s no way I’d have learnt everything I know now about software in today’s environment. So much of software development - debugging techniques, architectural decisions, structuring data etc is learned through trial and error. As the models get better is becoming easier and easier to just not look too closely at their output. I suspect it’s human nature.
Oral exams don't scale at the beginning with hundreds of students per course, agreed. So make them in-person written exams, German universities managed that just fine a few decades ago. Oral exams were reserved to advance from Grundstudium (basic) to Hauptstudium (main).
In my very limited experience (I have German major degrees in Informatik and Philosophy, I was a TA for Logic in Philosophy) the more advanced the topic the thinner the attendance anyway.
The country of Argentina, despite its crippling debt, is able to provide free university education where the majority of classes for most majors are graded with an oral exam.
They've shown that it is entirely possible to scale a system like that, as long as the society sincerely values the role of an educator.
> I now shifted to in-person interactions with a TA. After every assignment, each student needs to schedule a 15-minute meeting with a TA to answer a couple of questions in a live conversation
Isn't the answer the 'flipped classroom' - students are assigned reading/studying to do before a class and the class becomes more interactive, answering questions/discussing topics/solving problems etc depending on subject. Of course, this is more expensive than a hall with 400 people listening to a lecture with a couple of tests/essays.
This only works if you either have very intrinsically motivated students or you verify that really everyone has actually read the assigned materials before class. Otherwise, some people will turn up unprepared, you will have to explain stuff they should have gotten from the reading assignment, other students will be annoyed, students will stop preparing for class, vicious cycle.
Maybe the Germans were right? One in-person test, preferably an interview, at the end of the semester should be enough to motivate the smarter part of the bunch to make sure they understand the issues at hand. The rest is chaff anyway. Don't optimize for chaff.
We still have developers because we can't outsource responsibility to AI. If shit breaks, we still need someone to fix it. Maybe you think that can also be automated. I have a counter-example. AI will rarely use the existing state when solving a bug. It will tack on it's own event queue, it will make up it's own messaging system, it is a cancerous growth. And this observation is based on both opus5.5 and gpt5.6-astra. The results are way past magical at this point, but you still have to weigh in pretty heavily for orderly code growth where solutions are succint, focused and preserve the existing structure. They are still noobs at fixing/adding stuff.
This makes me think of exams — I always did well on open-book exams, but I didn't do well on exams where I couldn't look at the book. Whether it's because of ADHD or just a lack of memorization ability.
Similarly, I did well in interviews where I could use the internet, but I didn't do well in interviews where I couldn't use the internet.
Everyone has a testing format they're good at. I'm curious about how this kind of thing is evaluated, and whether there are any papers related to this.
I rely on written exams that are open book, but forbid any use of computers and smartphones in class.
The university insists that homework can't be optional, but it lost its meaning. I've tried to explain to my students that it's in their interest to think about home assignments on their own, but the majority will obviously use gen AI, and it's such a waste of time to check and grade LLM output.
I just had an experience where all students were given "sample problems" to try at home and prepare for the written test. Many of them just dumped the document into an LLM, asked it to produce some "exam guide", and showed up with that thing printed out, asking me during the test to explain what LLM output meant.
It seems like the university also has many people pushing for AI use for everything, but I teach basic stuff where the goal is to make students think on their own and digest some fundamental ideas, LLMs can produce perfect solutions, but relying on them is pointless.
At least in this course students seem to be taking their strict 'no AI' policy quite seriously. Then again this is also the kind of course you do when you want to actually learn the material. It provides a qualification for further higher education in Swedish, so if we don't legitimately learn the stuff there's just no point as we won't manage in future courses.
And do a final exam on oen and paper and nothing else, and an exam that relies more on thinking than rote memorization of formulas (or print the relevant parts of the course on the exam document itself).
Perhaps some people do not belong at university? It is just a waste of everyone time, to insist 50% of population studies until 25 years old.
Clearly teaching got a lot better, since we now have a new teaching tool at our use, the LLMs. It's evaluating that got harder, because students can use the same tool to cheat in traditional forms of evaluation.
If "teaching the students" means the same as "evaluating the students" to you, maybe you shouldn't teach in the first place.
It’s like copy-paste Wikipedia presentations 2 decades ago. People used to learn things doing research for a project. Then suddenly it became copying off Wikipedia and people not even pre-reading what they copied into PPT.
* Alternate between theory and practice in the course of a single 2-hours slot. Theory is necessary, but the attention span of people is very limited. * Grade either a homework assignment in the form of a real project with specifications. You give the assignment half-semester, so that students who start early can ask questions. Otherwise, grade a in-session exam, with every resource available, including the course and the whole Internet. However, you don't assess the knowledge, but if the student is able to apply their knowledge to the different tasks at hand.
These approaches are now completely moot in the age of AI.
I was surprised the first time a colleague asked me to do an oral exam. I thought it was a burden on people with poor social interaction skills. It went surprisingly well: you could see in a couple of minutes if the student had integrated the concepts of the course, even if they were shy. Now I wonder if it's the way to go. There's one caveat, though: it doesn't scale.
There’s no way I’d have learnt everything I know now about software in today’s environment. So much of software development - debugging techniques, architectural decisions, structuring data etc is learned through trial and error. As the models get better is becoming easier and easier to just not look too closely at their output. I suspect it’s human nature.
In my very limited experience (I have German major degrees in Informatik and Philosophy, I was a TA for Logic in Philosophy) the more advanced the topic the thinner the attendance anyway.
They've shown that it is entirely possible to scale a system like that, as long as the society sincerely values the role of an educator.
Isn't the answer the 'flipped classroom' - students are assigned reading/studying to do before a class and the class becomes more interactive, answering questions/discussing topics/solving problems etc depending on subject. Of course, this is more expensive than a hall with 400 people listening to a lecture with a couple of tests/essays.
https://fltmag.com/the-flipped-classroom/
https://en.wikipedia.org/wiki/Flipped_classroom
Similarly, I did well in interviews where I could use the internet, but I didn't do well in interviews where I couldn't use the internet.
Everyone has a testing format they're good at. I'm curious about how this kind of thing is evaluated, and whether there are any papers related to this.
AI can talk the talk, but it can't walk the walk.
A simple "explain why this is the best solution" will make it shit the bed 10 out 10 times.