What does it mean when we say a student gets 6/10 in a quiz? Well, they have answered 6 questions out of 10 correctly. But is that a good result for this student? How hard is the quiz? Attaining 6/10 in one quiz may be much harder than another quiz. Which 6 questions did they answer incorrectly? How can this be used for comparison - to other students in the class and previous performance for this student. Whilst understandable, simple metrics like this are not that useful.
Well, we think so. Working with Microsoft Research, we have developed a machine learning algorithm which can predict how students will answer future questions based on their past answers. Then using these predictions we calculate a score for a student in the following way:
Now scores are comparable, if one student scores lower than another, we believe them to be less able. If a student’s score goes up over time we believe they are making more progress than other students.
In order to calculate a score for a student we need to be able to predict their responses to 1000s of questions. Our machine learning algorithm can do this and it requires surprisingly few answers from the student. However, the quality of the predictions depends on the questions the student has answered.
Our machine learning algorithm is able to choose the best next question for a student based on their previous answers. It finds the question which maximises the “Information Reward”, a measure of the amount of information gained by answering the question. Maximising IR increases our ability to predict how students will answer questions they have never seen before.
We serve up 10 questions to each student using this method in a dynamic quiz. There are 1024 different paths students can take through this quiz. The benefit of a dynamic quiz like this is that it adjusts to the ability level of the student. Harder questions will be selected for more able students and easier questions for less able students. Metrics based on the number of correct answers, e.g. 6/10, now make no sense because students are shown different questions.
Dynamically adjusting questions will give more confidence to less able students and stretch the most able students. But that's not all. Asking questions that students will definitely get wrong/right has little value.
After the student has completed the dynamic quiz, and their initial score has been calculated, we recalculate the score every week.
The score is a useful summary for monitoring student progress over time and helping to understand where they are in their cohort of similar users. We can also calculate scores for different topic areas allowing us to highlight personal strengths and any weak spots which might benefit from a bit of help.
In conclusion, we have developed a score which is a powerful summary of student knowledge. The process of calculating the score is more complex than stating the number of correct answers in a quiz but the score is far more useful. It can be used to position a student within their year group, monitor their progress over time, and identify areas of strength and weakness.
