March 15, 2022
Motivation (last blog post)
- In the last blog post we justified the need for a new measure of ability and argued for the Eedi score being a suitable candidate.
- In this blog post we describe the model we use to calculate the Eedi score.
- Microsoft Research....
Learning from other students' answers
- When a student makes a mistake an experienced teacher may be able to predict...
- How can we say much about a student when they have only answered a few questions
- Teachers can do this because they have seen many other students make the same mistakes.
- On Eedi there are over 100 million answers, lots of mistakes, and lots of patterns to learn from.
How do we learn?
- Machine Learning is.... learning from big data
- Jokey - no "crazy robots", learning patterns in the data
- Collaborative filtering is specifically this user-answer-thing
What are we predicting?
- Probability a student answers a question correctly or not.
- Every question in database
How does this work?
- The model has to be "trained"
- Means tweaking the model until it predicts the observed answers as accurately as possible.
- Can take days to run on a computer.
- Compare this to the speed you normally expect a computer to operate... There is a LOT going on.
What do we get at the end of the training?
- The result is a function which takes the answers a student has given and can predict how they will answer every other question in the dataset.
- This is typically around 73-76% accurate.
Summarising these predictions
- 100000 probabilities not much use to anyone (but very useful within Eedi Family).
- Need something more actionable.
- Can be aggregated (grouped) in many more useful ways.
- E.g. by topic which can help us predict which topic they need to focus on next.
- Also by concept, this allows us to predict whether a student will understand a given concept or not... (not quite)
- We count up the number of concepts they are more likely to answer correctly. This gives their score.
- By the end of the current academic year.
- Predictive (The model predicts whether a student is likely to answer a question correctly when they are taught that topic.)
- So there is a great deal of extrapolation, but we don't trust this extrapolation. (Only use current year). Because it is actionable and therefore we hope students will be better than we predict.
- The questions aim to capture the most information about their knowledge.
- The questions that they are asked is dependant on their previous answers: The dynamic quiz.
- Reference all the papers.
- Happy to chat....
Eedi Research Blog