Technical Details

Motivation (last blog post)

• In the last blog post we justified the need for a new measure of ability and argued for the Eedi score being a suitable candidate.
• In this blog post we describe the model we use to calculate the Eedi score.
• Microsoft Research....

Learning from other students' answers

• When a student makes a mistake an experienced teacher may be able to predict...
• How can we say much about a student when they have only answered a few questions
• Teachers can do this because they have seen many other students make the same mistakes.
• On Eedi there are over 100 million answers, lots of mistakes, and lots of patterns to learn from.

How do we learn?

• Machine Learning is.... learning from big data
• Jokey - no "crazy robots", learning patterns in the data
• Collaborative filtering is specifically this user-answer-thing
• Netflix

What are we predicting?

• Probability a student answers a question correctly or not.
• Every question in database
• <Diagram>

How does this work?

• The model has to be "trained"
• Means tweaking the model until it predicts the observed answers as accurately as possible.
• Can take days to run on a computer.
• Compare this to the speed you normally expect a computer to operate... There is a LOT going on.

What do we get at the end of the training?

• The result is a function which takes the answers a student has given and can predict how they will answer every other question in the dataset.
• This is typically around 73-76% accurate.

Summarising these predictions

• 100000 probabilities not much use to anyone (but very useful within Eedi Family).
• Need something more actionable.
• Can be aggregated (grouped) in many more useful ways.
• E.g. by topic which can help us predict which topic they need to focus on next.
• Also by concept, this allows us to predict whether a student will understand a given concept or not... (not quite)

Calculating scores

• We count up the number of concepts they are more likely to answer correctly. This gives their score.
• By the end of the current academic year.
• Predictive (The model predicts whether a student is likely to answer a question correctly when they are taught that topic.)
• Actionable
• So there is a great deal of extrapolation, but we don't trust this extrapolation. (Only use current year). Because it is actionable and therefore we hope students will be better than we predict.

Picking Questions

• The questions aim to capture the most information about their knowledge.
• The questions that they are asked is dependant on their previous answers: The dynamic quiz.

Further Reading

• Reference all the papers.
• Happy to chat....

‍