# Technical Details

Digory Smith

## Motivation (last blog post)

• In the last blog post we justified the need for a new measure of ability and argued for the Eedi score being a suitable candidate.
• In this blog post we describe the model we use to calculate the Eedi score.
• Microsoft Research....

## Learning from other students' answers

• When a student makes a mistake an experienced teacher may be able to predict...
• How can we say much about a student when they have only answered a few questions
• Teachers can do this because they have seen many other students make the same mistakes.
• On Eedi there are over 100 million answers, lots of mistakes, and lots of patterns to learn from.

## How do we learn?

• Machine Learning is.... learning from big data
• Jokey - no "crazy robots", learning patterns in the data
• Collaborative filtering is specifically this user-answer-thing
• Netflix

## What are we predicting?

• Probability a student answers a question correctly or not.
• Every question in database
• <Diagram>

## How does this work?

• The model has to be "trained"
• Means tweaking the model until it predicts the observed answers as accurately as possible.
• Can take days to run on a computer.
• Compare this to the speed you normally expect a computer to operate... There is a LOT going on.

## What do we get at the end of the training?

• The result is a function which takes the answers a student has given and can predict how they will answer every other question in the dataset.
• This is typically around 73-76% accurate.

## Summarising these predictions

• 100000 probabilities not much use to anyone (but very useful within Eedi Family).
• Need something more actionable.
• Can be aggregated (grouped) in many more useful ways.
• E.g. by topic which can help us predict which topic they need to focus on next.
• Also by concept, this allows us to predict whether a student will understand a given concept or not... (not quite)

## Calculating scores

• We count up the number of concepts they are more likely to answer correctly. This gives their score.
• By the end of the current academic year.
• Predictive (The model predicts whether a student is likely to answer a question correctly when they are taught that topic.)
• Actionable
• So there is a great deal of extrapolation, but we don't trust this extrapolation. (Only use current year). Because it is actionable and therefore we hope students will be better than we predict.

## Picking Questions

• The questions aim to capture the most information about their knowledge.
• The questions that they are asked is dependant on their previous answers: The dynamic quiz.

• Reference all the papers.
• Happy to chat....