# Technical Details

Digory Smith

March 15, 2022

## Motivation (last blog post)

- In the last blog post we justified the need for a new measure of ability and argued for the Eedi score being a suitable candidate.
- In this blog post we describe the model we use to calculate the Eedi score.
- Microsoft Research....

## Learning from other students' answers

- When a student makes a mistake an experienced teacher may be able to predict...
- How can we say much about a student when they have only answered a few questions
- Teachers can do this because they have seen many other students make the same mistakes.
- On Eedi there are over 100 million answers, lots of mistakes, and lots of patterns to learn from.

## How do we learn?

- Machine Learning is.... learning from big data
- Jokey - no "crazy robots", learning patterns in the data
- Collaborative filtering is specifically this user-answer-thing
- Netflix

## What are we predicting?

- Probability a student answers a question correctly or not.
- Every question in database
- <Diagram>

## How does this work?

- The model has to be "trained"
- Means tweaking the model until it predicts the observed answers as accurately as possible.
- Can take days to run on a computer.
- Compare this to the speed you normally expect a computer to operate... There is a LOT going on.

## What do we get at the end of the training?

- The result is a function which takes the answers a student has given and can predict how they will answer every other question in the dataset.
- This is typically around 73-76% accurate.

## Summarising these predictions

- 100000 probabilities not much use to anyone (but very useful within Eedi Family).
- Need something more actionable.
- Can be aggregated (grouped) in many more useful ways.
- E.g. by topic which can help us predict which topic they need to focus on next.
- Also by concept, this allows us to predict whether a student will understand a given concept or not... (not quite)

## Calculating scores

- We count up the number of concepts they are more likely to answer correctly. This gives their score.
- By the end of the current academic year.
- Predictive (The model predicts whether a student is likely to answer a question correctly when they are taught that topic.)
- Actionable
- So there is a great deal of extrapolation, but we don't trust this extrapolation. (Only use current year). Because it is actionable and therefore we hope students will be better than we predict.

## Picking Questions

- The questions aim to capture the most information about their knowledge.
- The questions that they are asked is dependant on their previous answers: The dynamic quiz.

## Further Reading

- Reference all the papers.
- Happy to chat....

### Eedi Research Blog