surprise (statistical)

The negative log of evidence

$- l n (P (\underline{y}))$ Note that $0 \leq P (\underline{y}) \leq 1$ , so surprise will always be positive when $P (\underline{y}) \neq = 1$ . There is no surprise when there is certainty.

Surprise represents how well (or bad) a model predicts data. A bad model will have a low probability associated with evidence and the surprise will be large.