brier_score

pycalib.scoring.brier_score(y, p_pred)

Compute the Brier score.

The smaller the Brier score, the better, hence the naming with “loss”. Across all items in a set N predictions, the Brier score measures the mean squared difference between (1) the predicted probability assigned to the possible outcomes for item i, and (2) the actual outcome. Therefore, the lower the Brier score is for a set of predictions, the better the predictions are calibrated. Note that the Brier score always takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1). The Brier loss is composed of refinement loss and calibration loss.

Note: We interface the sklearn.metrics.brier_score_loss method here to provide a consistent method signature.

Parameters

• y (array-like) – Ground truth labels. Here a dummy variable for cross validation.

• p_pred (array-like) – Array of confidence estimates.

Returns

score – Brier score

Return type

float