reliability_diagram

pycalib.plotting.reliability_diagram(y, p_pred, filename, title='Reliability Diagram', n_bins=100, show_ece=False, show_legend=False, model_name=None, xlim=None, plot_height=4, plot_width=4)[source]

Plot a reliability diagram

This function plots the reliability diagram 1 2 and histograms from the given confidence estimates. Reliability diagrams are a visual aid to determine whether a classifier is calibrated or not.

Parameters

  • y (array, shape = [n_methods, n_samples]) – Ground truth labels.

  • y_pred (array, shape = [n_methods, n_samples]) – Predicted labels.

  • p_pred (array or list) – Array of confidence estimates. If this is a list, multiple reliability diagrams will be plotted and arranged side-by-side.

  • filename (str) – Path or name of output plot files.

  • title (str or list) – Title of plot. If p_pred is a list, titles of each individual reliability diagram.

  • n_bins (int, optional, default=20) – The number of bins into which the y_pred are partitioned.

  • show_ece (bool) – Whether the expected calibration error (ECE) should be displayed in the plot.

  • show_legend (bool) – Whether the legend should be displayed in the plot.

  • model_name (str) – Name of the model from which the probabilities were generated. Displayed when showing the legend.

  • xlim (array, shape = (2,), default=None) – X-axis limits. If note provided inferred from y.

References

1

DeGroot, M. H. & Fienberg, S. E. The Comparison and Evaluation of Forecasters. Journal of the Royal Statistical Society. Series D (The Statistician) 32, 12–22.

2

Niculescu-Mizil, A. & Caruana, R. Predicting good probabilities with supervised learning in Proceedings of the 22nd International Conference on Machine Learning (2005)