MultiScorer

class pycalib.scoring.MultiScorer(metrics, plots)

Bases: object

Use this class to encapsulate and/or aggregate multiple scoring functions so that it can be passed as an argument for scoring in scikit’s cross_val_score function. Instances of this class are also callables, with signature as needed by cross_val_score. Evaluating multiple scoring function in this way versus scikit learns native way in the cross_validate function avoids the unnecessary overhead of predicting anew for each scorer. This class is slightly adapted from Kyriakos Stylianopoulos’s implementation 1.

1

https://github.com/StKyr/multiscorer

Methods Summary

__call__(estimator, X, y)	To be called by for evaluation from sklearn’s GridSearchCV or cross_val_score.
get_metric_names()	Get all the metric names as given when initialized.
get_results([metric, fold])	Get the results of a specific or all the metrics.

Methods Documentation

__call__(estimator, X, y)

To be called by for evaluation from sklearn’s GridSearchCV or cross_val_score. Parameters are as they are defined in the respective documentation.

Returns

dummy – A dummy value of 0.5 just for compatibility reasons.

Return type

float

get_metric_names()

Get all the metric names as given when initialized.

Returns

metric_names – A list containing the given names (str) of the metrics

Return type

list

get_results(metric=None, fold='all')

Get the results of a specific or all the metrics.

This method should be called after the object itself has been called so that the metrics are applied.

Parameters

• metric (str or None (default)) – The given name of a metric to return its result(s). If omitted the results of all metrics will be returned.

• fold (int in range [1, number_of_folds] or 'all' (Default)) – Get the metric(s) results for the specific fold. The number of folds corresponds to the number of times the instance is called. If its value is a number, either the score of a single metric for that fold or a dictionary of the (single) scores for that fold will be returned, depending on the value of metric parameter. If its value is ‘all’, either a list of a single metric or a dictionary containing the lists of scores for all folds will be returned, depending on the value of metric parameter.

Returns

• metric_result_for_one_fold – The result of the designated metric function for the specific fold, if metric parameter was not omitted and an integer value was given to fold parameter. If the value of metric does not correspond to a metric name, None will be returned.

• all_metric_results_for_one_fold (dict) – A dict having as keys the names of the metrics and as values their results for the specific fold. This will be returned only if metric parameter was omitted and an integer value was given to fold parameter.

• metric_results_for_all_folds (list) – A list of length number_of_folds containing the results of all folds for the specific metric, if metric parameter was not omitted and value ‘all’ was given to fold. If the value of metric does not correspond to a metric name, None will be returned.

• all_metric_results_for_all_folds (dict of lists) – A dict having as keys the names of the metrics and as values lists (of length number_of_folds) of their results for all folds. This will be returned only if metric parameter was omitted and ‘all’ value was given to fold parameter.

Raises

• UserWarning – If this method is called before the instance is called for evaluation.

• ValueError – If the value for fold parameter is not appropriate.