MNISTData¶
-
class
pycalib.benchmark.MNISTData(run_dir, clf_output_dir, classifier_names, cal_methods, cal_method_names, n_splits=10, test_size=0.9, train_size=None, random_state=None)[source]¶ Bases:
pycalib.benchmark.BenchmarkModel evaluation using the benchmark vision dataset MNIST.
Implements a data generation method returning a new evaluation data set for each scoring round.
- Parameters
run_dir (str) – Directory to run benchmarking in and save output and logs to.
clf_output_dir (str) – Directory containing calibration data obtained from MNIST classification.
classifier_names (list) – Names of classifiers to be calibrated. Classification results on MNIST must be contained in data_dir.
cal_methods (list) – Calibration methods to benchmark.
cal_method_names (list) – Names of calibration methods.
n_splits (int, default=10) – Number of splits for cross validation.
test_size (float, default=0.9) – Size of test set.
train_size (float, default=None) – Size of calibration set.
random_state (int, RandomState instance or None, optional (default=None)) – If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.
Methods Summary
classify_val_data(file, clf_name, classifier)Classify the MNIST evaluation data set with a given model.
data_gen()Returns the full dataset or a generator of datasets.
plot(out_file, results_file, score, methods)Plot results from benchmark experiments as an error bar plot.
run([n_jobs])Train all models, evaluate on test data and save the results.
Methods Documentation
-
static
classify_val_data(file, clf_name, classifier, output_folder='clf_output')[source]¶ Classify the MNIST evaluation data set with a given model.
- Parameters
file (str) – Output from the model is saved in the given directory.
clf_name – Name of classifier.
classifier (sklearn.base.BaseEstimator) – Classifier to classify MNIST data with.
output_folder (str, default='clf_output') – Name of folder where output is to be stored. This folder is created if non-existent.
-
data_gen()[source]¶ Returns the full dataset or a generator of datasets.
- Returns
- Return type
X, y giving uncalibrated predictions and corresponding classes.
-
static
plot(out_file, results_file, score, methods, classifiers='all', width=5.0, height=2.5)¶ Plot results from benchmark experiments as an error bar plot.
- Parameters
out_file (str) – File location for the output plot.
results_file (str) – The location of the csv files containing experiment results.
score (str) – Type of score to plot.
methods (list) – Calibration methods to plot.
classifiers (list or "all") – List of classifiers for which to show results.
width (float, default=5.) – Width of the plot.
height (float, default=2.5) – Height of the plot.